Chapter 8: Case Studies and Best Practices

This chapter examines real-world case studies of AI bias, lessons learned, and best practices for building fair AI systems. Through concrete examples, we'll see how 弘익人間 principles apply in practice.

Case Study 1: Amazon's Recruiting Tool

Background

Amazon developed an AI recruiting tool to automate resume screening. The system was trained on resumes submitted to Amazon over a 10-year period, with the goal of identifying top candidates.

The Problem

The AI learned to penalize resumes containing the word "women's" (as in "women's chess club") and downgrade candidates from women's colleges. This occurred because the historical data showed a male-dominated hiring pattern in technical roles.

Root Cause

Historical bias: Training data reflected past hiring discrimination
Proxy discrimination: Gender signals existed in resumes even without explicit gender field
Lack of disaggregated evaluation: System wasn't tested separately by gender
Insufficient oversight: Bias wasn't detected before deployment testing

Outcome

Amazon ultimately scrapped the tool after failing to guarantee it wouldn't discriminate. The company stated they never used it for real hiring decisions.

Lessons Learned

Historical data can encode and amplify existing biases
Gender-blind hiring requires removing all gender proxies, not just explicit gender fields
Early testing with disaggregated evaluation is essential
Some problems may be better solved with human judgment supported by AI, not replaced by it

What Should Have Been Done

Audit training data for representation and label bias
Identify and remove proxy features correlated with gender
Test performance separately for different demographic groups
Apply debiasing techniques (reweighting, adversarial debiasing)
Implement human oversight for final decisions

Case Study 2: COMPAS Recidivism Risk Assessment

Background

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a risk assessment tool used in criminal justice to predict recidivism likelihood. It's used for bail, sentencing, and parole decisions across the United States.

The Problem

ProPublica's 2016 investigation found that COMPAS exhibited racial bias:

Black defendants labeled higher risk but didn't reoffend: 45%
White defendants labeled higher risk but didn't reoffend: 23%
Black defendants labeled lower risk but did reoffend: 28%
White defendants labeled lower risk but did reoffend: 48%

Analysis

The algorithm satisfied predictive parity (equal precision for high-risk predictions across races) but violated equalized odds (different false positive and false negative rates). This illustrates the impossibility of satisfying all fairness criteria simultaneously.

Lessons Learned

Different fairness metrics can give conflicting results
Context matters: in criminal justice, false positives (wrongly predicting recidivism) harm individuals
High-stakes decisions require careful selection of appropriate fairness criteria
Transparency is crucial but controversial in criminal justice
Historical arrest data may reflect biased policing, not actual crime rates

Best Practices for Criminal Justice AI

Prioritize equalized odds to ensure equal treatment of innocent and guilty defendants
Conduct intersectional analysis (race × gender, race × age, etc.)
Provide explanations and allow challenges to risk scores
Regular fairness audits by independent third parties
Human review required for all high-risk scores
Consider whether automation is appropriate for these decisions at all

Case Study 3: Healthcare Algorithm Racial Bias

Background

A widely-used healthcare algorithm assigned risk scores to over 200 million people annually, determining who would benefit from high-risk care management programs.

The Problem

The algorithm used healthcare costs as a proxy for health needs. Because Black patients faced systemic barriers to healthcare access and thus incurred lower costs, they were assigned lower risk scores despite being sicker than White patients with similar scores.

Impact

At a given risk score, Black patients were considerably sicker than White patients:

Black patients had 26.3% more chronic conditions
When the algorithm was adjusted to reduce bias, the percentage of Black patients receiving extra help increased from 17.7% to 46.5%

Root Cause

Measurement bias: Healthcare costs are not an unbiased measure of health needs when different populations have different levels of healthcare access.

Solution Implemented

Changed objective from predicting costs to predicting actual health conditions
Used number of active chronic conditions as the outcome instead of costs
Reduced bias by over 80%

Lessons Learned

Choice of target variable (what you're predicting) is crucial
Proxies must be valid across all demographic groups
Domain expertise is essential to identify biased measurements
Post-deployment monitoring can catch bias that wasn't apparent during development
Fixing bias is possible but requires rethinking fundamental modeling choices

Best Practices Summary

1. Data Collection and Preparation

Representative Sampling

Ensure all populations are adequately represented in training data
Use stratified sampling when necessary
Document demographic composition of datasets
Address underrepresentation through targeted data collection

Label Quality

Develop clear, unbiased labeling guidelines
Train annotators on fairness considerations
Use diverse annotation teams
Measure inter-annotator agreement across demographic groups

Feature Selection

Identify and remove unnecessary proxy features
Validate that features measure the same construct across groups
Document rationale for including potentially sensitive features
Test feature importance separately for each demographic group

2. Model Development

Fairness-Aware Training

Define fairness criteria appropriate for your use case
Incorporate fairness constraints into model training
Use techniques like reweighting, adversarial debiasing, or fairness regularization
Accept accuracy-fairness tradeoffs when necessary

Disaggregated Evaluation

Always evaluate performance separately for each demographic group
Report worst-case group performance, not just averages
Conduct intersectional analysis
Test on held-out data that matches deployment distribution

3. Deployment and Monitoring

Continuous Monitoring

Track fairness metrics in real-time
Set up alerts for fairness violations
Monitor for distribution drift
Conduct regular comprehensive audits

Human Oversight

Require human review for high-stakes decisions
Provide explanations for AI recommendations
Enable appeal processes
Train human reviewers on recognizing and addressing bias

4. Organizational Practices

Team Diversity

Build diverse development teams
Include domain experts and affected stakeholders
Establish ethics review boards
Encourage different perspectives in design discussions

Documentation and Transparency

Create comprehensive model cards
Document all fairness-related decisions
Maintain detailed audit trails
Be transparent about limitations and tradeoffs

Stakeholder Engagement

Involve affected communities in system design
Conduct user research across diverse populations
Establish feedback mechanisms
Respond to fairness concerns promptly

Implementation Roadmap

Phase	Activities	Deliverables	Timeline
Phase 1: Assessment	Audit existing systems, identify risks, define fairness criteria	Risk assessment report, fairness requirements	2-4 weeks
Phase 2: Data	Audit data, identify biases, implement corrections	Data quality report, cleaned datasets	4-8 weeks
Phase 3: Modeling	Implement debiasing techniques, test alternatives	Fair models, performance reports	6-12 weeks
Phase 4: Deployment	Set up monitoring, train staff, deploy gradually	Monitoring dashboards, training materials	4-6 weeks
Phase 5: Ongoing	Monitor, audit, refine continuously	Monthly reports, quarterly audits	Continuous

Checklist for Fair AI Systems

弘益人間 Fairness Checklist

Data Stage:

☐ Representative sampling across all demographic groups
☐ Balanced representation (no group < 10% unless unavoidable)
☐ Unbiased labeling with diverse annotators
☐ Proxy feature analysis completed
☐ Data quality consistent across groups

Model Stage:

☐ Fairness criteria defined and documented
☐ Debiasing techniques applied
☐ Disaggregated evaluation completed
☐ Intersectional analysis performed
☐ Model card created

Deployment Stage:

☐ Monitoring system operational
☐ Alerting configured for fairness violations
☐ Human oversight process established
☐ Explanation system implemented
☐ Appeal process available

Organizational Stage:

☐ Diverse team assembled
☐ Ethics review completed
☐ Stakeholders consulted
☐ Documentation comprehensive
☐ Audit trail maintained
☐ Regulatory compliance verified

Ongoing:

☐ Monthly fairness reports generated
☐ Quarterly comprehensive audits conducted
☐ Annual stakeholder reviews held
☐ Continuous improvement plan active

Common Pitfalls to Avoid

Fairness Washing: Claiming fairness without rigorous evaluation
One-Time Assessment: Testing fairness once and never checking again
Aggregate-Only Metrics: Reporting only overall performance without group breakdowns
Ignoring Intersectionality: Not examining compound discrimination
Lack of Stakeholder Input: Building systems without consulting affected communities
Insufficient Documentation: Failing to document fairness-related decisions
Automation Bias: Over-relying on AI without human oversight
Narrow Fairness Definition: Optimizing for one metric while ignoring others
Quick Fixes: Applying superficial solutions to deep-rooted bias problems
Siloed Responsibility: Treating fairness as one team's job rather than organization-wide priority

Final Recommendations

Building fair AI systems is not a one-time project but an ongoing commitment. Success requires:

Leadership commitment: Fairness must be a priority from the top
Resource allocation: Dedicate time, budget, and personnel to fairness
Cultural change: Make fairness part of organizational values
Continuous learning: Stay current with research and best practices
Collaboration: Work with researchers, regulators, and affected communities
Humility: Acknowledge that perfect fairness is aspirational, not achievable
Transparency: Be honest about limitations and ongoing challenges
Action: Continuously improve rather than waiting for perfect solutions

弘益人間 - Benefit All Humanity

The principle of 弘益人間 reminds us that technology should serve all people equitably. As AI becomes increasingly influential in society, ensuring fairness is not optional—it's a moral imperative and practical necessity.

By following the practices outlined in this guide, you can build AI systems that truly benefit all humanity, upholding the dignity and rights of every individual regardless of their background or characteristics.

The journey toward fair AI is ongoing, but every step forward creates a more just and equitable future for all.

Chapter Summary

Amazon's recruiting tool case illustrates dangers of historical bias and proxy discrimination
COMPAS case shows impossibility of satisfying all fairness criteria and importance of context
Healthcare algorithm case demonstrates impact of using biased proxy variables
Best practices span data collection, model development, deployment, and organizational culture
Representative sampling, unbiased labeling, and proxy feature removal are essential data practices
Fairness-aware training and disaggregated evaluation are crucial modeling practices
Continuous monitoring and human oversight are necessary for deployment
Team diversity, documentation, and stakeholder engagement support organizational fairness
Implementation requires phased approach with ongoing commitment
弘益인간 principle guides us toward AI that truly benefits all humanity

Review Questions

What were the root causes of bias in Amazon's recruiting tool? How could they have been prevented?
Explain the fairness tradeoff illustrated by the COMPAS case. Which metric should be prioritized and why?
How did the healthcare algorithm case demonstrate measurement bias? What was the solution?
Why is disaggregated evaluation more informative than aggregate metrics?
What role does team diversity play in building fair AI systems?
Design a deployment plan for a fair lending AI that includes all necessary safeguards.
What are the dangers of "fairness washing"? How can it be prevented?
Why is fairness an ongoing commitment rather than a one-time achievement?
How would you convince leadership to invest resources in AI fairness?
Reflect on how 弘익인간 philosophy has informed your understanding of AI bias throughout this guide.

Congratulations!

You've completed the AI Bias Detection guide. You now have the knowledge and tools to build fairer AI systems that benefit all humanity.

弘益人間 · Benefit All Humanity