Chapter 2

Algorithmic Bias and Fairness

Algorithmic bias in healthcare AI represents one of the most pressing ethical challenges of our time. When AI systems perform differently across demographic groups, they can perpetuate and amplify existing health disparities, undermining medicine's commitment to treating all patients equitably.

Understanding Algorithmic Bias

Algorithmic bias occurs when an AI system systematically produces outcomes that favor certain groups over others in ways that are unjustified and potentially harmful. In healthcare, this manifests as differential performance across patient populations—an AI that detects skin cancer accurately on light-skinned patients but misses melanomas on darker skin, or a risk prediction model that underestimates illness severity in Black patients.

The consequences of biased healthcare AI extend beyond individual misdiagnoses. Biased algorithms can systematically direct resources away from populations that need them most, reinforce historical patterns of discrimination, and erode trust in healthcare systems among already marginalized communities. Understanding the sources of bias is the first step toward building fairer systems.

30%

Algorithms with Demographic Bias

40%

Dermatology AI Accuracy Drop (Dark Skin)

10M

Black Patients Affected by Optum Bias

50%

Fewer Black Patients Identified for Care

Sources of Bias in Healthcare AI

Training Data Bias

AI systems learn from historical data, and that data often reflects decades of healthcare inequities. Clinical trials have historically underrepresented women, minorities, and elderly patients. Electronic health records capture healthcare as it was delivered—including patterns of undertreatment, missed diagnoses, and disparate access that varied by demographic. When AI learns from this data, it learns these disparities as "normal."

Bias Source	Mechanism	Example
Selection Bias	Training data not representative of population	Dermatology AI trained mostly on light-skinned patients
Historical Bias	Data reflects past discriminatory practices	Pain management algorithms undertreat minorities
Measurement Bias	Outcomes measured differently across groups	Using healthcare costs as proxy for health needs
Label Bias	Ground truth labels contain systematic errors	Underdiagnosis of depression in men affects labels
Missing Data Bias	Important variables absent for some populations	Social determinants missing for uninsured patients

Algorithmic Design Bias

Even with representative data, algorithmic choices can introduce or amplify bias. The selection of features, the definition of outcomes, the choice of optimization objectives, and model architecture decisions all influence fairness. An algorithm optimizing for overall accuracy may sacrifice performance on minority subgroups if that improves aggregate metrics.

Deployment Context Bias

How AI systems are deployed can create disparities even if the underlying algorithm is fair. Hospitals serving affluent populations may have higher-quality imaging equipment, making AI interpretations more accurate. Patients with limited health literacy may not understand AI-driven recommendations. Clinicians may override AI suggestions more frequently for patients they perceive as similar to themselves.

Case Studies in Healthcare AI Bias

Case Study: Optum Healthcare Algorithm

In 2019, researchers discovered that a widely-used algorithm for identifying patients who would benefit from additional healthcare resources was systematically disadvantaging Black patients. The algorithm used healthcare costs as a proxy for health needs, but Black patients historically had less access to healthcare services, resulting in lower costs even when equally or more sick.

Impact: ~10 million Black patients affected annually
Effect: Only 17.7% of patients flagged for extra care were Black vs. expected 46.5%
Root Cause: Using costs (affected by access) as proxy for medical needs
Fix: Replacing cost proxy with health-based prediction reduced bias by 84%

Case Study: Dermatology AI Performance Disparities

Multiple studies have demonstrated that dermatology AI systems perform worse on darker skin tones. A Stanford study found that some AI systems had accuracy drops of 20-40% on dark-skinned patients compared to light-skinned patients.

Root Cause: Training datasets vastly overrepresent light-skinned patients
Consequence: Potential for missed skin cancer diagnoses in Black and Brown patients
Industry Response: Efforts to create diverse dermatology image datasets

Definitions of Fairness

A challenge in addressing algorithmic bias is that fairness itself has multiple definitions that are often mathematically incompatible. Different stakeholders may prioritize different fairness criteria, and an algorithm cannot simultaneously satisfy all of them.

Fairness Definition	Criterion	Limitation
Demographic Parity	Equal positive rates across groups	Ignores actual differences in base rates
Equal Opportunity	Equal true positive rates	Allows different false positive rates
Equalized Odds	Equal TPR and FPR across groups	May require accepting lower overall accuracy
Calibration	Same meaning of scores across groups	Can still have different error rates
Individual Fairness	Similar individuals treated similarly	Requires defining "similar"
Counterfactual Fairness	Outcome unchanged if protected attribute changed	Requires causal modeling assumptions

The Impossibility of Simultaneous Fairness

Mathematical analysis has shown that certain fairness criteria cannot be simultaneously achieved when base rates differ between groups. If disease prevalence differs across populations (as it often does), an algorithm cannot have equal false positive rates, equal false negative rates, and equal positive predictive values across groups all at once. This "impossibility theorem" forces difficult trade-offs in algorithm design.

Bias Detection Methods

Pre-Deployment Assessment

Dataset Audit: Analyze demographic representation in training data
Feature Analysis: Examine correlation between features and protected attributes
Proxy Detection: Identify features that may serve as proxies for protected characteristics
Subgroup Performance: Evaluate model accuracy metrics across demographic groups
Synthetic Data Testing: Test model behavior with artificially varied demographic inputs

Post-Deployment Monitoring

Outcome Tracking: Monitor actual patient outcomes by demographic group
Drift Detection: Identify changes in model performance over time
User Feedback: Collect clinician reports of unexpected or inappropriate recommendations
Audit Dashboards: Real-time visualization of fairness metrics
External Audits: Independent evaluation by third parties

Mitigation Strategies

Data-Level Interventions

Strategy	Approach	Considerations
Diverse Data Collection	Actively recruit underrepresented populations	Expensive; may require community partnerships
Oversampling	Increase weight of minority group samples	Risk of overfitting to minority samples
Synthetic Data	Generate artificial samples for underrepresented groups	May not capture real-world complexity
Label Correction	Re-label potentially biased ground truth	Requires gold standard for correction

Algorithm-Level Interventions

Fairness Constraints: Add fairness criteria to optimization objective
Adversarial Debiasing: Train to make predictions unpredictable from protected attributes
Threshold Adjustment: Use different decision thresholds for different groups
Ensemble Methods: Combine models trained on different subgroups

Summary

            Key Takeaways
            Pervasive Problem: Approximately 30% of healthcare AI algorithms show some form of demographic bias.
Multiple Sources: Bias enters through training data, algorithm design, and deployment context.
Real Harms: Biased algorithms have affected millions of patients, directing resources away from those who need them most.
Multiple Fairness Definitions: Different fairness criteria exist and cannot all be satisfied simultaneously.
Detection Required: Both pre-deployment assessment and post-deployment monitoring are necessary.
Mitigation Exists: Data-level and algorithm-level interventions can reduce but not eliminate bias.
Ongoing Vigilance: Bias mitigation is not a one-time fix but requires continuous monitoring and improvement.

        

Review Questions

What is algorithmic bias and how does it manifest in healthcare AI? Give three examples from the chapter.
Describe the five sources of training data bias. How does each contribute to unfair AI systems?
What happened in the Optum healthcare algorithm case? What was the root cause and how was it fixed?
Explain the difference between demographic parity, equal opportunity, and equalized odds. Why can't all be achieved simultaneously?
What methods can be used to detect bias before deploying a healthcare AI system?
What are post-deployment monitoring approaches for bias detection? Why is ongoing monitoring necessary?
Describe three data-level interventions for reducing algorithmic bias. What are the trade-offs of each?
Why is bias mitigation an ongoing process rather than a one-time fix? What factors can cause bias to emerge over time?