Chapter 2

Algorithmic Bias and Fairness

Algorithmic bias in healthcare AI represents one of the most pressing ethical challenges of our time. When AI systems perform differently across demographic groups, they can perpetuate and amplify existing health disparities, undermining medicine's commitment to treating all patients equitably.

Understanding Algorithmic Bias

Algorithmic bias occurs when an AI system systematically produces outcomes that favor certain groups over others in ways that are unjustified and potentially harmful. In healthcare, this manifests as differential performance across patient populations—an AI that detects skin cancer accurately on light-skinned patients but misses melanomas on darker skin, or a risk prediction model that underestimates illness severity in Black patients.

The consequences of biased healthcare AI extend beyond individual misdiagnoses. Biased algorithms can systematically direct resources away from populations that need them most, reinforce historical patterns of discrimination, and erode trust in healthcare systems among already marginalized communities. Understanding the sources of bias is the first step toward building fairer systems.

Sources of Bias in Healthcare AI

Training Data Bias

AI systems learn from historical data, and that data often reflects decades of healthcare inequities. Clinical trials have historically underrepresented women, minorities, and elderly patients. Electronic health records capture healthcare as it was delivered—including patterns of undertreatment, missed diagnoses, and disparate access that varied by demographic. When AI learns from this data, it learns these disparities as "normal."

Bias Source	Mechanism	Example
Selection Bias	Training data not representative of population	Dermatology AI trained mostly on light-skinned patients
Historical Bias	Data reflects past discriminatory practices	Pain management algorithms undertreat minorities
Measurement Bias	Outcomes measured differently across groups	Using healthcare costs as proxy for health needs
Label Bias	Ground truth labels contain systematic errors	Underdiagnosis of depression in men affects labels
Missing Data Bias	Important variables absent for some populations	Social determinants missing for uninsured patients

Algorithmic Design Bias

Even with representative data, algorithmic choices can introduce or amplify bias. The selection of features, the definition of outcomes, the choice of optimization objectives, and model architecture decisions all influence fairness. An algorithm optimizing for overall accuracy may sacrifice performance on minority subgroups if that improves aggregate metrics.

Deployment Context Bias

How AI systems are deployed can create disparities even if the underlying algorithm is fair. Hospitals serving affluent populations may have higher-quality imaging equipment, making AI interpretations more accurate. Patients with limited health literacy may not understand AI-driven recommendations. Clinicians may override AI suggestions more frequently for patients they perceive as similar to themselves.

Case Studies in Healthcare AI Bias

Definitions of Fairness

A challenge in addressing algorithmic bias is that fairness itself has multiple definitions that are often mathematically incompatible. Different stakeholders may prioritize different fairness criteria, and an algorithm cannot simultaneously satisfy all of them.

Fairness Definition	Criterion	Limitation
Demographic Parity	Equal positive rates across groups	Ignores actual differences in base rates
Equal Opportunity	Equal true positive rates	Allows different false positive rates
Equalized Odds	Equal TPR and FPR across groups	May require accepting lower overall accuracy
Calibration	Same meaning of scores across groups	Can still have different error rates
Individual Fairness	Similar individuals treated similarly	Requires defining "similar"
Counterfactual Fairness	Outcome unchanged if protected attribute changed	Requires causal modeling assumptions

The Impossibility of Simultaneous Fairness

Mathematical analysis has shown that certain fairness criteria cannot be simultaneously achieved when base rates differ between groups. If disease prevalence differs across populations (as it often does), an algorithm cannot have equal false positive rates, equal false negative rates, and equal positive predictive values across groups all at once. This "impossibility theorem" forces difficult trade-offs in algorithm design.

Bias Detection Methods

Pre-Deployment Assessment

• Dataset Audit: Analyze demographic representation in training data
• Feature Analysis: Examine correlation between features and protected attributes
• Proxy Detection: Identify features that may serve as proxies for protected characteristics
• Subgroup Performance: Evaluate model accuracy metrics across demographic groups
• Synthetic Data Testing: Test model behavior with artificially varied demographic inputs

Post-Deployment Monitoring

• Outcome Tracking: Monitor actual patient outcomes by demographic group
• Drift Detection: Identify changes in model performance over time
• User Feedback: Collect clinician reports of unexpected or inappropriate recommendations
• Audit Dashboards: Real-time visualization of fairness metrics
• External Audits: Independent evaluation by third parties

Mitigation Strategies

Data-Level Interventions

Strategy	Approach	Considerations
Diverse Data Collection	Actively recruit underrepresented populations	Expensive; may require community partnerships
Oversampling	Increase weight of minority group samples	Risk of overfitting to minority samples
Synthetic Data	Generate artificial samples for underrepresented groups	May not capture real-world complexity
Label Correction	Re-label potentially biased ground truth	Requires gold standard for correction

Algorithm-Level Interventions

• Fairness Constraints: Add fairness criteria to optimization objective
• Adversarial Debiasing: Train to make predictions unpredictable from protected attributes
• Threshold Adjustment: Use different decision thresholds for different groups
• Ensemble Methods: Combine models trained on different subgroups

Summary

Review Questions

1. What is algorithmic bias and how does it manifest in healthcare AI? Give three examples from the chapter.
2. Describe the five sources of training data bias. How does each contribute to unfair AI systems?
3. What happened in the Optum healthcare algorithm case? What was the root cause and how was it fixed?
4. Explain the difference between demographic parity, equal opportunity, and equalized odds. Why can't all be achieved simultaneously?
5. What methods can be used to detect bias before deploying a healthcare AI system?
6. What are post-deployment monitoring approaches for bias detection? Why is ongoing monitoring necessary?
7. Describe three data-level interventions for reducing algorithmic bias. What are the trade-offs of each?
8. Why is bias mitigation an ongoing process rather than a one-time fix? What factors can cause bias to emerge over time?