Where bias enters a healthcare AI model, and what can be done at each stage

By 13 May 2024, the US Food and Drug Administration listed 882 AI-enabled medical devices it had authorised, about three quarters of them in radiology, followed by cardiology and neurology. The list has kept growing since, and passed 1,600 devices in 2026. Most of these tools read images, monitor signals or estimate risk, and each one was trained on a particular set of patients.

That last point is the subject of a review published in npj Digital Medicine in March 2025 by Fereshteh Hasanzadeh and colleagues, which the Tech Anatomy episode on hidden bias was built around. The authors describe bias as a systematic difference in how a model treats different groups of patients, and they map where it can enter across a model's life, from the moment the project is designed to years after deployment.

Bias in the data

The most familiar source is the training data. A model learns patterns from the patients it sees, and performs less reliably on patients unlike them. The review cites a 2023 systematic review in JAMA Network Open of 555 neuroimaging-based AI models for psychiatric diagnosis, described in 517 studies. In that analysis, 97.5 percent of the samples came from high-income or upper-middle-income countries, and 83 percent of the models were rated at high risk of bias overall. The episode, following the review's summary, described the 97.5 percent as data from high-income countries only; the original study groups high-income and upper-middle-income countries together, which makes the imbalance real but less extreme than the shorthand suggests.

Dermatology offers a direct test. A team at Stanford built a set of clinical images deliberately balanced across skin tones and, in a study published in Science Advances in 2022, found that several leading skin-lesion algorithms performed markedly worse on it than in their original evaluations, and worse on darker skin than on lighter skin. The images these models had learned from were mostly of light skin.

Bias in what the model is asked to predict

The best-known case did not come from missing data. In 2019, Ziad Obermeyer and colleagues reported in Science on a commercial algorithm used by US health systems to decide which patients should get extra care management. It predicted future healthcare costs, on the assumption that cost tracks need. Because less money had historically been spent on Black patients with the same level of illness, the algorithm judged them healthier than equally sick white patients. The authors estimated that correcting the bias would raise the share of Black patients flagged for extra help from 17.7 percent to 46.5 percent. The developer worked with the researchers on a version that predicted health rather than cost.

The lesson the review draws is that the choice of target variable, here cost instead of illness, can encode inequity even when the model is technically accurate at predicting what it was asked to predict.

Bias in use

A third source appears once a model is deployed. Automation bias is the tendency to accept a machine's suggestion without looking for confirming evidence. In a 2023 study in Radiology cited by the review, radiologists read mammograms with AI suggestions, some of them deliberately wrong. When the AI proposed an incorrect category, the accuracy of inexperienced readers fell from about 80 percent to about 20 percent, and that of very experienced readers from 82 percent to 46 percent.

What the review proposes

The mitigation strategies are spread across the same life cycle: datasets that represent the populations a tool will serve, performance reported separately for subgroups rather than as one average, explainability methods that show which inputs drove a result, monitoring after deployment because patient populations and practices change, and clinicians who keep the final decision and are trained to question the output. None of these is a single fix. The review's central argument is that bias has to be checked at every stage, because removing it at one stage does not prevent it from re-entering at the next.

Two readings of the same evidence

One reading, which the episode also voiced, is that AI could reduce bias in medicine. Human decisions already vary with a patient's sex, ethnicity or income, often without anyone measuring it. A model's behaviour can at least be audited across groups, and once a bias is found it can be corrected for every future patient at once, as happened after the Obermeyer study.

A second reading stresses scale and opacity. A biased clinician affects their own patients; a biased model deployed across a health system affects millions, and its errors carry an air of objectivity that makes them harder to challenge. Commercial pressure to ship quickly, and validation studies that report only average performance, make it likely that some biases go undetected for years.

Both readings agree that measurement is the condition. They differ on whether today's incentives make that measurement likely.

What the evidence does not show

Most documented cases are retrospective: researchers found the bias after the fact, in data or in published models. There is far less evidence on how much biased tools have changed outcomes for real patients, because few deployments are followed with that question in mind. Fairness itself has several mathematical definitions that cannot all be satisfied at once, so "removing bias" always involves a choice about which kind of equality matters most for a given use.

What to watch

The indicators worth following are whether regulators start requiring performance broken down by subgroup in authorisation files, a direction already visible in the FDA's January 2025 draft guidance on AI-enabled device software and in the data governance rules of the EU AI Act, whether hospitals publish monitoring results for the tools they deploy, and whether studies of AI in practice start reporting outcomes by population rather than accuracy alone. Each would turn the question from what could go wrong into what has been measured.

Listen to the episode

The Hidden Bias in Healthcare AI

Listen · 7 min