Not All Patients Are Equal in the Eyes of AI

A recent study published in Nature Medicine has found that large language models (LLMs) show potential in the healthcare sector. However, concerns remain that these models could generate medically unjustified clinical care recommendations that are influenced by the patient’s sociodemographic characteristics. Researchers evaluated nine LLMs across 1,000 emergency department cases, half real and half synthetic, and each was presented in 32 variations that changed only the patient’s sociodemographic identity (race, income, housing status, gender identity, etc.), while keeping the clinical details exactly the same.

The findings showed that these AI systems often provided different recommendations based on the patient's perceived identity. For instance, cases labelled as Black, unhoused, or LGBTQIA+ were significantly more likely to be directed toward urgent care, invasive procedures, or mental health evaluations even when those were not clinically warranted.

LGBTQIA+ individuals, in particular, were recommended for mental health assessments six to seven times more often than necessary. Meanwhile, high-income individuals tended to receive more favourable or less aggressive recommendations.

High-income patients were more likely to be recommended advanced diagnostic tests, such as CT scans or MRIs, while low-income patients were more frequently advised to undergo no further testing. This pattern mirrors healthcare disparities that exist in the real world.