AI-Enabled Emergency Triage Tools Mirror Human Biases

A new study conducted by researchers at UT Southwestern Medical Center has revealed that the use of artificial intelligence in emergency room triage can be influenced by clinical stigmatization seen in human evaluations. The findings, published in the journal npj Digital Medicine, show that this situation can lead to the misprioritization of patients and harmful biases.

Researchers examined patient records from the emergency departments of Stanford Medicine in California and Beth Israel Deaconess Medical Center in Massachusetts. By matching patients with similar initial conditions, three open-source AI language models were asked which patients might deteriorate and require priority care. The models successfully identified high-risk patients before demographic or potentially stigmatizing information was added.

However, when stigmatizing information such as mental health history, substance use, age, and emergency room utilization patterns was added, AI decisions were found to be affected. Particularly in cases involving frequent emergency visits or a history of psychiatric emergencies, it was determined that up to approximately 10 percent of patients at higher risk were incorrectly assigned lower priority. The researchers emphasize that every AI tool must be tested against actual patient outcomes before being put into use.