By Stacey Kusterbeck
There is much anticipation regarding artificial intelligence’s (AI’s) capacity to transform medicine, including the emergency department (ED) setting. “But there’s also uncertainty regarding how courts will allocate liability when AI contributes to patient injury,” says Neel Guha, a JD candidate at Stanford Law School and PhD candidate at Stanford Computer Science.
Guha and a colleague explored this evolving legal landscape by studying how courts have approached allocation of liability for both AI and other complex medical software.1 The researchers analyzed 51 medical malpractice cases involving allegations of physical injury caused by AI or software systems. Most of the cases involved patients harmed by defective or malfunctioning software (such as hospitals failing to update software). Malpractice claims also alleged negligence on the part of physicians who used AI tools or software to make care decisions (such as whether to screen patients for certain conditions). Some of the lawsuits alleged that the physician should have ignored erroneous software recommendations.
“AI is new, and cases are only just emerging. But courts do not seem to be exceptionalizing AI. Right now, it’s being treated as similar to traditional software,” concludes Guha. Overall, the analysis suggests these are two important approaches for EDs to use when AI tools are implemented:
• EDs should be aware that an AI tool’s performance can be sensitive to particular patient characteristics.
A tool that works well for one ED may not work well for another because of differences in their patient populations. “It’s therefore important that EDs take steps to verify the accuracy and performance characteristics of potential tools for the types of patients they see,” says Guha.
• ED clinicians should be very familiar with how the AI tool fits within clinical workflows.
“It’s critical for practitioners who use and rely upon the tool to know how to use it,” says Guha.
Some emergency physicians (EPs) use AI tools to create a differential diagnosis list.2,3 This can help EPs to think of things they should check for, but it also can lead them astray. “In theory, these AI tools can be very helpful. But they are heavily dependent on sound data collection,” cautions Dean F. Sittig, PhD, professor at UTHealth McWilliams School of Biomedical Informatics. ED malpractice claims alleging missed or delayed diagnoses often involve failure to collect the right data or ask the right questions. When AI tools are used, says Sittig, “clinicians need to be careful and vigilant, and continue using their brains and critical thinking skills to assess whether the computer is right or wrong.”
Sometimes the AI tool makes the correct diagnosis, but the ED clinician either ignores it or does not even see it. This mishap could result in a bad outcome and malpractice claim. “The second problem is that the computer is very good at documenting its findings. Therefore, plaintiffs can often find these computer-generated diagnoses and castigate clinicians for their failures,” warns Sittig.
Plaintiff attorneys will begin using AI tools to help them figure out whether a clinician should have made a diagnosis sooner, predicts Sittig. Regardless of whether the AI tool is right or wrong, there are malpractice implications for ED providers. The plaintiff attorney could argue that the ED provider could have prevented a bad outcome by relying on the diagnosis indicated by the AI tool. Conversely, the attorney could argue that ED providers relied too much on an AI tool that provided an incorrect diagnosis. “Computers are often wrong, especially in these types of cases in which the accuracy and timeliness of the data is often in question. Bad data almost always leads to bad decisions,” observes Sittig. An ED provider may sense that something is wrong with the data just by looking at the patient, adds Sittig.
Sittig recommends these approaches to reduce the risk of AI in healthcare, including the ED setting:
• Hospitals should monitor the performance of AI systems closely post-implementation, in the same way they would for a new EP with admitting privileges.
“Clinicians should review every computer-generated diagnosis, conclusion, alert, or interpretation with the same level of scrutiny as they would a brand-new nurse or resident on their first week in the hospital,” says Sittig. A pharmacy and therapeutics committee is responsible for monitoring the effectiveness of new medications or therapies in hospital settings. Similarly, says Sittig, a committee should carefully review data describing the clinical evaluation of the AI tool and then make a decision about whether they approve or disapprove its use.
• Hospitals should make sure that AI tools were developed using data from patients similar to their patient population.
• Hospitals should ask to see sensitivity and specificity benchmarks for any AI tool they are considering.
“Having locally measured sensitivity and specificity rates would help the hospital in any legal proceedings,” says Sittig. Hospitals can do this by keeping track of the AI tool’s recommendations (such as the suggested diagnosis or suggested tests). Hospitals then can compare it with patients’ final diagnosis or test results. Such data enable ED providers to show why they did, or did not, follow the AI recommendations for a particular patient. “Without the ability to compare AI systems’ performance with that of humans in a prospective manner, plaintiffs can argue whichever side — should have used the AI or shouldn’t have used the AI — helps them the most,” Sittig explains.
The overarching concern for ED providers, and healthcare providers in general, is that it remains to be seen whether AI tools are, on balance, more beneficial or more harmful. “Therefore, in these early days of AI-enabled systems, everyone must be extra careful and particularly vigilant, and constantly aware of what is happening,” says Sittig.
In the ED, overreliance on AI and the abandonment of good clinical judgment “could lead to a medical misadventure,” warns Philip B. Gorelick, MD, MPH, professor in the Davee Department of Neurology at Northwestern University Feinberg School of Medicine. ED clinicians are likely to bear liability for any patient care decisions made — regardless of what any AI tool recommends. “Thus, the clinician must balance their clinical instincts, judgment, and evidence-based understanding of a case in the context of what AI may be recommending,” says Gorelick.
In the ED, there are some unique risks. AI-powered decision tools may lack the “big picture” focus for making critical decisions in real time for ED patients, often without comprehensive patient background. “While AI tools can be useful in collaboration with clinician-guided medical care, the human component must not be lost,” underscores Gorelick. For instance, an early septic patient with borderline vital signs potentially could be flagged as “OK to discharge” based on AI criteria. To the ED provider, the patient may “just not look right.” “An ED physician may have a ‘Spidey sense’ that there is a high likelihood for decompensation in the short term. This acute and deep-seated sense of impending danger is well-known to seasoned ED clinicians — and should not be overlooked,” offers Gorelick.
- Mello MM, Guha N. Understanding liability risk from using health care artificial intelligence tools. N Engl J Med 2024;390:271-278.
- Berg HT, van Bakel B, van de Wouw L, et al. ChatGPT and generating a differential diagnosis early in an emergency department presentation. Ann Emerg Med 2024;83:83-86
- Burkett EL, Todd BR. A novel use of an electronic differential diagnosis generator in the emergency department setting. Cureus 2023;15:e34211.
