- Moderator: Katie Palmer, health tech correspondent, STAT News
- Curt Langlotz, M.D., professor of radiology, medicine, and biomedical data science and director of the Center for Artificial Intelligence in Medicine and Imaging (AIMI Center) at Stanford University
- Paul Lukac, M.D., chief AI officer, UCLA Health
- Melissa Wong, M.D., director, Informatics & Clinical Artificial Intelligence, Enterprise Information Services; assistant professor of Obstetrics & Gynecology, Division of Maternal-Fetal Medicine – Cedars-Sinai Health Sciences University
With more than a thousand FDA-authorized artificial intelligence (AI) tools entering health care, the technology feels revolutionary. But inside hospital halls, some clinicians describe a more nuanced reality.
That gap was the central focus of a panel featuring physicians who work on AI projects at leading health care organizations at AHCJ’s fall summit in Los Angeles. Central to the discussion was one question: How do we know if medical AI works in practice?
Doctors recounted how earlier AI models generated frequent false positives, causing alert fatigue and reduced staff buy-in.
“Once an alert has triggered for me wrong twice, I’m ignoring that alert forever,” said Melissa Wong, M.D., director of clinical AI at Cedars-Sinai Health Sciences University.
Wong added that AI models can be powerful but, based on her hospital’s experience with preeclampsia prevention for pregnant women, sometimes lower-tech tools outperform AI simply because doctors actually use them.
Look beyond the hype
Panelists urged health journalists to look beyond vendor claims and FDA clearances to examine how tools perform on the frontlines of patient care.
Curt Langlotz, M.D., Ph.D., director of Stanford University’s Artificial Intelligence in Medicine and Imaging Center, said that traditional regulatory frameworks are inadequate to keep up with AI tools, which researchers describe as “too broad, adaptive and opaque.”
“I think the people at the FDA are doing an outstanding job under a regulatory framework that was built 50 years ago for the realm of fax machines and paper records and physical devices,” Langlotz said.
Roughly three-quarters of FDA-authorized AI devices are in radiology, including generative AI tools that draft radiology reports and summaries. Langlotz likened these tools to a helpful “e-resident” that can enhance productivity, but errors in automated reports remain a concern, making ongoing post-market evaluation essential to ensure that an AI tool “behaves on day 100 the same way it behaved on day 1,” he said.
“[The AI systems] hallucinate. They make stuff up,” Langlotz said, noting that busy clinicians may lack the time to verify automated summaries against massive medical records.
Still, he remains optimistic as FDA regulators — some of whom are radiologists — have started asking the right questions about how to properly evaluate AI devices. The FDA has also prioritized generative AI in its upcoming 2027 guidances.
Patient-facing chatbots
Patient-facing AI chatbots are another aspect of the technology that’s introducing unexpected dynamics into clinical communication.
Paul Lukac, M.D., chief AI officer at the University of California in Los Angeles Health, said that patients made more than 10 million API calls — a digital message sent from one software program to another asking it to provide data or perform an action — connecting ChatGPT directly to UCLA medical records in just two months. Wong described how clinic inboxes are overflowing with messages from patients using ChatGPT regarding ultrasound results and other records. To manage the influx, some health systems have turned to automated response technology, where AI drafts a reply for the clinician to review and approve before sending.
“So now our AIs are literally just talking to our AIs,” Wong said.
The full recordings of this panel discussion and other fall summit coverage assets are available on our website.
