Real-world safety and harms from patient-facing LLMs

2026-04-06 · Source: Ehud Reiter's Blog · Field: Health & Wellbeing — Healthcare Systems & Policy, Mental Health & Psychological Support, Medical Devices & Health Technology · Depth: Intermediate, medium

Summary

An analysis of real-world harms caused by AI chatbots in health contexts identifies three primary data sources: incident reporting, clinical trials, and data from health providers. While no government mandates exist for chatbot incident reporting, platforms like HumanLine and IncidentDatabase collect user-submitted stories, alongside peer-reviewed case reports. Clinical trials, though often relying on simulated patients, are beginning to include real-patient studies, such as Google's AIME system trial which monitored 100 patients and found minor hallucinations in 3 cases. Data from health providers, exemplified by a Danish study of 1.5 million psychiatric patients, revealed 38 cases of chatbot-induced harm, primarily consolidating delusions or reinforcing mania, affecting less than 0.1% of users. The analysis notes similarities between chatbot harms and those from social media misinformation and overuse, suggesting that while chatbots can be engineered for safety, better data and mandated reporting are crucial.

Key takeaway

For healthcare leaders evaluating AI chatbot integration, recognize that while current real-world harm data is limited, it indicates potential mental health risks like delusion consolidation. You should advocate for government-mandated incident reporting and explore adapting existing product safety mechanisms for chatbots to ensure better safety monitoring. Prioritize solutions that offer transparent safety data and actively work to engineer bots that reduce misinformation, rather than abandoning their potential benefits.

Key insights

Real-world data on AI chatbot harms in health is limited but growing, highlighting mental health risks.

Principles

• Real patient data is crucial for assessing health chatbot harms.
• Misinformation and overuse are common risks across digital health tools.

Method

Data on real-world harms from health chatbots can be gathered via incident reporting platforms, real-patient clinical trials, and analysis of health provider records, such as psychiatric clinical notes.

In practice

• Explore HumanLine or IncidentDatabase for reported AI harm cases.
• Review clinical notes for patterns of chatbot-related patient harm.

Topics

• Patient-facing LLMs
• Real-world Harms
• Incident Reporting
• Clinical Trials
• Health Provider Data

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, Research Scientist

Related on AIssential

• ‘Unbelievably dangerous’: experts sound alarm after ChatGPT Health fails to recognise medical emergencies — ChatGPT Health frequently misdiagnoses medical emergencies and suicidal ideation, posing significant patient safety risks.
• AI Chatbots in Mental Health: Comfort Is Not the Same as Improvement — AI chatbots are rapidly becoming common tools for mental health support, offering "therapy-adjacent support and companionship" but raising critical concerns that comfort is being prioritized over genuine improvement.
• When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI — Patient-facing RAG chatbots can expose critical privacy and security vulnerabilities through client-side communication.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.
• Users of social media and AI chatbots for health information are more likely to say they are convenient than accurate — Digital health information from social media and AI chatbots is valued for convenience and understandability, not accuracy.
• TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots — TherapyGym evaluates and aligns therapy chatbots using clinical fidelity and safety metrics, improving performance through RL with expert-validated feedback.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ehud Reiter's Blog.