‘Unbelievably dangerous’: experts sound alarm after ChatGPT Health fails to recognise medical emergencies

2026-02-26 · Source: AI (artificial intelligence) | The Guardian · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Safety · Depth: Intermediate, medium

Summary

A study published in the February edition of "Nature Medicine" found that OpenAI's ChatGPT Health, launched in January 2026, frequently fails to recognize medical emergencies and suicidal ideation, potentially leading to significant harm. Researchers created 60 realistic patient scenarios, reviewed by three independent doctors, and generated nearly 1,000 responses from ChatGPT Health under varied conditions. The platform under-triaged 51.6% of cases requiring immediate hospital care, advising patients to wait or book routine appointments. In one simulation, it sent a suffocating woman to a future appointment 84% of the time. Additionally, it was nearly 12 times more likely to downplay symptoms if a "friend" suggested they were not serious and failed to display crisis intervention banners for suicidal ideation when normal lab results were added to the prompt.

Key takeaway

For AI Product Managers developing healthcare applications, this study highlights critical safety gaps in current large language models. Your teams must prioritize rigorous, context-sensitive validation and implement fail-safe mechanisms for emergency detection, especially concerning mental health crises. Over-reliance on AI without robust guardrails and independent oversight could lead to severe patient harm and significant legal liability.

Key insights

ChatGPT Health frequently misdiagnoses medical emergencies and suicidal ideation, posing significant patient safety risks.

Principles

• AI medical advice requires robust safety standards.
• Contextual cues heavily influence AI diagnostic accuracy.

Method

Researchers created 60 patient scenarios, validated by doctors, then queried ChatGPT Health under varied conditions (e.g., gender, test results, family comments) to compare AI recommendations against expert assessments.

In practice

• Do not rely on AI chatbots for urgent medical advice.
• Implement independent auditing for AI healthcare tools.

Topics

• ChatGPT Health
• AI Medical Diagnosis
• AI Safety Evaluation
• Medical AI Ethics
• Suicidal Ideation Detection

Best for: CTO, AI Scientist, AI Product Manager, AI Ethicist, Policy Maker, Research Scientist

Related on AIssential

• Real-world safety and harms from patient-facing LLMs — Real-world data on AI chatbot harms in health is limited but growing, highlighting mental health risks.
• Universal Medical Intelligence: OpenAI's Plan to Elevate Human Health, with Karan Singhal — OpenAI's ChatGPT Health achieves physician-level performance, integrating AI into global medical practice with robust safety and privacy.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.
• Google puts users at risk by downplaying health disclaimers under AI Overviews — Google's AI Overviews downplay health disclaimers, risking user harm from AI-generated medical advice.
• Helping ChatGPT better recognize context in sensitive conversations — ChatGPT now uses contextual cues and safety summaries to better recognize and respond to emerging risks in sensitive conversations.
• Florida AG announces investigation into OpenAI over shooting that allegedly involved ChatGPT — Florida's Attorney General is investigating OpenAI over ChatGPT's alleged role in a deadly shooting and broader "AI psychosis" concerns.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI (artificial intelligence) | The Guardian.