A Patient Simulation Framework for Risk Assessment of Conversational Healthcare AI: Evaluation of an Antidepressant Decision Aid

2026-02-11 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Healthcare AI · Depth: Expert, quick

Summary

A new patient simulation framework has been developed and validated for assessing risks in conversational healthcare AI, aligning with the National Institute of Standards and Technology (NIST) AI Risk Management Framework's MAP and MEASURE functions. This framework empirically identifies and characterizes performance risks across medical, linguistic, and behavioral patient variations. Applied to a conversational decision aid for antidepressant selection in major depressive disorder, the simulator integrates medical profiles from All of Us electronic health records, linguistic profiles modeling health literacy and communication styles, and behavioral profiles representing cooperative, distracted, or adversarial engagement. After generating 500 simulated conversations, the framework demonstrated high fidelity for medical concepts (96.6% accurate across 8,210 concepts) with human inter-annotator agreement of 0.73 κ and LLM-judge agreement of 0.78 κ. It revealed a monotonic degradation in the AI Decision Aid's performance, with rank-1 concept retrieval dropping from 81.9% for proficient health literacy to 47.6% for limited health literacy, alongside declines in antidepressant recommendation accuracy.

Key takeaway

For AI Scientists developing conversational healthcare AI, particularly decision aids, you must integrate robust patient simulation frameworks into your risk assessment. This study demonstrates that patient variations, especially health literacy, critically impact AI performance, with concept retrieval dropping from 81.9% to 47.6% for limited literacy. Prioritize testing across diverse medical, linguistic, and behavioral profiles to identify and mitigate performance degradation before clinical deployment, ensuring equitable and safe AI outcomes.

Key insights

A patient simulation framework effectively identifies conversational healthcare AI risks by modeling diverse patient profiles.

Principles

• Patient variation significantly impacts conversational AI performance.
• Simulating diverse patient profiles is crucial for AI risk assessment.
• Health literacy directly correlates with AI decision aid accuracy.

Method

The simulator integrates medical, linguistic, and behavioral patient profiles, generating conversations for AI evaluation. Fidelity is assessed via human and LLM judges.

In practice

• Construct medical profiles from EHRs using risk-ratio gating.
• Model health literacy gradients in linguistic profiles.
• Evaluate AI performance across cooperative, distracted, adversarial behaviors.

Topics

• Patient Simulation
• Conversational Healthcare AI
• AI Risk Management
• NIST AI RMF
• Health Literacy
• Antidepressant Decision Aids

Best for: AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• Benchmarking GPT Models for Conversational AI Systems: Can AI Read a Doctor’s Notes? — LLMs struggle with medical abbreviation expansion, especially context-dependent ones, despite larger models performing better.
• A randomized controlled trial of artificial intelligence-based analytics for clinical deterioration — Passive AI-based predictive analytics displays did not significantly improve patient outcomes in a clinical trial.
• Rethinking Patient Education as Multi-turn Multi-modal Interaction — Patient education requires multi-turn, multimodal interaction with robust visual grounding and safety considerations.
• Exploring the feasibility of conversational diagnostic AI in a real-world clinical study — Supervised conversational AI for pre-visit history taking is feasible, safe, and well-received by patients and clinicians.
• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows — AIPatient Arena reveals LLMs need process-based evaluation beyond final answers for clinical readiness, highlighting interaction failures.
• Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task — Skill-augmented AI agents showed a directional, non-confirmatory quality improvement in medical research analysis tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.