When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations

2026-06-05 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Medical Devices & Health Technology · Depth: Advanced, quick

Summary

A study evaluated the robustness of Large Language Models (LLMs) in healthcare, revealing their significant sensitivity to subtle prompt variations. Researchers conducted a systematic sensitivity analysis on general-purpose models like GPT-3.5 and Llama3, alongside medical-specific LLMs such as ClinicalBERT, BioLlama3, and BioBERT, using the MedMCQA benchmark. The analysis categorized perturbations into natural and adversarial types, examining their impact on model consistency, accuracy, and reliability in clinical reasoning tasks. Findings indicate that medical LLMs are not intrinsically safe; even minor phrasing changes can alter clinical advice, and targeted adversarial prompts can lead to harmful outputs, including incorrect dosages or omitted critical findings. This fragility, particularly under syntactic reordering or misleading contextual cues, is evident across both general-purpose and domain-specific models, highlighting unacceptable unpredictability in high-stakes clinical settings.

Key takeaway

For AI Scientists and clinicians deploying LLMs in healthcare, you must recognize that current models, even specialized ones, are not intrinsically safe. Their extreme sensitivity to minor prompt variations, especially syntactic reordering, can lead to altered clinical advice or dangerous outputs like incorrect dosages. You should prioritize robust prompt engineering and integrate rigorous adversarial testing into your development and deployment pipelines to mitigate patient safety risks.

Key insights

Large Language Models, including medical-specific variants, are critically fragile to prompt variations in healthcare.

Principles

• LLM robustness is paramount for safety-critical applications.
• Syntactic reordering and contextual cues degrade LLM reliability.
• Adversarial prompts can induce clinically dangerous outputs.

Method

Conduct systematic sensitivity analysis on LLMs using benchmarks like MedMCQA, evaluating consistency, accuracy, and reliability against natural and adversarial prompt perturbations.

In practice

• Test LLMs for sensitivity to syntactic reordering.
• Implement adversarial prompt testing in LLM evaluation.
• Validate LLM clinical advice with human experts.

Topics

• Large Language Models
• Healthcare AI
• Prompt Sensitivity
• Model Robustness
• Adversarial Prompts
• Clinical Reasoning

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, AI Security Engineer, Research Scientist

Related on AIssential

• When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations — Healthcare LLMs are not intrinsically safe, exhibiting high sensitivity to prompt variations that can lead to harmful clinical outputs.
• Safety and accuracy follow different scaling laws in clinical large language models — Clinical LLM safety is a deployment property, not a passive consequence of scaling model size or compute.
• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• Linguistic Symptoms: Augmented Generation by Retrieval and Reasoning in LLMs under Portuguese-English Variation in Medical Contexts — Linguistic variation significantly impacts smaller LLMs in medical contexts, with larger models showing greater cross-language robustness.
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance — LLM annotation reliability hinges on definition alignment, not just text memorization, as prompt-based corrections are largely ineffective.
• From `May' to `Is': Certainty Distortion in Language Model Rewriting — Language Models systematically distort expressed certainty, often inflating it, especially in high-stakes domains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.