Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control

2026-04-29 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Medical Devices & Health Technology · Depth: Advanced, quick

Summary

A new study benchmarks the safety of 72 large language models (LLMs) when used as control components for robotic health attendants. Researchers introduced a dataset of 270 harmful instructions across nine categories, based on American Medical Association ethical principles, and evaluated LLMs in a simulation environment. The average violation rate across all models was 54.4%, with over half exceeding 50%. Proprietary models demonstrated significantly higher safety (median 23.7% violation rate) compared to open-weight models (median 72.8%). Model size and release date were key factors for open-weight model safety. Medical domain fine-tuning offered no significant safety improvement, and prompt-based defenses provided only a modest reduction in violation rates, indicating current LLMs are not yet safe for clinical deployment.

Key takeaway

For CTOs and VPs of Engineering evaluating LLMs for healthcare robotics, recognize that current models, especially open-weight ones, have unacceptably high safety violation rates (median 72.8%). You must prioritize rigorous, domain-specific safety benchmarking as a core development criterion, as neither medical fine-tuning nor basic prompt defenses significantly mitigate risks for clinical deployment.

Key insights

Current LLMs exhibit high safety violation rates as robotic health attendant controllers, precluding clinical deployment.

Principles

• Safety evaluation is a first-class criterion.
• Proprietary LLMs are safer than open-weight models.

Method

A dataset of 270 harmful instructions, grounded in AMA ethics, was used to evaluate 72 LLMs in a Robotic Health Attendant simulation environment.

In practice

• Prioritize safety evaluation for LLM-controlled robotics.
• Consider proprietary LLMs for safety-critical applications.

Topics

• Large Language Models
• Robotic Health Attendants
• LLM Safety Benchmarking
• Medical Ethics
• Harmful Instructions Dataset

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, AI Security Engineer, Robotics Engineer

Related on AIssential

• Safety and accuracy follow different scaling laws in clinical large language models — Clinical LLM safety is a deployment property, not a passive consequence of scaling model size or compute.
• Class of LLMs: Benchmarking Large Language Models on the Brazilian National Medical Examination — Generalist LLMs often outperform specialized medical fine-tunes on high-stakes, language-specific medical exams.
• When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations — Large Language Models, including medical-specific variants, are critically fragile to prompt variations in healthcare.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• Americans ask AI for health care. Hospitals think the answer is more chatbots. — Health systems are deploying branded AI chatbots despite unproven patient outcomes and significant accuracy concerns.
• Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions — LLMs provide useful repair assistance but are unreliable for high-risk real-world tasks without rigorous evaluation and explicit safety safeguards.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.