MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences

2026-03-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI in Healthcare · Depth: Intermediate, short

Summary

MedArena is an interactive evaluation platform designed to compare large language models (LLMs) for medical applications based on real-world clinician preferences, addressing limitations of static benchmarks. The platform allows clinicians to submit their own medical queries and then select a preferred response from two randomly chosen LLM outputs. As of November 1, 2025, MedArena collected 1571 preferences across 12 LLMs, with Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o emerging as the top three models by Bradley-Terry rating. The study found that only one-third of clinician questions were factual recall tasks, while the majority involved treatment selection, clinical documentation, or patient communication, with approximately 20% being multi-turn conversations. Clinicians prioritized depth, detail, and clarity over raw factual accuracy, indicating the importance of readability and clinical nuance in medical LLM responses.

Key takeaway

For AI Engineers developing medical LLMs, traditional factual recall benchmarks are insufficient for assessing real-world clinical utility. You should focus on optimizing models for depth, detail, and clarity in responses, as these factors are crucial for clinician preference in tasks like treatment selection and patient communication. Incorporate interactive evaluation methods like MedArena to gather direct clinician feedback and refine model performance beyond mere accuracy.

Key insights

Real-world clinician preferences reveal that medical LLM utility extends beyond factual recall to include depth, clarity, and clinical nuance.

Principles

• Static benchmarks fail to capture real-world clinical utility.
• Clinician preference prioritizes response quality over raw accuracy.

Method

MedArena uses an interactive platform where clinicians submit queries and select preferred LLM responses from a pair, enabling direct comparison and preference collection for medical LLMs.

In practice

• Evaluate medical LLMs using interactive, clinician-driven platforms.
• Prioritize response depth, detail, and clarity in medical LLM development.

Topics

• Medical LLMs
• LLM Evaluation
• Clinician Preferences
• MedArena Platform
• Clinical Decision Support

Best for: AI Engineer, NLP Engineer, AI Scientist, AI Researcher, Machine Learning Engineer, Domain Expert

Related on AIssential

• When eight popular large language models (LLMs) answer common parent questions about early maxillary expansion, how reliable are they—and how readable is what they say? — LLM performance in healthcare varies significantly across accuracy, comprehensiveness, and readability, with no single model excelling in all.
• MedCTA: A Benchmark for Clinical Tool Agents — MedCTA reveals current medical AI agents struggle with multi-step clinical tool use despite strong perception, highlighting a need for better agentic reliability.
• MediX-R1: Open Ended Medical Reinforcement Learning — Open-ended RL with composite rewards and LLM-based evaluation improves medical multimodal model reasoning.
• Benchmarking GPT Models for Conversational AI Systems: Can AI Read a Doctor’s Notes? — LLMs struggle with medical abbreviation expansion, especially context-dependent ones, despite larger models performing better.
• ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents — ClinEnv reveals LLMs struggle with sequential, uncertain clinical decisions and information gathering in a multi-stage environment.
• A domain-adapted large language model to support clinicians in psychiatric clinical practice — Domain-adapted LLMs can significantly enhance psychiatric clinical workflows, improving accuracy and efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.