MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

2026-07-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Medical Devices & Health Technology · Depth: Expert, quick

Summary

MedRealMM is a new large-scale multimodal benchmark designed for evaluating large language models in Chinese online medical consultation. Developed from de-identified patient-doctor interactions at a nationwide Chinese internet hospital, it addresses the poor alignment of existing benchmarks with real clinical practice, which often rely on synthetic data or lack medical images. MedRealMM employs a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments, converting them into standardized next-response generation tasks while preserving text-image context. Each of the 5,620 real-world multimodal cases, spanning 64 clinical departments, includes a physician-refined rubric to assess clinical quality. Evaluations of 19 general-purpose and medical-specialized LLMs revealed that image information is critical for reliable performance, and while frontier models meet positive clinical criteria, they exhibit more negative criteria, highlighting safety-sensitive error avoidance as a key bottleneck. The dataset will be publicly available on Hugging Face.

Key takeaway

For AI scientists and ML engineers developing medical LLMs for online consultation, MedRealMM highlights critical safety gaps. Your models must integrate image information effectively, as it is crucial for reliable clinical performance. Prioritize rigorous error avoidance, particularly for safety-sensitive responses, even if your models satisfy positive clinical criteria. This benchmark indicates that current frontier models still fall short of physician-level safety, necessitating focused development on robust, multimodal clinical reasoning.

Key insights

MedRealMM provides a real-world, multimodal benchmark revealing LLMs' safety gaps in Chinese online medical consultation, especially regarding image integration.

Principles

• Real-world data improves benchmark relevance.
• Multimodal input is critical for clinical reliability.
• Physician-refined rubrics enhance evaluation quality.

Method

MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) framework to extract demanding moments from authentic consultations, converting them into next-response generation tasks with text-image context and physician-refined rubrics.

In practice

• Integrate medical images into LLM training.
• Prioritize safety-critical error reduction in models.
• Use physician feedback for clinical response rubrics.

Topics

• Multimodal LLMs
• Medical Consultation
• Benchmark Datasets
• Clinical AI Safety
• Chinese Healthcare Data
• Online Health Platforms

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation — Real-world multimodal data and physician-refined rubrics are essential for robustly benchmarking medical LLMs in clinical practice.
• MedAct: Removing the Human Bottleneck in Benchmarking Clinical LLM Safety — MedAct provides a scalable, open-source method for action-level clinical LLM safety benchmarking using synthetically generated cases.
• MedRLM: Recursive Multimodal Health Intelligence for Long-Context Clinical Reasoning, Sensor-Guided Screening, Evidence-Grounded Decision Support, and Community-to-Tertiary Referral Optimization — MedRLM enables recursive, multimodal clinical reasoning by treating patient data as an inspectable external environment.
• Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models — Multimodal medical models often lack true image comprehension despite strong performance on specific vision-language tasks.
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences — Real-world clinician preferences reveal that medical LLM utility extends beyond factual recall to include depth, clarity, and clinical nuance.
• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.