MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

2026-07-10 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI in Healthcare · Depth: Expert, quick

Summary

MedRealMM is a new large-scale multimodal benchmark designed for evaluating large language models in Chinese online medical consultation, addressing limitations of existing synthetic benchmarks. Developed from 5,620 de-identified patient-doctor interactions across 64 clinical departments from a nationwide Chinese internet hospital, MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) framework to create standardized next-response generation tasks with preserved text-image context. Each instance includes a physician-refined rubric to assess clinical quality, rewarding desirable behaviors and penalizing unsafe or contradictory responses. Evaluations of 19 general-purpose and medical-specialized LLMs revealed that image information is crucial for reliable clinical performance. While some frontier models met positive clinical criteria comparable to physicians, they exhibited more negative criteria, highlighting safety-sensitive error avoidance as a key bottleneck. The dataset will be publicly available on Hugging Face.

Key takeaway

For Machine Learning Engineers developing or deploying medical LLMs, recognize that current frontier models, while achieving some positive clinical criteria, still exhibit significant safety-sensitive errors. You should prioritize integrating real-world multimodal data, especially images, into your model training and evaluation pipelines. Focus your fine-tuning efforts on robust error avoidance and leverage physician-refined rubrics to ensure clinical safety and reliability, as image information is critical for performance.

Key insights

Real-world multimodal data and physician-refined rubrics are essential for robustly benchmarking medical LLMs in clinical practice.

Principles

• Benchmarks need real-world clinical alignment.
• Multimodal data is critical for clinical LLMs.
• LLM safety error avoidance remains a bottleneck.

Method

MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) framework to extract clinically demanding moments from de-identified patient-doctor interactions, converting them into standardized next-response generation tasks with text-image context and physician-refined rubrics.

In practice

• Incorporating image data for medical LLM training.
• Prioritizing LLM safety and error avoidance.
• Using physician-refined rubrics for evaluation.

Topics

• MedRealMM
• Multimodal LLMs
• Online Medical Consultation
• Clinical Benchmarking
• LLM Safety
• Chinese Internet Hospital

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation — MedRealMM provides a real-world, multimodal benchmark revealing LLMs' safety gaps in Chinese online medical consultation, especially regarding image integration.
• MedAct: Removing the Human Bottleneck in Benchmarking Clinical LLM Safety — MedAct provides a scalable, open-source method for action-level clinical LLM safety benchmarking using synthetically generated cases.
• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences — Real-world clinician preferences reveal that medical LLM utility extends beyond factual recall to include depth, clarity, and clinical nuance.
• Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models — Multimodal medical models often lack true image comprehension despite strong performance on specific vision-language tasks.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.