MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Clinical Care & Medical Practice · Depth: Expert, quick

Summary

MMIR-TCM is a novel framework improving Traditional Chinese Medicine (TCM) diagnosis, particularly tongue inspection. It addresses subjectivity, reproducibility, the semantic gap between visual and textual data, and limited large-scale datasets. The framework emulates TCM expert diagnostic processes by integrating a multimodal large language model (MLLM) with memory-augmented segmentation and retrieval-augmented generation (RAG). Its three-stage architecture includes a training-free Memory-SAM for robust tongue extraction and a fine-tuned Qwen3-VL for structured diagnosis. A Qwen3-based RAG component provides evidence-grounded clinical decision support. Developed with the new MedTCM dataset and evaluated using the TDEU metric, MMIR-TCM significantly outperforms GPT-4o and Gemini 2.5 Flash.

Key takeaway

For AI Scientists developing clinical decision support systems, MMIR-TCM offers a robust approach to integrating multimodal data and expert reasoning. You should consider adopting memory-augmented segmentation and retrieval-augmented generation with fine-tuned MLLMs. This helps bridge semantic gaps and ensures evidence-grounded outputs, especially with complex visual and textual medical data. The framework highlights the necessity of creating domain-specific datasets and evaluation metrics for accurate clinical validation.

Key insights

MMIR-TCM integrates MLLM with memory-augmented segmentation and RAG to enhance TCM clinical decision support.

Principles

• Multimodal integration bridges semantic gaps.
• Memory-augmented segmentation improves visual feature extraction.
• Domain-specific metrics are crucial for clinical evaluation.

Method

MMIR-TCM employs a three-stage process: Memory-SAM for tongue extraction, fine-tuned Qwen3-VL for structured diagnosis generation, and Qwen3-based RAG for evidence-grounded clinical decision support.

In practice

• Consider Memory-SAM for robust image segmentation.
• Fine-tune MLLMs like Qwen3-VL for specific tasks.
• Develop custom evaluation metrics for domain accuracy.

Topics

• Traditional Chinese Medicine
• Multimodal AI
• Clinical Decision Support
• Large Language Models
• Retrieval-Augmented Generation
• Image Segmentation
• MedTCM Dataset

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents — ClinEnv reveals LLMs struggle with sequential, uncertain clinical decisions and information gathering in a multi-stage environment.
• PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs — PathMem integrates structured pathology knowledge into MLLMs via a Memory Transformer for improved diagnostic reasoning.
• Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings — LLM embeddings from clinical records can predict post-traumatic epilepsy with high accuracy.
• Psychological Digital Twins : Conceptual Models, AI Foundations and Ethical Challenges — Robust Psychological Digital Twins require hybrid AI, combining LLM flexibility with symbolic guardrails for precision mental health.
• MedRLM: Recursive Multimodal Health Intelligence for Long-Context Clinical Reasoning, Sensor-Guided Screening, Evidence-Grounded Decision Support, and Community-to-Tertiary Referral Optimization — MedRLM recursively processes heterogeneous patient data as an external clinical environment for auditable, workflow-aware decision support.
• Case-Aware Medical Image Classification with Multimodal Knowledge Graphs and Reliability-Guided Refinement — Medical image classification improves by integrating multimodal knowledge graphs from similar cases with reliability-guided refinement.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.