Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation

2025-05-11 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Medical AI · Depth: Expert, extended

Summary

Researchers introduce ViMed-PET, a novel Vietnamese-language multimodal medical dataset and benchmark designed to advance Vision-Language Foundation Models (VLMs) in medical imaging. This dataset comprises 1,567,062 paired CT-PET images and 2,757 full-length Vietnamese clinical reports, addressing critical gaps in PET/CT imaging data and low-resource language representation. The study also proposes a training framework incorporating data augmentation and expert-validated test sets. Benchmarking state-of-the-art VLMs on tasks like medical report generation and visual question answering demonstrates significant performance improvements when using ViMed-PET. However, models still show limitations in clinically critical areas, such as lung cancer diagnosis, indicating further research is needed.

Key takeaway

For AI Scientists and ML Engineers developing medical Vision-Language Models, you should prioritize integrating specialized, multimodal datasets like ViMed-PET, particularly for functional imaging modalities such as PET/CT and low-resource languages. Relying solely on general-domain or English-centric models will yield inadequate clinical performance. Leverage expert-validated clinical metrics, not just standard NLP scores, to accurately assess model utility and drive the development of more robust and equitable healthcare AI solutions.

Key insights

A new Vietnamese PET/CT dataset and clinical benchmarks significantly enhance medical VLM performance, especially for low-resource languages.

Principles

• Medical VLMs require domain-specific data for functional imaging modalities.
• Low-resource languages need dedicated multimodal datasets for equitable AI development.
• Clinical accuracy demands specialized evaluation metrics beyond standard NLP scores.

Method

The method involves a three-stage fine-tuning pipeline: adapting 3D vision encoders, aligning visual and textual embeddings via a linear projection layer, and instruction-tuning the VLM using Low-Rank Adaptation (LoRA) on augmented VQA and report generation datasets.

In practice

• Fine-tune VLMs with domain-specific PET/CT data for improved medical report generation.
• Integrate CT-ViT as a vision encoder for enhanced clinical accuracy in 3D medical imaging.
• Utilize GPT-4o for data augmentation and structuring clinical evaluation metrics.

Topics

• Vision-Language Models
• Medical Imaging
• PET/CT
• Vietnamese Language
• Medical Report Generation
• Clinical AI Benchmarks

Best for: NLP Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation — Radiology-specific Vision Foundation Models show promising transferability but face significant hurdles for clinical translation.
• OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models — Broad open pretraining on diverse medical data enables SOTA performance with smaller models.
• Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data? — Raw, heterogeneous medical data standardization is a critical, unaddressed bottleneck for real-world VLM applications in clinical AI.
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography — The framework improves 3D CT diagnosis by integrating hybrid visual encoding, disease-level contrastive learning, and diagnosis-aware prompting.
• MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry — MetaDent provides a novel dataset and benchmarks to advance VLM capabilities in fine-grained intraoral image analysis.
• Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology — A multi-LLM collaborative system generates accurate 3D MRI reports for brain oncology, overcoming data scarcity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.