Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research, Medical Devices & Health Technology · Depth: Expert, extended

Summary

The study "Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation" by Fudan University and collaborators reveals a significant modality discrepancy and "generalization illusion" in current 3D medical foundation models. Researchers curated the UMD dataset, comprising 490 whole-body PET/CT and 464 whole-body PET/MRI scans, totaling approximately 675,000 2D images and 12,000 3D organ annotations for 13 organs. Through intra-subject controlled comparisons, the evaluation isolated imaging modality as the primary independent variable. The analysis of five models (SAM-Med3D-turbo, SegVol, SAT-Pro, VISTA3D, nnInteractive) showed a stark performance degradation when transitioning from structural (CT/MRI) to functional (PET) domains, indicating these models are far from truly general-purpose and are modality-locked to structural priors.

Key takeaway

For AI Scientists developing 3D medical segmentation models, you must prioritize multi-modal training and evaluation to overcome the observed structural bias. Your current models, despite high reported benchmarks, likely lack true generalization across diverse imaging modalities like PET. Focus on creating truly modality-agnostic systems by incorporating functional imaging data and rigorous intra-subject validation to ensure clinical utility beyond structural priors.

Key insights

Current 3D medical foundation models exhibit a structural bias, failing to generalize effectively from anatomical to functional imaging modalities.

Principles

• Validation must isolate modality discrepancy.
• Structural proficiency does not imply functional understanding.
• Benchmarking needs multi-modal training.

Method

The UMD dataset, with 490 paired PET/CT and 464 PET/MRI scans, enables intra-subject controlled comparisons of 13 organs, isolating imaging modality as the independent variable for rigorous evaluation.

In practice

• Develop multi-modal training strategies.
• Integrate functional imaging data.
• Test models on unseen, diverse modalities.

Topics

• 3D Medical Segmentation
• Foundation Models
• Modality Discrepancy
• PET/CT Imaging
• PET/MRI Imaging
• UMD Dataset
• Generalization Illusion

Code references

• YichiZhang98/UMD

Best for: AI Scientist, Research Scientist, Computer Vision Engineer

Related on AIssential

• A universal foundation model for grounded biomedical image interpretation — UniBiomed unifies diagnostic findings and target segmentation in biomedical images, enhancing AI interpretability for clinical use.
• Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation — Mutual distillation of dual-foundation models improves semi-supervised PET/CT segmentation with minimal labeled data.
• Benchmarks and methods for 3D medical image retrieval — The 3D-MIR benchmark and methods advance medical image retrieval by providing standardized evaluation and multi-modal search strategies.
• Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation — Radiology-specific Vision Foundation Models show promising transferability but face significant hurdles for clinical translation.
• Pan-FM: A Pan-Organ Foundation Model with Saliency-Guided Masking for Missing Robustness — Pan-FM is a pan-organ foundation model robust to missing data via saliency-guided masking.
• HadBalance: A Plug-and-Play Unified Global Geometric Prior Framework for Generalizable Biomedical Segmentation — HadBalance unifies geometric priors for biomedical segmentation, adapting to shape heterogeneity via conflict-aware gradient balancing.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.