Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models

2026-05-21 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Vision · Depth: Expert, medium

Summary

Director-Experts (DEX) is a novel modular network designed to address the challenge of Non-IID feature statistics in multi-modality medical vision (MV) foundation models. Traditional monolithic self-supervised optimization on heterogeneous medical imaging data often causes representations to collapse due to conflicting gradients. DEX reframes this issue as an imbalance between specialization and coordination, proposing a stacked modular architecture that explicitly regulates these dynamics. Each DEX module incorporates a pool of experts, dynamically adapted by an image-wise activation strategy, which specialize in modality-dominant statistics. Concurrently, a director, updated via a group exponential moving average, distills multi-expert knowledge into a shared space for semantic integration across modalities, fostering emergent modular representations. The researchers curated a new benchmark, Medical Vision Universe, comprising over 4 million images across 10 modalities, providing extensive pre-training data for DEX. Evaluations across 26 downstream tasks demonstrated enhanced optimization behavior and transferability, positioning DEX as a significant advancement toward general-purpose multi-modality medical AI.

Key takeaway

For AI Scientists developing multi-modality medical foundation models, you should consider adopting modular architectures like Director-Experts (DEX). Your current monolithic approaches likely face representation collapse due to heterogeneous data. DEX's explicit regulation of specialization and coordination, demonstrated on over 4 million images, provides a path to improved optimization and transferability. Explore its open-sourced code and the Medical Vision Universe benchmark to enhance your model's general-purpose capabilities.

Key insights

DEX uses modular networks with experts and a director to manage modality-specific and shared representations in multi-modality medical FMs.

Principles

• Non-IID features challenge monolithic self-supervised learning.
• Balancing specialization and coordination prevents representation collapse.
• Modular architectures can regulate emergent dynamics.

Method

DEX employs stacked modules, each with dynamically adapted experts for modality-dominant statistics and a director using group exponential moving average to distill multi-expert knowledge into a shared semantic space.

In practice

• Apply DEX to pre-train multi-modality medical FMs.
• Utilize the Medical Vision Universe benchmark for evaluation.
• Develop image-wise activation for expert adaptation.

Topics

• Multi-modality Medical Vision
• Foundation Models
• Director-Experts
• Modular Representations
• Self-supervised Learning
• Medical Vision Universe

Code references

• YutingHe-list/DEX
• ricklisz/MedDINOv3
• showlab/Code2Video
• uni-medical/UniMedVL

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models — Regulating specialization and coordination in modular networks overcomes Non-IID challenges in multi-modality medical vision foundation models.
• DaX: Learning General Pathology Representations Across Scales — DaX is a pathology vision foundation model using DINOv3-style self-supervised learning for robust, transferable representations across diverse histopathology tasks.
• Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI — Explicitly modeling intra- and cross-modal interactions is crucial for maximizing the utility of multimodal 3D brain MRI.
• Towards generalizable AI in medicine via Generalist–Specialist Collaboration — Combining generalist and specialist AI models improves medical diagnosis and report generation.
• A universal foundation model for grounded biomedical image interpretation — UniBiomed unifies diagnostic findings and target segmentation in biomedical images, enhancing AI interpretability for clinical use.
• BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning — Large-scale slice-wise self-supervised learning yields a unified, transferable brain MRI representation for diverse neuroimaging tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.