Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models

2025-02-15 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Medical Vision AI · Depth: Expert, extended

Summary

Director–Experts (DEX) is a novel modular network designed to address the Non-IID feature statistics challenge in multi-modality medical vision (MV) foundation models (FMs), which typically cause representation collapse. DEX regulates specialization and coordination using stacked modules: a pool of experts, dynamically adapted by image-wise activation, specializes in modality-dominant statistics, while a director, updated via group exponential moving average (GEMA), distills multi-expert knowledge into a shared semantic space. The model was pretrained on the new Medical Vision Universe (MedVerse) benchmark, comprising over 4 million 2D medical images across 10 modalities. Evaluations on 26 downstream tasks demonstrated DEX's superior optimization and transferability, achieving the highest average performance of 78.4% and ranking in the top-3 across 9 out of 10 modalities. It also showed strong data efficiency, matching baselines with 40% less fine-tuning data.

Key takeaway

For AI Scientists and Machine Learning Engineers developing general-purpose multi-modality medical AI, you should consider modular network designs like Director–Experts (DEX). Traditional monolithic foundation models struggle with the Non-IID nature of diverse medical imaging data, leading to representation collapse. DEX's explicit regulation of specialization and coordination, coupled with large-scale pretraining on datasets like MedVerse, offers a robust path to improved transferability and balanced performance across heterogeneous medical tasks.

Key insights

Regulating specialization and coordination in modular networks overcomes Non-IID challenges in multi-modality medical vision foundation models.

Principles

• Non-IID medical data causes representation collapse.
• Emergent modularity requires balancing specialization and coordination.
• Progressively strengthen alignment in deeper network layers.

Method

DEX employs stacked modules with image-wise activated experts for specialization and a GEMA-updated director for coordination, optimized via self-supervised, alignment, and balance losses.

In practice

• Curate large-scale, diverse multi-modality datasets for pretraining.
• Implement image-wise expert activation for high-resolution medical data efficiency.
• Apply layer-wise alignment factors to enhance shared representation learning.

Topics

• Medical Vision Foundation Models
• Modular Networks
• Non-IID Data
• Representation Learning
• Director-Experts
• Multi-modality Imaging
• Medical Vision Universe

Code references

• YutingHe-list/DEX

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models — DEX uses modular networks with experts and a director to manage modality-specific and shared representations in multi-modality medical FMs.
• Representation learning to advance multi-institutional studies with electronic health record data from US and France — A graph-based framework harmonizes heterogeneous EHR data across institutions using representation learning and privacy-preserving methods.
• Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic — Neural networks on modular arithmetic tasks form task-intrinsic cyclic geometries, deviating from neural collapse due to a structured trade-off.
• Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion — A neuroscientifically inspired RL framework disentangles dynamics and reward features using LLEs and adaptive fusion for improved learning.
• Dimensionality Controls When Modularity Helps in Continual Learning — Representational dimensionality, influenced by initialization scale, dictates modularity's effectiveness in continual learning.
• Towards generalizable AI in medicine via Generalist–Specialist Collaboration — Combining generalist and specialist AI models improves medical diagnosis and report generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.