ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

2026-07-07 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

ProMoE-FL, a Prototype-conditioned Mixture-of-Experts framework, addresses the problem of multimodal federated learning with missing modalities. This novel approach overcomes limitations of existing methods, which often rely on additional public datasets or naive feature synthesis. ProMoE-FL constructs a global client-aware prototype bank to capture clinically meaningful modality priors across different institutions. Its Mixture of Experts component is conditioned on these prototypes and modality indices, enabling direction-aware expert routing for dynamic synthesis of missing features. The framework was extensively evaluated on four public chest X-ray datasets: MIMIC-CXR, NIH Open-I, PadChest, and CheXpert. ProMoE-FL consistently demonstrated superior performance compared to other methods in both homogeneous and heterogeneous settings, indicating its robustness in diverse federated learning environments.

Key takeaway

For AI Scientists developing multimodal federated learning systems with incomplete data, ProMoE-FL offers a robust solution. If your models struggle with missing modalities or heterogeneous client data, consider integrating a prototype-conditioned Mixture-of-Experts approach. This can significantly improve feature synthesis and overall model performance, especially in clinical imaging applications like chest X-rays, by leveraging global client-aware priors.

Key insights

ProMoE-FL uses prototype-conditioned Mixture-of-Experts to synthesize missing features in multimodal federated learning, improving robustness across institutions.

Principles

• Global prototypes capture clinical modality priors.
• Expert routing is conditioned on prototypes and modality indices.
• Dynamic feature synthesis enhances robustness.

Method

ProMoE-FL builds a global client-aware prototype bank, then uses a Mixture of Experts conditioned on these prototypes and modality indices to dynamically synthesize missing features for multimodal federated learning.

In practice

• Apply to chest X-ray datasets like MIMIC-CXR.
• Improve multimodal FL performance with incomplete data.
• Enhance robustness in heterogeneous FL settings.

Topics

• Multimodal Federated Learning
• Missing Modalities
• Mixture-of-Experts
• Prototype Learning
• Chest X-ray Analysis
• Medical Imaging AI

Best for: Computer Vision Engineer, AI Scientist, Research Scientist

Related on AIssential

• MIND: multimodal integration with neighbourhood-aware distributions — MIND integrates incomplete multimodal data using a VAE with neighbourhood-aware priors, improving robustness and performance in tasks like cancer stratification.
• Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models — Self-evolving LMMs often under-condition visually; VISE uses invariance rewards to enforce spatial and semantic consistency.
• PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning — PRISM recovers missing modality semantics from the federation, integrating them into local graph propagation with topology-aware control.
• Federated orthogonal learning for detection of liver lesions from multi-phase contrast-enhanced CT images — FedOG enhances federated learning for medical image segmentation by using orthogonal gradient decomposition to counter data heterogeneity and incomplete phases.
• Latent World Recovery for Multimodal Learning with Missing Modalities — Latent World Recovery (LWR) enables robust multimodal learning with missing data by fusing only available modalities in a shared latent space.
• Comprehensive and Reliable Feature Attribution for Diverse Modalities and Models via Frequency-Domain Insights — FreqX offers fast, privacy-preserving, and detailed interpretability for Personalized Federated Learning through frequency-domain insights.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.