Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

Mixture of Probes (MoP) is a novel framework designed for Multimodal Large Language Models (MLLMs) to effectively utilize "privileged modalities" – auxiliary data available only during training, not inference. Traditional MLLMs struggle to leverage such modalities, treating them as interchangeable inputs. MoP addresses this by disentangling modality-specific and modality-general signals within the MLLM, preserving modality-dependent structure while learning transferable representations. This is achieved through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, moving beyond final-layer alignment. The framework also includes MoP Cross-modal Training (MoP-X), featuring a probe disentanglement loss to prevent collapse and encourage cross-modal learning. Evaluated across two domains, eight tasks, and four modalities, MoP consistently outperformed strong MLLM baselines, demonstrating up to a 65% relative improvement. Code and models will be released by Sony on 2026-07-09.

Key takeaway

For Machine Learning Engineers developing Multimodal LLMs where auxiliary modalities are only available during training, you should consider implementing the Mixture of Probes (MoP) framework. This approach allows your models to effectively utilize privileged training data, potentially yielding up to a 65% relative performance improvement. By disentangling modality signals and using structured probing, you can build more robust MLLMs even with inference constraints.

Key insights

MoP effectively uses training-only auxiliary modalities in MLLMs via structured probing and signal disentanglement.

Principles

• Auxiliary modalities offer substantial gains even if inference-unavailable.
• Disentangle modality-specific and general signals for MLLMs.
• Structured probing of intermediate representations improves learning.

Method

MoP disentangles MLLM signals via structured probing of intermediate encoder representations. MoP-X training uses a probe disentanglement loss to prevent collapse and foster cross-modal learning.

In practice

• Apply MoP to MLLMs with training-only auxiliary data.
• Implement structured probing for richer cross-modal learning.
• Use probe disentanglement loss to stabilize MLLM training.

Topics

• Multimodal LLMs
• Privileged Modality
• Mixture of Probes
• Cross-modal Learning
• Modality Disentanglement
• Probing Mechanisms

Code references

• Sony/MoP

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion — LoMo addresses VLM "carrier sensitivity" by creating semantically equivalent text and image representations through local modality substitution.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Explicitly supervising modality transitions in interleaved thinking prevents "Modal Isolation," enhancing cross-modal coherence and task accuracy in multimodal models.
• MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models — MLLM-Microscope reveals MLLM internal representation dynamics, showing modality fusion impacts linearity, dimension, and anisotropy across transformer layers.
• Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval — Integrating LLaVA and two-stage fine-tuning significantly enhances fashion image retrieval by generating attribute-aware data and improving contrastive learning.
• Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs — Modality interference in full-duplex SLMs arises from deep-layer gradient conflicts and semantic dilution.
• Native Multimodality — Beyond Language-Centric Multimodal Models — Native multimodality, particularly early fusion, offers superior parameter efficiency and scaling over compositional models by integrating modalities from layer zero.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.