MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis

2026-07-12 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

MRUF, a novel reliability-aware fusion method, enhances multimodal sentiment analysis by addressing varying modality quality caused by issues like occlusion, background noise, or imperfect transcripts. This approach combines multi-granularity routing with uncertainty-aware calibration to prevent over-trusting unreliable modalities. MRUF summarizes sentiment-relevant representations, then performs subspace- and modality-level routing, supervising this process with leave-one-out error increases to gauge utterance-level modality importance. It further predicts modality-wise uncertainty and refines fusion weights using inverse-variance reweighting. A modality-invariant contrastive alignment component stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI datasets demonstrate consistent improvements over strong baselines, with analysis confirming that modalities exhibiting higher predicted uncertainty receive lower fusion weights.

Key takeaway

For Machine Learning Engineers developing robust multimodal sentiment analysis systems, MRUF offers a critical advancement. If your models struggle with real-world data quality issues like noise or occlusion, consider implementing uncertainty-aware fusion. This approach ensures your system dynamically prioritizes reliable modalities, preventing performance degradation from low-quality inputs. You should explore integrating similar multi-granularity routing and inverse-variance reweighting techniques to enhance the resilience and accuracy of your own multimodal models.

Key insights

MRUF robustly fuses multimodal sentiment data by dynamically weighting modalities based on their predicted uncertainty and importance.

Principles

• Modality reliability varies per utterance due to real-world noise.
• Uncertainty-aware weighting improves multimodal fusion robustness.
• Leave-one-out error can estimate modality importance.

Method

MRUF summarizes representations, routes them at subspace and modality levels, estimates importance via leave-one-out error, predicts modality uncertainty, and refines fusion gates using inverse-variance reweighting.

In practice

• Apply uncertainty estimation to dynamically weight multimodal inputs.
• Use leave-one-out error to quantify modality importance.
• Stabilize shared representations with contrastive alignment.

Topics

• Multimodal Sentiment Analysis
• Uncertainty Estimation
• Modality Fusion
• Deep Learning
• CMU-MOSI
• CMU-MOSEI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis — Separating modality refinement from cross-modal interaction improves multimodal fusion performance.
• DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling — DT2IT-MRM improves multimodal reward models by debiasing preference data and using iterative training.
• EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation — Explicit uncertainty supervision significantly enhances multimodal emotion recognition by dynamically weighting modalities based on learned variance.
• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach — A text-centered multimodal fusion model effectively recognizes ambivalence and hesitancy by anchoring on text and integrating other modalities.
• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.
• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification — Deep ensembles of unimodal networks effectively classify multimodal data, outperforming fusion methods without explicit modality integration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.