SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A study on SHAP-weighted cross-modal expert fusion for multimodal emotion and sentiment recognition addresses the impact of SHAP attribution reduction methods on performance, particularly when experts have unequal feature dimensionalities. The research introduces a "sum-abs" reduction method, which preserves total attribution mass, outperforming "mean-abs" and "median-abs" reductions. On the MELD 7-class emotion recognition dataset, the sum-abs XGAF variant achieved 0.5983 weighted-F1, statistically matching early fusion's 0.6018 weighted-F1 and significantly surpassing late fusion's 0.4598 weighted-F1. For CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF reached 0.6519 weighted-F1, slightly exceeding early fusion's 0.6485 weighted-F1 with statistical significance ($p=0.0452$). Diagnostic analysis revealed that performance gains stem from including cross-modal experts, especially the trimodal expert, rather than complex per-sample routing.

Key takeaway

For Machine Learning Engineers designing multimodal fusion strategies, you should prioritize using sum-abs SHAP attribution reduction for expert weighting, especially with varying expert dimensionalities. The compact four-expert XGAF-Lite configuration offers a modular, interpretable approach that matches early fusion performance on MELD. When implementing, evaluate not just accuracy but also expert weight diversity to ensure genuinely adaptive routing, rather than just expert dominance.

Key insights

Sum-abs SHAP reduction is critical for effective multimodal expert fusion when experts have unequal feature dimensionalities, preserving total attribution mass.

Principles

• Sum-abs SHAP reduction is crucial for experts with unequal feature dimensionalities.
• Cross-modal experts, especially trimodal, are key to recovering early-fusion performance.
• Attribution-based gating may lead to expert dominance, not diverse routing.

Method

XGBoost classifiers form unimodal and cross-modal expert pools, combined via a temperature-scaled softmax gate where weights are derived from sum-absolute TreeSHAP attribution magnitudes.

In practice

• Apply sum-abs SHAP reduction for expert gating in multimodal systems.
• Include cross-modal experts, especially trimodal, in expert pools.
• Analyze expert weight diversity and entropy alongside accuracy.

Topics

• Multimodal Fusion
• Emotion Recognition
• Sentiment Analysis
• SHAP Attribution
• Mixture-of-Experts
• XGBoost

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• SCoPE: Shift-Aware Speaker-Conditioned Priors for Emotion Recognition in Conversations — SCoPE dynamically fuses speaker-conditioned emotional history with multimodal evidence, guided by predicted emotion shifts, for robust ERC.
• Team JAT at SemEval-2026 Task 9: Enhancing Polarization Detection with Cross-Lingual Transfer and Feature Fusion — Team JAT's system enhances polarization detection using RoBERTa, feature fusion, and cross-lingual transfer, achieving 0.763 F1-score.
• CITD@UIT at SemEval-2026 Task 2: Temporal Mixture-of-Experts for Longitudinal Valence and Arousal Prediction from Ecological Essays — A temporal Mixture-of-Experts RoBERTa framework effectively predicts longitudinal emotional states from text.
• Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition — Combining audio, text, and psycholinguistic features via cross-attention improves ambivalence recognition.
• Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection — EC-Net uses hyperbolic geometry and hypergraphs for resilient multimodal emotion analysis, especially with incomplete or noisy data.
• Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models — Preserving cross-corpus expert coverage with generic data improves MoE pruning without downstream calibration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.