Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

Team RAS developed a single text-centered multimodal approach for ambivalence and hesitancy recognition in the 11th Affective & Behavior Analysis in-the-Wild (ABAW) Challenge. This "Text Residual Fusion" model combines linguistic, acoustic, facial, and scene features, treating text as the anchor modality and applying gated residual adjustments from other inputs. Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirmed text as the strongest unimodal component. The model achieved an average Macro F1-score (MF1) of 75.14% on the Development and Public Test subsets, and 78.24% on the Private Test subset, surpassing the text-only model by 4.03%. This demonstrates that complementary multimodal information can enhance recognition performance without requiring computationally expensive model ensembles.

Key takeaway

For Machine Learning Engineers developing affective computing systems, consider anchoring your multimodal models on text. The Text Residual Fusion approach demonstrates that integrating linguistic, acoustic, facial, and scene features, with text as the primary modality, significantly boosts ambivalence and hesitancy recognition (78.24% MF1). This allows you to achieve superior performance without the computational overhead of large model ensembles, streamlining deployment and reducing inference costs.

Key insights

A text-centered multimodal fusion model effectively recognizes ambivalence and hesitancy by anchoring on text and integrating other modalities.

Principles

• Text is a strong anchor for ambivalence recognition.
• Multimodal fusion improves recognition performance.
• Ensemble-free multimodal models are feasible.

Method

The Text Residual Fusion model combines linguistic, acoustic, facial, and scene features, using text as the anchor modality and applying gated residual adjustments from other inputs.

In practice

• Prioritize text features in ambivalence detection.
• Explore gated residual fusion for multimodal tasks.
• Avoid large ensembles for performance gains.

Topics

• Multimodal Fusion
• Ambivalence Recognition
• Affective Computing
• Text Residual Fusion
• ABAW Challenge
• Macro F1-score

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion — Multimodal fusion with cross-attention and gated units significantly improves ambivalence/hesitancy recognition over unimodal approaches.
• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.
• Overview of the Shared Task on Multilevel Political Meme Classification in Tamil and Malayalam — Hierarchical multimodal classification is crucial for nuanced political meme analysis in low-resource languages.
• Data-Driven Decoding of Russell's Circumplex Model of Affect — Transformer embeddings, particularly multimodal fusion, intrinsically encode Russell's circumplex model of affect, bridging psychology and representation learning.
• Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis — Separating modality refinement from cross-modal interaction improves multimodal fusion performance.
• Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features — Multimodal valence-arousal estimation, using quality-aware fusion, implicitly encodes identity, enhancing face recognition in unconstrained settings.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.