Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

2026-07-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A multimodal framework for Ambivalence/Hesitancy (A/H) recognition in video was developed for the ABAW11 challenge at ECCV 2026. This approach integrates textual, acoustic, and visual modalities extracted from the BAH dataset using pretrained encoders: F2LLM-v2-0.6B for transcripts (1024-d), WavLM-Large for audio (1024-d), and VideoMAE V2 for facial video (768-d). Initial unimodal baselines, optimized with Optuna, showed text features alone achieved a Macro F1 of 0.6659 on the test set, substantially surpassing the zero-shot Video-LLaVA baseline's 0.2827. The proposed multimodal fusion architecture, which employs bidirectional cross-attention across all three modalities and a Gated Multimodal Unit (GMU), was also optimized via a 50-trial Optuna search. This model achieved a Macro F1 of 0.7394 on the validation set, demonstrating an 11.0% relative improvement over the best unimodal baseline and confirming the value of explicit cross-modal interaction.

Key takeaway

For AI Scientists developing robust human behavior recognition systems, this research indicates that integrating textual, acoustic, and visual modalities significantly enhances performance. You should prioritize architectures that facilitate explicit cross-modal interaction, such as bidirectional cross-attention and Gated Multimodal Units, to capture complementary cues. Leveraging tools like Optuna for hyperparameter and architectural optimization is also vital for achieving superior results, as unimodal approaches often fall short.

Key insights

Multimodal fusion with cross-attention and gated units significantly improves ambivalence/hesitancy recognition over unimodal approaches.

Principles

• Cross-modal interaction captures complementary cues that single modalities miss.
• Text features alone can substantially outperform zero-shot multimodal baselines.
• Optimization via Optuna is effective for model and hyperparameter selection.

Method

Extract textual, acoustic, and visual features using pretrained encoders, then fuse them with bidirectional cross-attention and a Gated Multimodal Unit (GMU), optimizing architecture and hyperparameters via Optuna.

In practice

• Use F2LLM-v2-0.6B for text, WavLM-Large for audio, VideoMAE V2 for video.
• Implement bidirectional cross-attention for robust multimodal fusion.
• Employ Optuna for hyperparameter and architectural optimization.

Topics

• Multimodal Fusion
• Ambivalence Recognition
• Hesitancy Recognition
• Cross-Attention
• Gated Multimodal Unit
• Optuna Optimization

Code references

• yassineouzar/IUSD_AH

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach — A text-centered multimodal fusion model effectively recognizes ambivalence and hesitancy by anchoring on text and integrating other modalities.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Effective interleaved multimodal reasoning requires explicit structural supervision at modality boundaries, not just end-task optimization.
• Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis — Separating modality refinement from cross-modal interaction improves multimodal fusion performance.
• Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features — Multimodal valence-arousal estimation, using quality-aware fusion, implicitly encodes identity, enhancing face recognition in unconstrained settings.
• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification — Deep ensembles of unimodal networks effectively classify multimodal data, outperforming fusion methods without explicit modality integration.
• Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability — Combining text and audio features significantly improves valence perception modeling in spontaneous Finnish speech.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.