HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition

2026-07-14 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

The HSEmotion Team presented its results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition, focusing on two distinct tasks. For multi-task learning involving valence, arousal, facial expressions, and action units on the s-Aff-Wild2 dataset, the team utilized frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0, combined with separate heads and systematic post-processing techniques like temporal Gaussian smoothing and AffectNet blending. This approach significantly surpassed the ConvNeXt baseline on the official validation set. In the second task, ambivalence/hesitancy video recognition on the expanded BAH dataset, an extended audiovisual pipeline achieved a video-level Macro F1 of 0.73 on the public-test set, with frame-level Weighted F1 increasing from 0.74 to 0.79 on the validation set. Notably, competitive performance in both tasks was achieved without fine-tuning heavy backbones, indicating that systematic prediction calibration and lightweight multimodal fusion can match heavier end-to-end methods while enhancing efficiency and deployment flexibility.

Key takeaway

For Machine Learning Engineers developing affective computing systems, prioritize systematic prediction calibration and lightweight multimodal fusion. Your projects can achieve competitive performance in tasks like facial expression and ambivalence recognition without relying on fine-tuning heavy backbones. This approach improves model efficiency and deployment flexibility, allowing you to deploy robust solutions on resource-constrained hardware. Consider integrating post-processing techniques and late fusion strategies to enhance accuracy.

Key insights

Systematic prediction calibration and lightweight multimodal fusion can achieve competitive affective behavior analysis without heavy backbone fine-tuning.

Principles

• Lightweight models with calibration can outperform heavy baselines.
• Multimodal fusion enhances video-level affective recognition.
• Post-processing is crucial for prediction refinement.

Method

For multi-task learning, use frozen lightweight facial extractors with separate heads and systematic post-processing. For video recognition, extend an audiovisual pipeline with late fusion of face, audio, and text classifiers.

In practice

• Apply temporal Gaussian smoothing for expression prediction.
• Tune per-AU thresholds for action unit accuracy.
• Fuse face, HuBERT audio, and RoBERTa text for video analysis.

Topics

• Affective Behavior Analysis
• Multi-Task Learning
• Facial Expression Recognition
• Multimodal Fusion
• Lightweight Models
• Video Recognition

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach — A text-centered multimodal fusion model effectively recognizes ambivalence and hesitancy by anchoring on text and integrating other modalities.
• Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features — Multimodal valence-arousal features, enhanced by quality-aware fusion, implicitly encode identity, improving face recognition in challenging conditions.
• SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition — SVF-CR enhances multimodal ambivalence and hesitancy recognition by synchronizing and cross-refining visual and facial cues.
• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.
• Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion — Multimodal fusion with cross-attention and gated units significantly improves ambivalence/hesitancy recognition over unimodal approaches.
• cclin at SemEval-2026 Task 2 : SLM-Enhanced Lightweight Multi-BERT Ensemble for Longitudinal Affect Assessment — Lightweight SLM-enhanced multi-BERT ensembles offer efficient, competitive longitudinal affect assessment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.