Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A two-stage multimodal framework was developed for the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, aiming to predict six continuous emotion intensities from in-the-wild video clips. This approach combines textual, acoustic, and visual representations, with an optional motion branch. It first trains modality-specific encoders independently, then fuses their learned representations via a lightweight regressor with modality dropout and controlled encoder adaptation. The best validation performance, an average Pearson correlation of 0.4722, was achieved by the text–audio–vision–motion fusion model under an expanded 4:1 data split. The team placed third in the EMI challenge with a test set Pearson correlation of 0.57, providing a reproducible baseline for EMI prediction. Text and audio were found to be the strongest unimodal predictors.

Key takeaway

For Machine Learning Engineers developing multimodal affective computing systems, this framework offers a robust strategy for handling sparse and imbalanced "in-the-wild" data. You should consider a staged training approach, pre-training unimodal encoders before fusion, and incorporating modality dropout to improve robustness. Experiment with expanding your training data, as a 4:1 train-to-validation split significantly boosted performance in this challenge.

Key insights

Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.

Principles

• Pre-train unimodal encoders before fusion for stable representations.
• Modality dropout enhances robustness to missing or weak channels.
• CCC-oriented loss is effective for imbalanced, sparse targets.

Method

Train modality-specific encoders independently. Fuse learned embeddings via a FusionRegressor with modality dropout and limited encoder adaptation, using a combined CCC and MSE loss for continuous emotion intensity prediction.

In practice

• Use DINOv2 for visual and customized wav2vec 2.0 for acoustic features.
• Employ OpenFace AU+pose features for motion representation.
• Combine CCC and MSE loss with α=0.7 for robust regression.

Topics

• Emotional Mimicry Intensity
• Multimodal Fusion
• Affective Computing
• Staged Training
• Wav2Vec 2.0
• DINOv2
• Hume-ABAW10 Challenge

Code references

• Dinithipurna/MimicMetrics

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition — SPCL dynamically balances multimodal contributions in conversational emotion recognition by adaptively selecting training samples.
• Data-Driven Decoding of Russell's Circumplex Model of Affect — Transformer embeddings, particularly multimodal fusion, intrinsically encode Russell's circumplex model of affect, bridging psychology and representation learning.
• EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detection — Emotion-augmented audio-visual features enhance deepfake detection generalization by complementing low-level manipulation cues.
• Frozen Multimodal Embeddings for Personality and Cognitive Ability Assessment in Asynchronous Video Interviews — Frozen multimodal embeddings effectively assess personality and cognitive ability from asynchronous video interviews, outperforming baselines.
• To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection — Cross-modal score consistency reliably detects active modalities, preventing noise from absent modalities in multimodal retrieval.
• Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations — Dynamic fusion of multimodal features via ODE-integrated GCNs improves conversational emotion recognition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.