Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A novel plug-and-play module, Self-Paced Curriculum Learning (SPCL), is introduced to address modality imbalance and misalignment in Multimodal Emotion Recognition in Conversations (MERC). SPCL incorporates a dual-level Difficulty Measurer, which assesses both utterance-level recognition performance and conversation-level modality discrepancy, alongside a Learning Scheduler that dynamically guides training from easier to more challenging instances. Extensive experiments on the IEMOCAP and MELD datasets demonstrate consistent performance improvements. On IEMOCAP, SPCL achieved weighted F1-score gains ranging from approximately +1.2% to +6.6% over baseline models, while on MELD, improvements reached up to +10.4%. This approach significantly enhances model robustness and emotion recognition accuracy across diverse model architectures.

Key takeaway

For machine learning engineers developing Multimodal Emotion Recognition in Conversations (MERC) systems, consider integrating the Self-Paced Curriculum Learning (SPCL) module. Your models can achieve significant weighted F1-score improvements, ranging from +1.2% to +10.4% on benchmark datasets, by dynamically balancing modality contributions. Ensure careful tuning of the curriculum's initial threshold ("ε") and aging hyper-parameter ("α") to maintain a consistent, progressive learning rate, preventing overfitting or underutilization of weaker modalities.

Key insights

SPCL dynamically balances multimodal contributions in conversational emotion recognition by adaptively selecting training samples.

Principles

• Modality imbalance significantly hinders multimodal model performance.
• Dual-level difficulty assessment (utterance and conversation) improves learning progression.
• A consistent curriculum expansion rate is crucial for optimal model training.

Method

SPCL uses a dual-level Difficulty Measurer (utterance loss, conversation-level modality standard deviation) and a Learning Scheduler with a hard regularizer and exponential pacing to dynamically select training samples, progressing from easy to hard.

In practice

• Integrate SPCL as a plug-in module into existing MERC architectures.
• Tune "ε" and "α" to maintain a steadily increasing curriculum expanding rate.
• Utilize hard regularization for stable modality balance and improved performance.

Topics

• Multimodal Emotion Recognition
• Self-Paced Curriculum Learning
• Modality Imbalance
• Conversational AI
• IEMOCAP Dataset
• MELD Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.
• Leveraging Multimodal Self-Consistency Reasoning in Coding Motivational Interviewing for Alcohol Use Reduction — Multimodal self-consistency reasoning significantly enhances automated Motivational Interviewing (MI) coding by integrating diverse audio-language model predictions.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Interleaved thinking in multimodal models fails due to "Modal Isolation," requiring explicit transition-level supervision for coherence.
• Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations — Dynamic fusion of multimodal features via ODE-integrated GCNs improves conversational emotion recognition.
• Frozen Multimodal Embeddings for Personality and Cognitive Ability Assessment in Asynchronous Video Interviews — Frozen multimodal embeddings effectively assess personality and cognitive ability from asynchronous video interviews, outperforming baselines.
• LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification — Iterative imbalance-aware fine-tuning of LLMs, combined with augmentation and post-processing, significantly improves rare-class NLP performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.