EmoNet: Speaker-Aware Transformers for Emotion Recognition — and What I’d Build Differently in 2026

2026-05-28 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

The EmoNet model, developed for Emotion Recognition in Conversation (ERC), achieved a Weighted F1 of 39.18 on the EmoryNLP dataset in March 2024, outperforming its CoMPM baseline by +1.81 F1. ERC is challenging due to the contextual and speaker-dependent nature of emotions in text-only dialogues. EmoNet introduced three key contributions: Global Speaker Identity, assigning stable IDs across dialogues; a Speaker Behaviour Module utilizing a GRU to compress speaker history; and Weighted Cross-Entropy Loss to address class imbalance without distorting conversational sequences. While Global Speaker Identity initially degraded performance, its combination with the Speaker Behaviour Module ultimately led to EmoNet's success. By 2026, the ERC field evolved to LLaMA-2–7B-based systems with LoRA fine-tuning and retrieval-augmented prompting, yet EmoNet's core intuitions regarding speaker-specific patterns persist, now integrated into LLM instruction tuning or retrieval contexts.

Key takeaway

For Machine Learning Engineers building conversational AI, recognize that speaker identity and historical context are critical, even as models evolve. If you are developing emotion recognition systems, consider integrating global speaker characteristics and their temporal behavior, perhaps via retrieval-augmented LLM prompts or instruction tuning, rather than solely relying on local dialogue context. Your architectural intuitions about speaker patterns can be adapted across different model paradigms.

Key insights

Speaker-specific patterns and historical context are crucial for accurate emotion recognition in conversations.

Principles

• Emotion is context- and speaker-dependent.
• Features need machinery to be valuable.
• Ideas survive paradigm shifts.

Method

EmoNet combines RoBERTa embeddings with a GRU for global, temporally decaying speaker history and weighted cross-entropy loss for imbalanced conversational data.

In practice

• Use global speaker IDs for context.
• Employ GRUs for speaker history compression.
• Apply weighted loss for imbalanced sequences.

Topics

• Emotion Recognition in Conversation
• Speaker Identity Modeling
• Transformers
• Large Language Models
• LoRA Fine-tuning
• Retrieval-Augmented Generation

Code references

• bijupv/emonet-erc

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations — MiSTER-E uses a Mixture-of-Experts framework to fuse speech and text for robust emotion recognition in conversations.
• How Transformers Architecture Powers Modern LLMs — LLMs use a transformer architecture to convert text into numerical representations, process context, and predict the next token.
• Why Do Vision Language Models Struggle To Recognize Human Emotions? — VLMs struggle with emotion recognition due to long-tailed datasets and poor temporal information processing.
• Stop Being Nice to Your Chatbot: The Surprising Science of Prompt Politeness — Polite prompts reduce LLM accuracy and increase computational cost, while direct or emotionally-charged prompts improve performance.
• Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue — Modeling dialogue as a dynamic tree improves LLM context management, task completion, and token efficiency in non-linear conversations.
• Sentiment Analysis with BERT: The Heart of the Transformer, The Soul of the Text — BERT's bidirectional context understanding via Transformers significantly advanced sentiment analysis and NLP tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.