MoodMetric at SemEval-2026 Task 4:Narrative Story Similarity and Narrative Representation Learning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

MoodMetric's system for SemEval-2026 Task 4 addresses narrative story similarity using transformer-based dense embedding approaches. This task is challenging due to the need to capture event progression, causal structure, character dynamics, and thematic coherence in long-form narratives. The system evaluated pretrained encoder-only architectures like DeBERTa-v3, BGE-Base, BGE-Large, and E5-Large, fine-tuned with triplet margin and contrastive objectives. Experiments under low-resource conditions (approximately 1,900 training triplets) showed larger contrastively pretrained models, particularly BGE-Large, performed best standalone. Optimal fine-tuning occurred within 4-5 epochs, with extended training causing overfitting. Instruction-tuned embeddings offered no significant advantage. Arithmetic averaging of embeddings from diverse models yielded the most robust representations, achieving approximately 65% validation accuracy.

Key takeaway

If you are an NLP Engineer developing narrative similarity systems with limited data, prioritize ensembling diverse transformer embeddings, like BGE-Large, and restrict fine-tuning to 4-5 epochs to achieve robust performance and prevent overfitting. Your efforts should focus on combining models rather than extended training or instruction-tuning for optimal results.

Key insights

Ensembling diverse model embeddings robustly improves narrative similarity.

Principles

• Larger contrastively pretrained models outperform smaller variants.
• Optimal fine-tuning for narrative similarity is brief (4-5 epochs).
• Instruction-tuned embeddings offer no significant advantage for this task.

Method

The system fine-tunes transformer-based dense embeddings (e.g., BGE-Large) using triplet margin and contrastive objectives, then arithmetically averages embeddings from diverse models.

In practice

• Prioritize BGE-Large for strong standalone performance.
• Limit fine-tuning to 4-5 epochs to avoid overfitting.
• Combine diverse model embeddings via arithmetic averaging.

Topics

• Narrative Similarity
• Transformer Models
• Dense Embeddings
• SemEval Task 4
• Contrastive Learning
• Ensemble Methods
• Low-Resource NLP

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• Principles of Concept Representation in Sentence Encoders — Sentence encoders' concept representation quality hinges on matching supervision to semantic composition types.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative content in LLM pretraining data is measurable and unevenly distributed, impacting model capabilities.
• Do Large Language Models Always Tell The Same Stories? — Large Language Models consistently produce narratives that are more homogeneous than human-written stories, even with common mitigation efforts.
• Solving the Embedding Step in NLP Projects — Effective embeddings are crucial for NLP model generalization and reliable downstream task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.