COGNAC at SemEval-2026 Task 4: Evaluating Narrative Components with LLMs for Hard Story Similarity Cases

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

The COGNAC system, developed for SemEval-2026 Task 4, addresses narrative similarity by evaluating stories based on abstract theme, course of action, and outcome. For Track A, the system employs a two-stage approach: initially using majority voting from multiple large language model (LLM) judgments for high-agreement cases. Difficult, low-agreement examples are routed to a second stage that decomposes stories into their narrative components, scoring them with learned weights or comparing them via structured chain-of-thought prompting. This method achieved first place with 0.78 test accuracy. In Track B, COGNAC generated embeddings from full stories and individual components, finding that embeddings derived solely from the course-of-action component delivered the best performance, securing first place with 0.72 accuracy. Analyses also highlighted significant annotation variability in the dataset, underscoring the need to manage ambiguity in narrative similarity modeling.

Key takeaway

For NLP Engineers developing narrative understanding systems, consider adopting a multi-stage LLM approach that first handles clear cases, then decomposes difficult examples into core components like "course of action." Your models will gain robustness, especially if you prioritize embeddings derived from action sequences, as this method achieved top performance in SemEval-2026 Task 4. Be prepared to explicitly address annotation variability in your datasets to improve overall system accuracy.

Key insights

Decomposing narratives into components and using a two-stage LLM approach significantly improves story similarity evaluation.

Principles

• Narrative similarity benefits from component-wise analysis.
• Handling annotation disagreement improves model robustness.
• Course-of-action is a strong signal for story embeddings.

Method

A two-stage system: majority voting by LLMs for easy cases, then component decomposition with weighted scoring or chain-of-thought prompting for difficult cases.

In practice

• Implement a two-stage LLM system for complex text tasks.
• Prioritize "course of action" for narrative embedding models.
• Address dataset annotation variability in model design.

Topics

• Narrative Similarity
• Large Language Models
• SemEval-2026
• Story Embeddings
• Chain-of-Thought Prompting
• Two-stage Systems

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• StoryCoder: Narrative Reformulation for Structured Reasoning in LLM Code Generation — Narrative reformulation of code problems significantly improves LLM code generation by structuring context.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative content in LLM pretraining data is measurable and unevenly distributed, impacting model capabilities.
• NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama — N-VSSM, a novel latent world model, significantly outperforms frontier LLMs in long-horizon audio drama consistency and controllability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.