NarSiL at SemEval-2026 Task 4: A Multi-Expert, Multi-Pathway System for Narrative Story Similarity

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

NarSiL (Narrative Similarity Learners), a system developed for SemEval-2026 Task 4 Track A on Narrative Story Similarity, employs a two-stage architecture. It begins with a Mixture-of-Experts (MoE) initial classifier that incorporates supermajority voting across three large language models: Gemma-3-12B, GPT-3.5-turbo-instruct, and Gemini-2.5-Flash, over multiple runs. For ambiguous cases, a structured three-pathway fallback system is utilized, with each path addressing one of the task's core similarity components: abstract theme, narrative outcome, and course of action. Individual pathway scores are then combined through weighted aggregation. NarSiL achieved 64.25% accuracy on the official test set, with an improved score of 70.25% when considering only GPT supermajority voting before the fallback.

Key takeaway

For NLP engineers developing narrative understanding systems, NarSiL's approach offers a robust strategy. You should consider a multi-stage architecture that integrates large language model supermajority voting for initial classification, especially with models like GPT-3.5-turbo-instruct. For ambiguous cases, decompose narrative similarity into components like abstract theme and course of action, processing them via dedicated pathways before weighted aggregation to improve accuracy.

Key insights

The NarSiL system combines multi-expert LLM voting with a structured, multi-pathway fallback for narrative story similarity.

Principles

• Multi-expert voting enhances initial classification accuracy.
• Decompose complex similarity into distinct components.
• Weighted aggregation combines component-specific scores.

Method

A two-stage architecture: an MoE classifier with LLM supermajority voting (Gemma-3-12B, GPT-3.5-turbo-instruct, Gemini-2.5-Flash), followed by a three-pathway fallback for ambiguous cases, and weighted score aggregation.

In practice

• Implement LLM supermajority voting for robust initial classification.
• Design dedicated pathways for specific similarity aspects.
• Use GPT-3.5-turbo-instruct for strong baseline performance.

Topics

• Narrative Story Similarity
• Mixture-of-Experts
• Large Language Models
• SemEval-2026 Task 4
• Multi-pathway Systems
• Supermajority Voting

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• A Self Consistency Based Reranking for Narrative Question Answering — Semantic agreement among multiple generated answers improves Narrative Question Answering robustness.
• Narrative Knowledge Weaver: Narrative-Centric Retrieval-Augmented Reasoning for Long-Form Text Understanding — Narrative Knowledge Weaver (NKW) enhances RAG by modeling evolving story worlds through structured narrative assets and multi-channel evidence assembly.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey — LLMs bridge abstract narrative theories with NLP, enabling scalable story generation and understanding despite definitional challenges.
• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.