Team HausaNLP at SemEval-2026 Task 4: Narratives via Semantic Embeddings

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Team HausaNLP submitted their approach to SemEval-2026 Task 4 (Track A), focusing on identifying the more narratively similar story from two candidates relative to an anchor. Narrative similarity was defined across abstract theme, course of action, and story outcomes. Their systematic ablation compared five methods: a TF-IDF baseline, two SBERT bi-encoder variants (all-MiniLM-L6-v2 and all-mpnet-base-v2), a paraphrase embedding model, and a cross-encoder re-ranker. On a 200-instance development set, all-mpnet-base-v2 performed best with 61.5% accuracy and 61.48 macro-F1, surpassing TF-IDF (54.5%) and the official SBERT baseline (55.0%). The cross-encoder re-ranker achieved 55.5% but did not improve on bi-encoders, likely due to long Wikipedia summaries. Their SBERT MiniLM submission secured 61.50% accuracy on the official test set, placing 33rd out of 44 teams. Error analysis revealed five failure categories, including 23 Lexical Trap and 23 Hard Cases.

Key takeaway

For NLP Engineers developing narrative similarity systems, prioritize bi-encoder SBERT models like all-mpnet-base-v2, which showed strong performance (61.5% accuracy) over TF-IDF and cross-encoders. Be aware that cross-encoders may underperform with long documents due to context window limitations. Your error analysis should categorize failures, such as "Lexical Trap" or "Hard Cases," to guide targeted model refinements and improve overall system robustness.

Key insights

Bi-encoder SBERT models demonstrate superior performance for narrative similarity tasks, especially with long documents.

Principles

• Bi-encoders are effective for long document similarity.
• Cross-encoders face context window limitations.
• Systematic error analysis guides future model improvements.

Method

A systematic ablation compares lexical, bi-encoder SBERT, paraphrase embedding, and cross-encoder re-ranker approaches for narrative similarity.

In practice

• Favor bi-encoder SBERT variants for narrative tasks.
• Evaluate cross-encoder context window for long texts.
• Use error categorization to refine similarity models.

Topics

• Narrative Similarity
• Semantic Embeddings
• SBERT Models
• Bi-encoder Architectures
• Cross-encoder Re-rankers
• SemEval-2026

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost — Cross-encoder rerankers offer inconsistent performance gains over strong embeddings, often failing on complex query types.
• Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News — Fine-tuned BERT models outperform few-shot LLMs for German climate news framing detection, especially with sentence context.
• Principles of Concept Representation in Sentence Encoders — Sentence encoders' concept representation quality hinges on matching supervision to semantic composition types.
• FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings — FLiP models effectively interpret sentence embeddings by linearly extracting lexical content, revealing underlying biases.
• Solving the Embedding Step in NLP Projects — Effective embeddings are crucial for NLP model generalization and reliable downstream task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.