VerbaNexAI at SemEval-2026 Task 4: Two-Stage Narrative Similarity via Fine-Tuned Bi-Encoder with MLP Ensemble

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

VerbaNex AI participated in SemEval-2026 Task 4: Narrative Similarity, a challenge focused on evaluating semantic relatedness in short narrative texts. The task included Track A, requiring selection of a more similar story, and Track B, demanding fixed-size story embeddings for cosine similarity. Their proposed two-stage system utilized Qwen3-Embedding-0.6B. The first stage involved fine-tuning the encoder as a bi-encoder with a 512-dimensional projection head, employing a composite loss combining margin ranking, pairwise softmax, and multiple negatives ranking objectives. The second stage trained a lightweight MLP head on frozen bi-encoder embeddings, incorporating pairwise interaction features, k-fold cross-validation, and logit-averaging ensemble inference. Despite ranking first in both tracks during development, the system's performance declined significantly on the official test set, finishing 47th on Track A and 22nd on Track B, without using the 1,900 synthetic triples.

Key takeaway

For NLP Engineers developing narrative similarity systems, you should carefully evaluate generalization capabilities beyond development sets. Your reliance on supervised data alone, even with sophisticated bi-encoder and MLP ensemble techniques, might not transfer to unseen test data. Consider incorporating diverse data sources, like the 1,900 synthetic triples, or robust domain adaptation strategies to improve real-world performance and avoid significant performance drops in competitive tasks.

Key insights

The two-stage bi-encoder and MLP ensemble system for narrative similarity showed strong dev performance but failed to generalize to the test set.

Principles

• Composite loss functions can enhance bi-encoder training.
• Ensemble inference improves model robustness.
• Dev set performance does not guarantee test set success.

Method

A two-stage system: first, fine-tune Qwen3-Embedding-0.6B as a bi-encoder with a 512-dimensional projection head and composite loss; second, train an MLP head on frozen embeddings using pairwise features, k-fold cross-validation, and logit-averaging ensemble.

In practice

• Use Qwen3-Embedding-0.6B for narrative tasks.
• Implement bi-encoder with 512D projection head.
• Apply k-fold cross-validation for robust training.

Topics

• Narrative Similarity
• SemEval-2026
• Bi-Encoder
• MLP Ensemble
• Qwen3-Embedding-0.6B
• Semantic Relatedness

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost — Cross-encoder rerankers offer inconsistent performance gains over strong embeddings, often failing on complex query types.
• IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources — Semantic deduplication and domain-balanced pretraining significantly enhance Persian PLM performance across diverse NLU tasks.
• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.