CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

CuriosAI's system for SemEval-2026 Task 4 on narrative similarity assessment conducted comprehensive experiments comparing zero-shot pre-trained models, prompt engineering with large language models, and various fine-tuning strategies using synthetic data. A surprising finding revealed that pre-trained sentence transformers in a zero-shot setting consistently outperformed all fine-tuning attempts. Specifically, their best system, utilizing "sentence-transformers/sentence-t5-xl", achieved 67.5% accuracy on the development set (95% CI: [61.0%, 74.0%]). In contrast, all fine-tuning approaches resulted in a significant performance degradation, ranging from 2 to 18 percentage points. The study provides a detailed analysis of the reasons behind the fine-tuning failures and discusses the broader implications for narrative similarity tasks.

Key takeaway

For Machine Learning Engineers developing narrative similarity systems, you should prioritize zero-shot approaches with pre-trained sentence transformers. Your initial experiments should benchmark against models like "sentence-transformers/sentence-t5-xl", which achieved 67.5% accuracy in zero-shot. Be aware that fine-tuning, even with synthetic data, might degrade performance by 2-18 percentage points. This makes fine-tuning a potentially counterproductive effort for this specific task.

Key insights

Zero-shot sentence transformers surprisingly outperform fine-tuned models for narrative similarity, suggesting pre-trained knowledge is key.

Principles

• Pre-trained knowledge can exceed fine-tuning.
• Synthetic data fine-tuning may degrade performance.
• Zero-shot evaluation is a strong baseline.

Method

The study evaluated zero-shot pre-trained models, prompt engineering with LLMs, and multiple fine-tuning strategies using synthetic data for narrative similarity assessment.

In practice

• Prioritize zero-shot sentence transformers.
• Re-evaluate fine-tuning with synthetic data.
• Benchmark against "sentence-transformers/sentence-t5-xl".

Topics

• Narrative Similarity
• Zero-Shot Learning
• Fine-Tuning
• Sentence Transformers
• SemEval-2026 Task 4
• Large Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• The Hardest Part of Fine-Tuning Isn’t the Training — The most challenging aspect of NLP fine-tuning is effective label design and meticulous data annotation, not the model training itself.
• Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets — Genuine zero-shot image retrieval benchmarks require novel, consistent data and robust evaluation to counter inflated performance from pre-trained models.
• Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model — Plug-and-Adapt repurposes a pre-trained vision-language alignment model for zero-shot multimodal coreference resolution, outperforming existing methods.
• Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs? — LLMs effectively detect rating-text incoherence in Brazilian Portuguese reviews with high F1 scores and strong inter-model agreement.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative structure in web-scale LLM pretraining data is measurable and unevenly distributed, impacting model reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.