Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Team CV's systems for SemEval-2026 Task 4 addressed Narrative Story Similarity (Track A) and Narrative Representation Learning (Track B). For Track A, they explored five prompting strategies and QLoRA fine-tuning. Few-shot prompting with Qwen-2.5 7B achieved 64.00%, surpassing fine-tuned variants (best 57.50%), while scaling to LLaMA-3.3-70B yielded 75.00%. For Track B, twelve text embedding models (384–4096 dimensions) were benchmarked, including E5-Large-v2, BGE, GTE, Qwen3 Embedding, OpenAI, Gemini, and Mistral. OpenAI text-embedding-3-large (3072-d) achieved the best dev accuracy at 67.00%. Fine-tuning Qwen3 Embedding 4B (2560-d) slightly decreased accuracy. Their final submission, LLaMA-3.3-70B (3-shot) for Track A and text-embedding-3-large for Track B, achieved 70.75% and 64.50% respectively, outperforming GPT-4o-mini and STORY-EMB baselines.

Key takeaway

For NLP Engineers developing systems for narrative story similarity, you should prioritize few-shot prompting with larger LLMs like LLaMA-3.3-70B for comparative judgments, as it significantly outperforms fine-tuned smaller models. For narrative embeddings, leverage robust commercial models such as OpenAI's text-embedding-3-large, as fine-tuning smaller open-source alternatives may not yield better results.

Key insights

Few-shot prompting with large LLMs and robust commercial embeddings excels at narrative story similarity tasks.

Principles

• Few-shot prompting can outperform QLoRA fine-tuning.
• Scaling LLM size significantly improves comparative judgment.
• Dedicated commercial embedding models offer strong performance.

Method

Team CV explored five prompting strategies and QLoRA fine-tuning for comparative judgment, and benchmarked twelve embedding models, fine-tuning one on task-specific triples.

In practice

• Prioritize few-shot prompting for LLM-based comparative tasks.
• Evaluate commercial embedding models before fine-tuning.
• Consider larger LLMs for higher accuracy in complex tasks.

Topics

• Narrative Story Similarity
• Large Language Models
• Text Embedding Models
• Few-shot Prompting
• SemEval-2026 Task 4
• QLoRA Fine-tuning

Best for: AI Engineer, Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian — QLoRA fine-tuning significantly boosts cross-lingual relation extraction performance for low-resource languages, outperforming prompt-only LLMs.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.