Yam at SemEval-2026 Task 4: Failure-Driven Prompt Evolution for Narrative Comparison

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Yen Yee Yam and Hong Meng Yam present a parameter-free system for SemEval-2026 Task 4 on Narrative Story Similarity. Their approach moves beyond scalar embedding proximity by decomposing stories into abstract theme, course of action, and outcome, enabling contrastive comparison aligned with the task ontology. The core innovation is a closed-loop, failure-driven prompt optimization procedure. This method iteratively refines guideline documents while keeping model parameters fixed, reverting any changes that degrade performance to ensure improvements stem from structured reasoning. Initially, ontology-aligned decomposition achieved 70% accuracy on both train and test sets. With the controlled guideline evolution, performance significantly improved to 76% on the train set and 73% on the test set, all without additional supervision or fine-tuning. This demonstrates the effectiveness of structured prompt optimization in enhancing contrastive narrative reasoning.

Key takeaway

For NLP Engineers developing narrative comparison systems, consider implementing failure-driven prompt optimization. This technique allows you to significantly improve model accuracy, from 70% to 73% on test sets, by refining guidelines iteratively without costly model fine-tuning or additional supervision. You should focus on decomposing complex tasks into ontology-aligned dimensions and establishing a feedback loop to revert ineffective prompt changes, ensuring robust and efficient performance gains.

Key insights

Failure-driven prompt evolution significantly enhances narrative comparison without model fine-tuning.

Principles

• Decompose narratives into theme, action, outcome for comparison.
• Isolate prompt improvements by fixing model parameters.
• Revert prompt changes that degrade performance.

Method

A closed-loop procedure iteratively refines concise guideline documents for a language model. It keeps model parameters fixed and reverts any guideline updates that cause performance degradation, thereby optimizing prompts based on observed failures.

In practice

• Implement iterative prompt refinement with performance checks.
• Structure narrative analysis into thematic components.
• Design prompts to guide reasoning, not just generate text.

Topics

• Narrative Comparison
• Prompt Engineering
• SemEval-2026 Task 4
• Parameter-Free Models
• Ontology Alignment
• Failure-Driven Optimization

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer, Prompt Engineer

Related on AIssential

• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment — Using contrastive learning with fine-tuned sentence transformers and multi-view alignment effectively models narrative structures and similarity.
• PromptMN: Pseudo Prompting Language — PromptMN is a pseudo-prompting language that adds structured, typed directives to natural language prompts, improving clarity and reusability.
• Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology — Empirical prompt engineering, focusing on construct definition and examples, significantly improves LLM text classification alignment with human expert judgment.
• How I Improved Speech-to-Text Accuracy — A two-pass LLM post-processing method effectively corrects common speech-to-text transcription errors.
• Do Large Language Models Always Tell The Same Stories? — Large Language Models consistently produce narratives that are more homogeneous than human-written stories, even with common mitigation efforts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.