PLlama at SemEval-2026 Task 4: Zero-shot Prompting with Llama-3.2 for Narrative Similarity

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

PLlama's submission to SemEval-2026 Task 4 addressed Narrative Story Similarity and Narrative Representation Learning. The team utilized the Llama-3.2-3B-Instruct pre-trained language model with prompt engineering for a zero-shot approach, avoiding explicit fine-tuning. The task involved binary classification, requiring models to determine which of two stories was more narratively similar to an anchor story based on perceived relatedness of event causality. On the test data, their system achieved approximately 55% accuracy in Track A. This result establishes a baseline for narrative similarity detection using large language models (LLMs), highlighting both their potential and the challenges of applying computationally efficient instruction-tuned models to such tasks. Analysis indicated LLMs struggle with capturing event causality and long-range narrative dependencies.

Key takeaway

For NLP engineers evaluating large language models for complex narrative understanding, you should recognize that current instruction-tuned LLMs like Llama-3.2, even with prompt engineering, show modest performance (55% accuracy) on tasks like narrative similarity. Your efforts might need to focus on addressing their inherent difficulties in capturing event causality and long-range dependencies, potentially through specialized fine-tuning or hybrid approaches, rather than relying solely on zero-shot prompting for high accuracy.

Key insights

Zero-shot Llama-3.2 achieved 55% accuracy on narrative similarity, establishing a baseline but revealing LLM struggles with causality.

Principles

• LLMs struggle with event causality.
• Long-range narrative dependencies challenge LLMs.
• Instruction-tuned models offer computational efficiency.

Method

The approach used Llama-3.2-3B-Instruct with prompt engineering for zero-shot binary classification, determining narrative similarity between stories and an anchor based on event causality.

Topics

• SemEval-2026
• Narrative Similarity
• Llama-3.2
• Zero-shot Prompting
• Large Language Models
• Event Causality
• Prompt Engineering

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian — QLoRA fine-tuning significantly boosts cross-lingual relation extraction performance for low-resource languages, outperforming prompt-only LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.