SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

The SwanNLP framework, presented at SemEval-2026 Task 5, introduces an LLM-based approach for plausibility scoring of homonymous word senses within narrative texts. This framework addresses the underexplored practical applicability of Large Language Models (LLMs) in real-world narrative contexts, moving beyond standard benchmarks. It employs a structured reasoning mechanism and explores two main strategies: fine-tuning low-parameter LLMs with various reasoning strategies and utilizing dynamic few-shot prompting for large-parameter models. The research demonstrates that commercial large-parameter LLMs, when combined with dynamic few-shot prompting, can closely mimic human-like plausibility judgments. Furthermore, the study found that ensembling multiple models slightly enhances performance, more accurately reflecting the agreement patterns observed among five human annotators than individual model predictions.

Key takeaway

For NLP Engineers developing systems for narrative understanding, you should prioritize commercial large-parameter LLMs with dynamic few-shot prompting for word sense disambiguation. This approach closely replicates human plausibility judgments in short stories. Additionally, consider implementing model ensembling to slightly improve performance and better simulate human annotator agreement patterns, enhancing the robustness of your narrative NLU applications.

Key insights

Commercial LLMs with dynamic few-shot prompting can replicate human narrative plausibility judgments, improved by ensembling.

Principles

• LLMs need structured reasoning for narrative WSD.
• Dynamic few-shot prompting enhances large LLM performance.
• Model ensembling improves human agreement simulation.

Method

An LLM-based framework uses structured reasoning for plausibility scoring of homonymous word senses in narrative texts. It involves fine-tuning low-parameter LLMs or dynamic few-shot prompting for large-parameter models.

In practice

• Apply dynamic few-shot prompting for narrative tasks.
• Consider model ensembling for improved human alignment.
• Fine-tune smaller LLMs with diverse reasoning.

Topics

• Large Language Models
• Word Sense Disambiguation
• Narrative Understanding
• Plausibility Scoring
• Few-shot Prompting
• Model Ensembling

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — In-context learning for Turkish LVCs is highly prompt-sensitive, with few-shot demonstrations improving LLM calibration to match supervised baselines.
• Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning — Everyday causal reasoning in humans and LLMs appears driven by pattern-matching, not abstract world models.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative content in LLM pretraining data is measurable and unevenly distributed, impacting model capabilities.
• Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring — LLM semantic similarity scores are sensitive to document structure, context coherence, and model identity.
• Prompt Repetition: Google's Surprisingly Strong Baseline for Non-Reasoning LLMs — Repeating prompts can dramatically improve LLM accuracy by providing a "clean second read" with full context.
• Zero-Shot Goal Recognition with Large Language Models — LLM goal recognition competence varies significantly, with some models integrating evidence effectively while others rely on world-knowledge priors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.