CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

CUCLASIC's contribution to SemEval-2026 Task 5 investigates Large Language Model (LLM) prompting strategies for rating ambiguous word senses, a long-standing challenge in computational semantics. The research evaluated six LLMs from the Llama and Gemini families, specifically assessing their ability to rate ambiguous textual excerpts. Experiments included zero- and few-shot prompting variants, alongside an analysis of how simple linguistic cues influence performance. A key methodology proposed involves eliciting human-like ratings from LLMs by utilizing examples with both low and high standard deviations in human ratings. The study also compared the prediction patterns of different models against human-generated ratings. The top performer, Gemini 3-Flash, achieved a 75% score, which combines Spearman correlation and accuracy within one standard deviation. This work aims to bridge the gap between human and computational evaluation of ambiguity.

Key takeaway

For NLP Engineers evaluating LLMs for word sense disambiguation, you should prioritize models like Gemini 3-Flash, which demonstrated 75% alignment with human ambiguity ratings. Incorporate both zero- and few-shot prompting, and specifically design your few-shot examples using human rating data that exhibits varying standard deviations. This approach can significantly improve your model's ability to mimic human judgment in complex semantic tasks.

Key insights

Gemini 3-Flash achieved 75% human-like ratings for ambiguous word senses using zero/few-shot LLM prompting and linguistic cues.

Principles

• LLMs can align with human ambiguity ratings.
• Linguistic cues enhance LLM ambiguity performance.
• Human rating variance guides LLM elicitation.

Method

Elicit human-like ambiguity ratings from LLMs by testing zero- and few-shot prompts, analyzing linguistic cues, and using examples with low and high human rating standard deviations.

In practice

• Apply Gemini 3-Flash for word sense ambiguity.
• Integrate linguistic cues into LLM prompts.
• Craft few-shot examples from human rating variance.

Topics

• Word Sense Disambiguation
• Large Language Models
• Prompt Engineering
• Gemini 3-Flash
• SemEval-2026
• Computational Semantics

Best for: Research Scientist, AI Scientist, NLP Engineer, Prompt Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• LLMs Can Better Capture Human Judgments--With the Right Prompts — Asking better questions to LLMs through specific prompting strategies improves their ability to capture human judgments and response distributions.
• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Gemini 3.5 FLASH: BAD to OUTSTANDING — Gemini 3.5 Flash offers significant cost savings but exhibits high solution variance and requires iterative prompting for complex tasks.
• UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding — UrduMMLU reveals current LLMs have uneven Urdu knowledge, particularly for regionally grounded content, despite strong performance from top models.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.