COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

The COGNAC system, developed by Azwad Anjum Islam and Tisa Islam Erana for SemEval-2026 Task 5, predicts 1–5 plausibility ratings for homonym candidate senses within ambiguous short stories. This system utilizes prompting with closed-source Large Language Models (LLMs) and evaluated three distinct prompting strategies: zero-shot, chain-of-thought, and comparative prompting, which jointly scores competing senses. A key finding was that simple unweighted ensembling of LLMs better aligns with subjective human judgments than individual models. The official submission achieved 4th place on the leaderboard with an average score of 0.86, with subsequent post-competition experiments further improving performance to 0.89.

Key takeaway

For NLP Engineers developing word sense disambiguation systems, consider integrating LLM ensembles with comparative prompting. This approach demonstrates superior alignment with human judgment for plausibility ratings, outperforming individual models. You should explore unweighted ensembling of closed-source LLMs to enhance accuracy and robustness in tasks requiring nuanced semantic understanding, especially within ambiguous narrative contexts.

Key insights

LLM ensembles, particularly with comparative prompting, achieve human-level word sense plausibility ratings in challenging narratives.

Principles

• Unweighted ensembling improves human alignment.
• Comparative prompting enhances sense scoring.
• Closed-source LLMs can rate plausibility.

Method

The system uses prompting (zero-shot, chain-of-thought, or comparative) with closed-source LLMs to rate homonym sense plausibility, then applies simple unweighted ensembling for final scores.

In practice

• Experiment with comparative prompting for disambiguation.
• Apply unweighted LLM ensembles for subjective tasks.
• Use LLMs for fine-grained plausibility scoring.

Topics

• LLM Ensembles
• Word Sense Disambiguation
• SemEval-2026
• Prompt Engineering
• Plausibility Rating
• Homonym Resolution

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstracts — Ensemble LLM frameworks reliably automate biomedical text classification, balancing performance and interpretability.
• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning — The NSNRL task operationalizes narrative similarity for evaluation and identifies areas for automated system improvement.
• LLMs Can Better Capture Human Judgments--With the Right Prompts — Asking better questions to LLMs through specific prompting strategies improves their ability to capture human judgments and response distributions.
• Prompt Repetition: Google's Surprisingly Strong Baseline for Non-Reasoning LLMs — Repeating prompts can dramatically improve LLM accuracy by providing a "clean second read" with full context.
• SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity — LLMs moderately generate equations from text but struggle with semantic accuracy and reliable self-evaluation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.