VerbaNexAI at SemEval-2026 Task 5: Few-Shot Chain-of-Thought with Selective Self-Consistency and Isotonic Calibration for Word Sense Plausibility Rating

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The VerbaNexAI system, developed for SemEval-2026 Task 5, rates word sense plausibility in ambiguous narrative contexts. It ensembles Llama-3.1 70B, Qwen-2.5 32B, and Gemma-2 27B using an uncertainty-aware pipeline. The approach combines few-shot chain-of-thought prompting with selective self-consistency, applying stochastic multiple sampling only to inherently ambiguous items, which reduces inference costs by approximately 45%. To correct LLM bias towards extreme ratings, isotonic regression shifts the output distribution to align with human judgment. The system achieved a Spearman correlation of 0.67 and accuracy within 0.76 standard deviations, ranking 34th out of 79 teams (top 43%) without task-specific fine-tuning. While strong on clear contexts (ρ = 0.78), it struggles with multimodal human disagreement in genuinely ambiguous cases (ρ = 0.58).

Key takeaway

For NLP engineers optimizing LLM performance on subjective semantic tasks, consider integrating selective self-consistency and isotonic calibration. Applying stochastic sampling only to ambiguous inputs can cut inference costs by 45% while maintaining robustness. Furthermore, isotonic regression effectively corrects LLM biases toward extreme ratings, aligning outputs with human judgment. You should analyze your system's performance on both clear and ambiguous contexts to identify areas for improvement.

Key insights

The VerbaNexAI system efficiently rates word sense plausibility by ensembling LLMs with targeted sampling and bias correction.

Principles

• Targeted stochastic sampling reduces inference costs.
• Isotonic regression corrects LLM rating biases.
• Modeling multimodal human disagreement is challenging.

Method

Ensembles Llama-3.1 70B, Qwen-2.5 32B, Gemma-2 27B with few-shot chain-of-thought, selective self-consistency for ambiguous items, and isotonic regression for bias correction.

In practice

• Apply selective self-consistency to ambiguous inputs.
• Use isotonic regression to calibrate LLM outputs.
• Ensemble multiple LLMs for improved robustness.

Topics

• Word Sense Disambiguation
• Large Language Models
• Chain-of-Thought Prompting
• Self-Consistency
• Isotonic Regression
• SemEval

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning — A new metric quantifies LLM lexical bias from preference learning without manual curation.
• LoRi: Low-Rank Distillation for Implicit Reasoning — Reasoning trajectories in LLMs exhibit low-rank structure, enabling efficient distillation into compact latent processes.
• Adaptive Thinking: Large Language Models Know When to Think in Latent Space — Self-consistency can predict thinking necessity, enabling adaptive allocation of LLM reasoning budgets.
• Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs — Base LLMs exhibit emergent semantic calibration in Q&A, which instruction-tuning and chain-of-thought reasoning degrade.
• Emergent retokenization symmetry in large language models: phenomenology and applications — Large language models partially develop retokenization symmetry, allowing diverse outputs from semantically equivalent input segmentations.
• Reliable Chain-of-Thought via Prefix Consistency — Truncating and regenerating LLM Chain-of-Thought traces reveals a reliability signal for improved answer aggregation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.