ChulaNLP at SemEval-2026 Task 5: Regression-Calibrated LLM for Word-Sense Scoring

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

ChulaNLP presented a regression-calibrated Large Language Model (LLM) for SemEval-2026 Task 5, focusing on word-sense plausibility rating. This task moves beyond traditional Word Sense Disambiguation (WSD) classification by requiring models to estimate the likelihood of multiple senses in context, aligning with graded human judgments. The team used GlossBERT and BEM as encoder-based baselines, observing that LLMs inherently provide more accurate plausibility estimates. Their proposed model enhances raw LLM outputs by applying linear regression for post-hoc adjustment, better matching human annotation patterns. This calibrated system achieved the highest within-standard-deviation accuracy among the evaluated approaches, demonstrating the substantial performance improvement lightweight calibration offers for graded semantic judgment tasks.

Key takeaway

For NLP engineers developing models for nuanced semantic understanding, consider integrating post-hoc calibration techniques. Your LLM's raw outputs for graded tasks like word-sense plausibility can be substantially improved by applying a simple linear regression. This approach, demonstrated by ChulaNLP at SemEval-2026 Task 5, offers a lightweight method to align model predictions more closely with human judgments, potentially boosting accuracy without extensive model retraining.

Key insights

Lightweight linear regression calibration significantly improves LLM performance on graded word-sense plausibility tasks.

Principles

• WSD benefits from plausibility rating.
• LLMs excel at graded semantic judgment.
• Post-hoc calibration enhances LLM accuracy.

Method

A regression-calibrated LLM model applies linear regression to adjust raw LLM outputs, aligning them with human annotation patterns for improved plausibility estimates.

In practice

• Use LLMs for graded semantic tasks.
• Apply linear regression for output calibration.
• Evaluate with within-standard-deviation accuracy.

Topics

• Word Sense Disambiguation
• Large Language Models
• Semantic Evaluation
• Regression Calibration
• Plausibility Rating
• SemEval-2026 Task 5

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs — Base LLMs exhibit emergent semantic calibration in Q&A, which instruction-tuning and chain-of-thought reasoning degrade.
• A Larger Annotated Corpus of Portuguese Coreference — LLMs can annotate Portuguese coreference, but a new LLM-based system did not outperform existing rule-based methods.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.