Team Habib Disambiguators at SemEval-2026 Task 5: Assessing Semantic Plausibility using Regularized Transformer Fine-Tuning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

Team Habib Disambiguators developed a system for SemEval-2026 Task 5, which focuses on rating the plausibility of word senses in ambiguous sentences within short stories where narrative context resolves the ambiguity. Their approach models this as a regression problem, fine-tuning a DeBERTa-v3 transformer to predict the distribution of human judgments rather than a single hard label. To counter limited training data and potential overfitting, the system incorporates R-Drop (Consistency Regularization) for prediction stability and Layer-wise Learning Rate Decay (LLRD) to preserve pre-trained linguistic knowledge. This combination of soft-label distribution and aggressive regularization improved generalization, achieving a Spearman correlation of 0.56 and an Accuracy (within SD) of 0.74 on the official test set.

Key takeaway

For NLP Engineers developing systems for semantic plausibility or word sense disambiguation, consider modeling the problem as a regression task predicting soft-label distributions. You should integrate aggressive regularization techniques like R-Drop and Layer-wise Learning Rate Decay during transformer fine-tuning, especially when working with limited training data. This approach can significantly improve generalization and robustness in handling ambiguous contexts.

Key insights

Treating semantic plausibility as a soft-label distribution with aggressive regularization improves generalization.

Principles

• Soft-label distributions enhance generalization for ambiguous samples.
• Aggressive regularization combats limited training data and overfitting.

Method

Fine-tuning a DeBERTa-v3 transformer for regression, predicting human judgment distributions, and applying R-Drop and Layer-wise Learning Rate Decay.

In practice

• Apply R-Drop for prediction stability across dropout masks.
• Utilize LLRD to preserve pre-trained model knowledge.

Topics

• SemEval-2026 Task 5
• Semantic Plausibility
• Word Sense Disambiguation
• Transformer Fine-Tuning
• DeBERTa-v3
• R-Drop
• Layer-wise Learning Rate Decay

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction — Soft-target loss fine-tuning significantly improves LLM and MLM performance in continuous value prediction tasks.
• An End-to-End Hybrid Framework for Rumour Detection in Low-Resources Algerian Dialect — A hybrid framework combining transformer embeddings with classical classifiers enables effective rumour detection in low-resource Algerian dialect.
• Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization — Formulaic expression desensitization and context-enhanced transformers improve problem/method sentence extraction from scientific papers, outperforming baselines.
• From Raw Text to Smart Predictions: A Beginner-Friendly Guide to the Complete NLP Pipeline — An NLP pipeline transforms raw text into machine-understandable numerical representations through systematic cleaning and vectorization.
• IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources — Semantic deduplication and domain-balanced pretraining significantly enhance Persian PLM performance across diverse NLU tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.