Paradise at SemEval-2026 Task 5: On the Limitations of Surface-Level Features for Graded Word Sense Plausibility Prediction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

The "Paradise" system, developed by Dhruv Goyal, Ishita Gupta, and Jatin Bedi for SemEval-2026 Task 5, explores a simple approach to predict graded word sense plausibility in short, ambiguous narratives. This system utilizes 13 hand-crafted features, including text statistics, word-level similarity based on set comparisons, and measures of human annotator disagreement. Five diverse traditional machine learning models are combined via a weighted ensemble with minimal tuning. Despite its theoretical grounding in classical disambiguation methods and interpretable features, the system achieved essentially random performance on the official test set, reporting a Spearman correlation (ρ) of −0.038 and an accuracy within standard deviation of 0.542. This outcome highlights that surface-level lexical features are insufficient for accurate graded sense plausibility prediction without incorporating deep semantic representations, offering crucial baseline insights for future research in word sense disambiguation.

Key takeaway

For NLP Engineers developing word sense disambiguation systems, you should recognize the limitations of surface-level features. Your efforts to predict graded sense plausibility will likely fail without incorporating deep semantic representations. Prioritize models that capture nuanced contextual meaning over those relying solely on lexical statistics or basic similarity measures. This approach will yield more robust and accurate systems for complex semantic tasks.

Key insights

Surface-level lexical features are inadequate for graded word sense plausibility without deep semantic representations.

Principles

• Classical WSD features offer interpretability.
• Human disagreement signals can inform plausibility.
• Simple ensembles may not overcome feature limitations.

Method

The system combines 13 hand-crafted features (text statistics, set-based word similarity, annotator disagreement) with a weighted ensemble of five traditional ML models for graded word sense plausibility.

In practice

• Consider deep semantic features for WSD tasks.
• Use human disagreement as a feature.
• Establish baselines with simple models first.

Topics

• Word Sense Disambiguation
• SemEval-2026
• Graded Plausibility Prediction
• Lexical Features
• Machine Learning Ensembles
• Semantic Representations

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• sense2vec reloaded: contextually-keyed word vectors — The updated sense2vec model efficiently generates contextually-keyed word vectors, enabling rapid development of NLP applications like NER.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures — Evaluating LLM adaptability across diverse, low-resource language-culture pairs reveals critical performance and misalignment issues.
• [NLP Series] Why Words Don’t Mean Just One Thing: Ambiguity in Language — Ambiguity is a fundamental language feature that NLP systems must explicitly address across multiple linguistic levels.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.