CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

The CiNet-Handai-Kyodai team developed a hybrid system for SemEval-2026 Task 5, addressing the challenge of graded word-sense plausibility in narrative contexts. Their innovative approach combines prompt-based large language model (LLM) scoring with three complementary features. These include semantic embedding similarity, story-conditioned definition generation, and a synthetic gaze signal based on predicted fixation time. An ordinary least squares regressor integrates these diverse signals to produce the final plausibility scores. On the official test set, the system achieved a 90.10 Acc±SD and a 79.19 Spearman correlation. This performance notably surpassed the reported human reference score on Acc±SD. It highlights the significant value of combining LLM-based judgments with targeted linguistic and cognitive-inspired features for complex semantic tasks.

Key takeaway

For NLP Engineers developing systems for nuanced semantic understanding, especially word-sense disambiguation, consider hybrid approaches that augment LLM outputs. Integrating features like semantic embedding similarity, story-conditioned definition generation, and synthetic cognitive signals such as gaze can significantly boost performance. This strategy achieved 90.10 Acc±SD, surpassing human reference scores. It suggests combining diverse data types with LLM judgments yields more robust, accurate models for complex linguistic tasks.

Key insights

Combining LLM judgments with linguistic and cognitive features improves word-sense plausibility.

Principles

• Hybrid systems outperform single approaches.
• Cognitive signals enhance semantic tasks.
• LLMs benefit from feature augmentation.

Method

The system uses an ordinary least squares regressor to combine prompt-based LLM scores, semantic embedding similarity, story-conditioned definition generation, and synthetic gaze signals.

In practice

• Integrate gaze signals for context understanding.
• Augment LLM outputs with semantic embeddings.
• Generate definitions to refine sense disambiguation.

Topics

• Large Language Models
• Word Sense Disambiguation
• Semantic Similarity
• Cognitive Computing
• SemEval
• Hybrid AI Systems

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Multi-View Attention Multiple-Instance Learning Enhanced by LLM Reasoning for Cognitive Distortion Detection — Combining LLMs with MIL and ELB components improves cognitive distortion detection by enhancing interpretability and reasoning.
• DySem: Uncovering Dynamic Semantic Components via Multilingual Consensus for Calculating Semantic Textual Similarity — DySem uses multilingual consensus to find dynamic, sample-specific semantic dimensions for improved STS in LLMs.
• Revisiting the Systematicity in Negation in the Era of In-Context Learning — LLMs partially grasp negation via in-context learning, but struggle with robust scope recognition and function vector composition for it.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction — LLMs struggle with complex linguistic constructions like CMC, necessitating novel corpus creation methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.