CrosSing: Cross-Scale Reasoning Evaluation on LLMs against Humans

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The study introduces CrosSing, a new dataset designed to evaluate large language models' (LLMs) capacity for semantic reasoning, specifically their grasp of scalar relationships in gradable adjectives across lexical scales. Researchers established a human baseline and probed how LLM understanding is affected by overinformative contexts. Evaluating ten high-performing LLMs, the study found that some models surpassed human performance when no extra information was present. However, LLM performance deteriorated in specific overinformative contexts, contrasting sharply with human performance, which significantly improved under the same conditions. This divergence highlights a fundamental difference between current LLMs and humans in processing scalar adjective relationships and adapting to overinformative linguistic environments.

Key takeaway

For NLP Engineers developing or deploying LLMs for tasks requiring nuanced semantic understanding, you should be aware that current models may struggle with scalar reasoning, particularly when presented with overinformative linguistic contexts. Your evaluation pipelines should include tests for such scenarios, as LLMs can perform worse than humans where human performance improves. Consider fine-tuning or architectural adjustments to improve robustness in complex, context-rich language environments.

Key insights

LLMs struggle with scalar reasoning in overinformative contexts where humans excel, revealing a fundamental difference.

Principles

• LLMs' scalar reasoning differs fundamentally from humans'
• Overinformative contexts can degrade LLM performance

Method

The study introduced the CrosSing dataset, providing a human baseline to evaluate ten LLMs' understanding of gradable adjectives and their scalar relationships, especially under overinformative contexts.

In practice

• Test LLMs with overinformative linguistic inputs
• Focus on scalar reasoning in semantic tasks

Topics

• Large Language Models
• Semantic Reasoning
• Scalar Relationships
• Gradable Adjectives
• CrosSing Dataset
• Overinformative Contexts
• Human-LLM Comparison

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning — Human and LLM everyday reasoning share pattern-matching mechanisms, leading to similar error patterns.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• Why isn’t LLM reasoning done in vector space instead of natural language?[D] — LLMs internally use vectors, but natural language reasoning offers crucial interpretability and debuggability.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.
• Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations — LLM robustness to CoT perturbations varies significantly by type and model scale, challenging assumptions that size alone ensures reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.