Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

This study investigates pragmatic reasoning in large language models (LLMs), addressing the challenge that evaluation methods can significantly alter observed model behavior. It specifically examines whether prompt-based judgments align with models' internal probability distributions. Following Hu & Levy (2023), the research compares direct probability measurement and metalinguistic prompting, utilizing scalar diversity as a graded diagnostic. Findings indicate that neither evaluation method consistently outperforms the other, and pragmatic behavior varies substantially across different model families, prompting strategies, and task structures. Crucially, scalar diversity gradients emerged only in specific model-condition combinations, suggesting LLM pragmatic reasoning results from an interaction between internal probabilistic representations and task-induced prompting, rather than a stable, universally captured competence. These results underscore the critical role of evaluation design in interpreting LLM pragmatic abilities.

Key takeaway

For NLP Engineers evaluating the pragmatic reasoning capabilities of large language models, you should critically assess your evaluation design. Recognize that observed performance reflects an interaction between the model's internal representations and your chosen prompting strategy and task structure. Do not assume a single evaluation paradigm captures stable competence; instead, consider employing diverse methods like direct probability measurement and metalinguistic prompting to gain a more nuanced understanding of LLM pragmatic abilities.

Key insights

LLM pragmatic reasoning is an interaction between internal representations and task-induced behavior, not stable competence.

Principles

• LLM pragmatic behavior varies by model, prompt, and task.
• Evaluation method significantly impacts observed LLM performance.
• Scalar diversity serves as a graded diagnostic for pragmatic inference.

Method

The study compares direct probability measurement and metalinguistic prompting, using scalar diversity, across various LLMs and experimental settings to evaluate pragmatic inference.

In practice

• Use scalar diversity as a diagnostic for pragmatic inference.
• Compare direct probability and metalinguistic prompting for evaluation.

Topics

• Large Language Models
• Pragmatic Reasoning
• LLM Evaluation
• Scalar Diversity
• Prompt Engineering
• Metalinguistic Prompting

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses — LLMs significantly struggle to infer pragmatic meaning from non-verbal communication, unlike their performance with verbal cues.
• Benchmarks in Leipzig — Iterative, multi-stage evaluation reveals impressive mathematical reasoning capabilities in Large Language Models.
• The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans — LLMs struggle with flexible reasoning, often conflating pattern matching with genuine understanding, unlike humans who adapt reasoning strategies.
• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• Human Evaluation of Large Language Models: How Good is Hugging Face’s BLOOM? — Human evaluation is crucial for assessing large language models' real-world performance beyond flawed academic benchmarks.
• Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models? — Large Language Models exhibit surprising sensitivity to semantically irrelevant "spurious prompts," which can systematically steer their behavior.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.