On the scaling relationship between cloze probabilities and language model next-token prediction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Research by Cassandra L Jacobs and Morgan Grobol, presented at CoNLL 2026, investigates the scaling relationship between language model capacity and human cloze task performance. While larger language models demonstrate improved predictive power for eye movement and reading time data, their alignment with human production statistics in cloze tasks has been less understood. The study reveals that even the most advanced models under-allocate probability mass to human responses. However, larger models provide higher-quality estimates for next tokens and their production likelihood in cloze data. This improvement stems from their reduced sensitivity to lexical co-occurrence statistics and enhanced semantic alignment with human cloze responses. The findings suggest that increased memorization capacity in larger models aids in predicting semantically appropriate words, though it concurrently diminishes their sensitivity to low-level word recognition information.

Key takeaway

For NLP Engineers evaluating language models for human-centric tasks, recognize that larger models excel in semantic prediction for cloze tasks. Your model's increased capacity improves semantic alignment with human responses, but be aware this comes with reduced sensitivity to low-level lexical cues. Consider this trade-off when optimizing for tasks requiring nuanced word recognition versus broad semantic coherence.

Key insights

Larger language models improve cloze task prediction by prioritizing semantic alignment over lexical co-occurrence, despite under-allocating human response probabilities.

Principles

• Larger LMs prioritize semantic alignment.
• Increased memorization aids semantic prediction.
• Trade-off: semantic gain vs. low-level sensitivity.

Topics

• Language Models
• Cloze Task
• Next-Token Prediction
• Semantic Alignment
• Model Scaling

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Compute Optimal Tokenization — Optimal language model scaling depends on data in bytes, not tokens, with a compute-dependent optimal compression rate.
• Stanford AI Lab Papers and Talks at ACL 2022 — ACL 2022 showcased diverse NLP research from Stanford, spanning model pretraining, analysis, and evaluation.
• The $250K Inverse Scaling Prize and Human-AI Alignment — Inverse scaling reveals a core misalignment between LLM training objectives and desired human values.
• Semantic Chunking and the Entropy of Natural Language — A new statistical model explains natural language redundancy through hierarchical semantic chunking and predicts variable entropy rates.
• Human Evaluation of Large Language Models: How Good is Hugging Face’s BLOOM? — Human evaluation is crucial for assessing large language models' real-world performance beyond flawed academic benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.