Probing for Reading Times

2026-04-20 · Source: Computation and Language · Field: Science & Research — Social Sciences & Behavioral Studies, Mathematics & Computational Sciences, Computational Linguistics & Cognitive AI · Depth: Expert, quick

Summary

This research investigates whether language model (LM) representations encode human reading times, a cognitive signal. Utilizing regularized linear regression on two eye-tracking corpora across five languages (English, Greek, Hebrew, Russian, Turkish), the study compares LM representations from each layer against scalar predictors like surprisal, information value, and logit-lens surprisal. Findings indicate that early LM layers surpass surprisal in predicting early-pass reading measures such as first fixation and gaze duration. This suggests that low-level structural or lexical representations in LMs capture human-like processing signatures, aligning model depth with early temporal stages of human reading. However, for late-pass measures like total reading time, scalar surprisal proves superior. Combining surprisal with early-layer representations also yields performance gains, though the optimal predictor varies significantly by language and eye-tracking measure.

Key takeaway

For NLP Engineers developing models intended to mimic human cognitive processing, you should consider that early-layer representations are more indicative of initial human reading behaviors (e.g., first fixation), while scalar surprisal remains critical for predicting overall reading duration. Integrating both types of predictors could enhance the model's ability to simulate human reading, but you must account for language-specific performance variations.

Key insights

Early LM layers predict early human reading times, while scalar surprisal predicts late reading times.

Principles

• Early LM layers capture low-level processing signatures.
• Model depth aligns with human reading's temporal stages.

Method

Regularized linear regression was used to compare language model representations against scalar predictors on eye-tracking corpora across five languages.

In practice

• Combine surprisal with early-layer representations.
• Consider language-specific predictor variations.

Topics

• Language Model Probing
• Human Reading Times
• Eye-tracking Corpora
• Surprisal Predictors
• Early Layer Representations

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• Probing for Reading Times — Early language model layers predict initial human reading effort better than scalar surprisal, which excels at later stages.
• Modeling semantic association in self-paced reading with language model embeddings — Methodological choices, especially embedding model type, critically impact semantic association quantification in reading comprehension.
• Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs — LLMs often match human linguistic judgments via surface patterns, not deep pragmatic reasoning.
• N-Grams and Markov Assumptions: The First Predictive Models of Language — N-grams were the first predictive language models, using statistical counts over short word sequences to anticipate the next word.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• The Dynamics of Human and AI-Generated Language: How Semantics Fluctuates across Different Timescales — The ACW-0 semantic timescale analysis effectively distinguishes temporal organization of generic versus specific content in human and AI speech.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.