Information-Theoretic Storage Cost in Sentence Comprehension

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Social Sciences & Behavioral Studies, Mathematics & Computational Sciences, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new study introduces an information-theoretic measure for processing storage cost in real-time sentence comprehension, addressing the significant load on working memory. This measure quantifies the information previous words carry about future context under uncertainty, distinguishing itself from prior discrete, grammar-based metrics by being continuous, probabilistic, and theory-neutral. Crucially, it can be estimated using pre-trained neural language models. The approach's validity is demonstrated through three analyses in English: it accurately recovers known processing asymmetries in center embeddings and relative clauses, correlates with grammar-based storage costs in a syntactically-annotated corpus, and predicts reading-time variance in two large-scale naturalistic datasets more effectively than traditional information-based predictors.

Key takeaway

For research scientists developing or evaluating language models, this information-theoretic measure offers a robust, continuous, and probabilistic method to quantify working memory load during sentence comprehension. You should consider integrating this approach to better model human cognitive processing, potentially leading to more human-like language understanding capabilities and improved predictions of reading behavior in naturalistic settings.

Key insights

A new information-theoretic measure quantifies working memory load in sentence comprehension using neural language models.

Principles

• Processing storage cost can be continuous and probabilistic.
• Information theory offers a theory-neutral formalization for cognitive load.
• Neural language models can estimate psycholinguistic processing costs.

Method

The proposed method measures processing storage cost as the amount of information previous words carry about future context under uncertainty, estimable from pre-trained neural language models.

In practice

• Analyze processing asymmetries in sentence structures.
• Correlate with existing grammar-based storage costs.
• Predict reading-time variance in naturalistic text.

Topics

• Information Theory
• Sentence Comprehension
• Working Memory Load
• Neural Language Models
• Psycholinguistics
• Reading Time Prediction

Code references

• kohei-kaji/info-storage

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• Measuring the entropy of English — Shannon's entropy estimation for English revealed that true compressibility requires understanding human prediction, not just statistical analysis.
• Heterogeneous Neural Predictivity from Language Models During Naturalistic Comprehension — Language model features effectively predict neural activity during naturalistic comprehension, distinct from shared computational claims.
• Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue — Reflective memory in long-horizon dialogue requires benchmarks and hierarchical frameworks beyond factual recall.
• Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data — UDDMs behave as associative memories, exhibiting a dataset-size-dependent memorization-to-generalization transition detectable via conditional entropy.
• Predicting Cognitive Load from Speech and Interaction Dynamics in Dyadic Conversations — Conversational interaction dynamics reliably predict perceived cognitive load in natural dyadic settings.
• Probing for Reading Times — Early language model layers predict initial human reading effort better than scalar surprisal, which excels at later stages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.