Word predictability estimates from language models are not robust to tokenizer vocabulary

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies · Depth: Advanced, quick

Summary

A recent study investigates the robustness of language model-derived word predictability estimates, specifically focusing on the impact of BPE tokenizer vocabulary size. Researchers examined how vocabulary size influences the linguistic plausibility of learned subword units, the surprisal estimates generated by models, and the quality of downstream reading time predictions. The findings indicate that while tokenizer vocabulary size does not substantially alter the rate of morphologically reasonable tokenizations, it significantly affects surprisal estimates and reading time predictions across 5-gram, LSTM, and GPT-2 language models. These discrepancies primarily manifest in words that are split by the tokenizer, highlighting a critical consideration for psycholinguists designing stimuli for computational modeling.

Key takeaway

For psycholinguists and NLP engineers using language models for cognitive modeling, you must account for tokenizer vocabulary effects. Your predictability estimates and reading time predictions can be significantly skewed, especially for words split into subword units. Carefully design your experimental stimuli with subword tokenization in mind to ensure the robustness and validity of your computational modeling results.

Key insights

Language model predictability estimates and reading time predictions are significantly impacted by tokenizer vocabulary size, particularly for words split into subword units.

Principles

• LM predictability estimates lack robustness to tokenizer vocabulary.
• Surprisal estimates are influenced by vocabulary size.
• Split words are most affected by tokenization differences.

Method

Researchers investigated the effect of BPE tokenizer vocabulary size on subword unit plausibility, surprisal estimates, and downstream reading time predictions using 5-gram, LSTM, and GPT-2 language models.

In practice

• Design psycholinguistic stimuli considering subword tokenization.
• Account for tokenizer effects in LM-based cognitive modeling.

Topics

• Language Models
• Tokenizer Vocabulary
• Surprisal Estimates
• Reading Time Predictions
• Psycholinguistics
• Subword Tokenization

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew — Tokenization strategy critically impacts Transformer models' ability to represent complex morphology, especially in non-concatenative languages.
• Probing for Reading Times — Early language model layers predict initial human reading effort better than scalar surprisal, which excels at later stages.
• How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation — LLMs exhibit high surprisal and tokenization costs with historical Italian, yet retain robust semantic understanding, mitigable with temporal context.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• What is Tokenization? All you need to know — Tokenization is a critical NLP decision impacting model efficiency, robustness, and language representation, not just a preprocessing step.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.