The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study by Omar Momen and Sina Zarrieß investigates the relationship between language-model (LM) surprisal, lexical frequency, and metaphor novelty judgments. The research addresses the confound where LM surprisal, often used as a proxy for contextual predictability, is tightly linked to word frequency. Analyzing surprisal estimates from eight Pythia model sizes and 154 training checkpoints, alongside two different word frequency measures, the authors found that word frequency consistently serves as a stronger predictor of metaphor novelty than surprisal. Furthermore, the association between surprisal and novelty peaks early in the model's training stages before declining, a pattern that mirrors the increasing association between surprisal and frequency. These findings suggest that previous interpretations of optimal LM surprisal settings might have mistakenly attributed metaphor novelty and processing difficulty to contextual predictability, when lexical frequency is likely the primary underlying factor.

Key takeaway

For research scientists evaluating language model outputs for semantic properties like metaphor novelty, you should prioritize lexical frequency as a primary explanatory variable over LM surprisal. This research indicates that surprisal's correlation with novelty is often a confound of frequency, especially in later training stages. Re-evaluating your experimental designs to account for this frequency confound will lead to more accurate interpretations of contextual predictability and processing difficulty in linguistic phenomena.

Key insights

The study reveals lexical frequency, not LM surprisal, is the primary predictor of metaphor novelty, challenging prior assumptions.

Principles

• Lexical frequency strongly predicts metaphor novelty.
• LM surprisal's novelty association diminishes with training.
• Surprisal and frequency associations are intertwined.

Method

The study analyzed LM surprisal from eight Pythia model sizes and 154 training checkpoints, correlating it with metaphor novelty ratings using two word frequency measures.

Topics

• Language Models
• Metaphor Novelty
• Lexical Frequency
• LM Surprisal
• Pythia Models
• Contextual Predictability

Best for: AI Scientist, Research Scientist

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Probing for Reading Times — Early language model layers predict initial human reading effort better than scalar surprisal, which excels at later stages.
• Semantic Chunking and the Entropy of Natural Language — A new statistical model explains natural language redundancy through hierarchical semantic chunking and predicts variable entropy rates.
• Floating or Suggesting Ideas? A Large-Scale Contrastive Analysis of Metaphorical and Literal Verb-Object Constructions — Metaphorical and literal language differences are largely construction-specific, not universally consistent.
• Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages — Idiomatic expressions challenge multilingual NLP, with literal meanings and low-resource languages posing significant hurdles.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.