Lexical Familiarity Predicts Processing Depth for Nonliteral Language in Large Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

A study investigated how large language models internally process nonliteral language. Analyzing five categories, including slang, metaphor, and idioms, across all 48 layers of Gemma-3-12B-IT using Gemma Scope 2 sparse autoencoders, researchers found a lexical familiarity gradient. This gradient indicates that processing depth depends on prior lexical knowledge, not the figurative type. Idioms diverge at Layer 1 as entrenched units, while expressions from familiar words like metaphors, semantic-shift, and constructional slang converge at Layers 7-9. Neologisms peak at Layer 41, activating three times more unique features. Paraphrase residual analysis confirmed strong signals at these gradient endpoints, revealing a three-tier hierarchy: entrenched retrieval, known-word reanalysis, and novel-word construction. This peak-layer structure replicates in base models such as Gemma-PT and Qwen-Base, suggesting it's a robust property of pretrained representations, not an alignment artifact. The study also identified an activation density confound in SAE feature counts.

Key takeaway

For NLP Engineers developing or fine-tuning large language models, understanding the lexical familiarity gradient is crucial. Your models process nonliteral language like idioms at L1 and neologisms at L41 based on prior lexical knowledge, not just figurative type. This insight should guide your strategies for improving robustness to non-standard language and refining interpretability using sparse autoencoders, ensuring more effective handling of diverse linguistic inputs.

Key insights

LLMs process nonliteral language based on lexical familiarity, not figurative type, across a three-tier hierarchy.

Principles

• Lexical familiarity dictates nonliteral language processing depth.
• Pretrained representations exhibit a robust processing depth gradient.
• Activation density can confound SAE feature counts.

Method

Analyzed 48 layers of Gemma-3-12B-IT with Gemma Scope 2 sparse autoencoders across five nonliteral language categories. Used paraphrase residual analysis to confirm processing signals.

In practice

• Improve LLM robustness to non-standard language.
• Enhance SAE-based interpretability for nonliteral language.

Topics

• Large Language Models
• Nonliteral Language Processing
• Lexical Familiarity
• Sparse Autoencoders
• Gemma-3-12B-IT
• Model Interpretability

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Interpreting Brain Responses to Language with Sparse Features from Language Models — Sparse autoencoder features and surprisal can interpret human language cortex responses to language.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• Revisiting the Systematicity in Negation in the Era of In-Context Learning — LLMs partially grasp negation via in-context learning, but struggle with robust scope recognition and function vector composition for it.
• The Hidden Fractal Structure of Language — Language's inherent fractal structure, with self-similarity and long-range dependence, explains LLMs' emergent reasoning from next-token prediction.
• Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models — Automated metrics can identify and quantify lexical overuse and preference learning shifts in LLMs without manual curation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.