Abstraction Induces the Brain Alignment of Language and Speech Models

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Science & Research — Artificial Intelligence & Machine Learning, Life Sciences & Biology, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This research demonstrates that the alignment between large language models (LLMs), speech audio models, and human brain activity stems from shared meaning abstraction, not next-word prediction. The study found that intermediate hidden layers, rather than output layers, are most effective at predicting fMRI and ECoG signals. This effectiveness correlates strongly with a peak in the layerwise "intrinsic dimension" (I_d), a measure of feature complexity. Models like OPT (125m, 1.3b, 13b), Pythia (160m, 410m, 6.9b), WavLM, and Whisper exhibit this I_d peak in their middle layers, coinciding with the construction of higher-order linguistic features. Finetuning models to better predict brain activity causally increases both their I_d and semantic content, reinforcing that rich meaning abstraction is the primary driver of model-brain similarity.

Key takeaway

For AI Scientists developing language or speech models, recognize that optimizing solely for next-token prediction may not yield the most brain-aligned or semantically rich representations. You should focus on architectural designs and training objectives that foster high intrinsic dimension and meaning abstraction in intermediate layers. Consider finetuning models directly on neural response data to enhance their semantic content and improve their correspondence with human brain processing, moving beyond autoregressive loss limitations.

Key insights

Brain-model alignment in language and speech processing is driven by meaning abstraction, not next-token prediction.

Principles

• Intermediate model layers exhibit peak intrinsic dimension (I_d).
• High I_d correlates with rich semantic feature building.
• Next-token prediction layers are less brain-aligned.

Method

The study measured model-to-brain encoding performance, computed layerwise intrinsic dimension (I_d) using GRIDE, performed layerwise linguistic probing, and calculated next-token surprisal.

In practice

• Prioritize intermediate layers for brain-aligned representations.
• Monitor intrinsic dimension (I_d) to identify abstraction phases.
• Finetune models on neural data to enhance semantic richness.

Topics

• Language Models
• Speech Models
• Brain-Computer Interfaces
• Intrinsic Dimension
• Neural Encoding Models
• Semantic Abstraction

Code references

• sissa-data-science/DADApy
• aalok-sathe/surprisal
• HuthLab/encoding-model-scaling-laws
• facebookresearch/SentEval

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• Do Large Language Models Acquire Phrase-Based Processing? Evidence from Eye Movements and Model-Brain Alignment After Fine-Tuning — Targeted fine-tuning enables LLMs to acquire human-like phrase-based processing, improving model-brain alignment.
• How Major Reasoning Models Converge to the Same “Brain” as They Model Reality Increasingly Better — AI models, regardless of modality, converge on a unified internal representation of reality as they scale.
• [R] Empirical evidence for a primitive layer in small language models — 18 experiments across 4 architectures — Small language models exhibit a consistent activation gap between scaffolding and content semantic primitives.
• Gemma 4 is not your standard transformer — Gemma 4's architectural deviations suggest a re-evaluation of core transformer assumptions, particularly regarding token information flow.
• Day 1: Understanding Large Language Models: A Journey From Words to AI — Large Language Models achieve complex capabilities by scaling a simple objective: predicting the next token in a sequence.
• Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometry — NLLB-200's internal geometry reflects language-universal conceptual structures, akin to human multilingual cognition.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.