Do Language Models Track Entities Across State Changes?

2026-05-28 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent study investigates how transformer language models (LMs) perform entity tracking (ET) in complex natural language scenarios involving multiple state-changing operations like PUT, REMOVE, and MOVE. Researchers found that LMs do not incrementally track world states across tokens or query-relevant states across layers. Instead, LMs aggregate all relevant information in parallel at the final token once the query becomes clear. The investigation further revealed that LMs implement the REMOVE operation using a fragile global suppression tag, a mechanism that predicts various behavioral failure modes confirmed by the study. A mechanistic solution involving nullifying this tag was proposed to partially mitigate the issue. Overall, the findings indicate LMs employ a non-sequential strategy to solve a fundamentally sequential task, demonstrating the interplay between behavioral and mechanistic analyses.

Key takeaway

For NLP Engineers designing systems requiring robust entity tracking, understand that current transformer LMs do not incrementally track state. Your models likely aggregate information in parallel, making them susceptible to specific failure modes, particularly with REMOVE operations due to fragile global suppression tags. You should consider these non-incremental processing characteristics when debugging or designing systems, and explore methods like nullifying suppression tags to improve reliability in state-changing contexts.

Key insights

Language models tackle sequential entity tracking non-incrementally, aggregating information in parallel, and exhibit fragile removal mechanisms.

Principles

• LMs solve sequential tasks non-sequentially.
• Global suppression tags cause LM removal failures.
• Behavioral and mechanistic analyses interact fruitfully.

Method

Researchers investigated LM mechanisms for PUT, REMOVE, and MOVE operations, characterizing non-incremental ET. A mechanistic solution of nullifying the global suppression tag was proposed for REMOVE.

In practice

• Nullify the global suppression tag for REMOVE.
• Inform mechanistic hypotheses with behavioral results.
• Build stronger evaluations from mechanistic insights.

Topics

• Language Models
• Entity Tracking
• Transformer Architectures
• Mechanistic Interpretability
• State Changes
• Failure Modes

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Large Language Models in NLP — LLMs integrate NLP advancements with massive scale to create general-purpose, prompt-driven language systems.
• Further Notes on Our Recent Research on AI Delegation and Long-Horizon Reliability — LLMs can accumulate semantic degradation in long, delegated workflows, despite strong short-horizon performance.
• Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms — LLM factual recall is sensitive to entity surface forms, highlighting the need for diverse evaluation methods.
• Position: The Turing-Completeness of Real-World Autoregressive Transformers Relies Heavily on Context Management — Real-world autoregressive Transformer Turing-completeness critically depends on context management within a fixed system.
• Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models — LLM unpredictability stems from numerical instability and chaotic error propagation in early Transformer layers.
• Words Don’t Have Meaning. Sentences Do. — The Transformer architecture, powered by self-attention, revolutionized language processing by enabling parallel word analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.