Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Expert, quick

Summary

A reproducible audit investigated GPT-2 small's sparse-autoencoder (SAE) features on the Indirect Object Identification (IOI) task. On 300 prompts, GPT-2 small achieved 79.7% accuracy. The audit identified 146 of 24,576 layer-8 residual-stream SAE features from Bloom (2024) that fired differently on failed versus successful trials, with 105 showing a large effect size. Feature 17,491, labeled 'cryptographic keys', was the strongest correlate of failure (d=+2.93), activating primarily when the transferred object was 'the keys', leading to a 93.3% failure rate. Three controls demonstrated this feature is a correlate, not a sufficient cause, and that the SAE basis adds interpretability over raw residual stream predictive power. The primary contribution is the cheap, model-agnostic audit pipeline itself, which runs on a laptop. Code, the 300-prompt corpus, activation matrix, and scripts are released.

Key takeaway

For AI Scientists or Machine Learning Engineers debugging model failures, this audit pipeline offers a concrete method to identify and validate feature-level correlates. You should consider applying this cheap, model-agnostic approach to understand specific failure modes in your own models, especially when distinguishing between causal features and mere correlations. This can guide more targeted interventions than relying solely on raw activation analysis.

Key insights

A sparse-feature audit of GPT-2 small reveals task failure correlates, distinguishing causation from interpretability gains.

Principles

• Feature audits can surface named correlates of model failure.
• Correlation does not imply sufficient causation in model activations.
• SAE bases enhance interpretability without necessarily boosting predictive power.

Method

The audit pipeline identifies sparse-autoencoder features firing differently on task failures, then applies causal ablation, representation baseline, and seed-robustness checks to validate findings.

In practice

• Use the released audit pipeline to analyze other model failures.
• Examine feature 17,491's behavior in similar language models.

Topics

• Sparse Autoencoders
• GPT-2 Small
• Model Interpretability
• Task Failure Analysis
• Mechanistic Interpretability
• Indirect Object Identification

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models — The five-stage methodology links correlation to causation in Transformer feature analysis, revealing partial causality and robustness gaps.
• I built a tool that shows you what GPT-2 is "thinking" in real-time as it generates 3D graph of concept activations per token [R] — AXON visualizes LLM internal states via real-time 3D graphs of SAE feature activations, revealing pre-token generation dynamics.
• Assessing Reliability of Symbol Detection in Concept Bottleneck Models — CBMs' high accuracy doesn't guarantee reliable symbol detection; independent component swapping reveals shortcut reliance.
• From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability — A framework certifies SAE-based LM interpretability by bounding risk using proxy fidelity, reconstruction, mismatch, and complexity.
• The Neglected Baseline in Model Interpretation — Neglecting baselines in model interpretation leads to imprecise results; a clear baseline and attribution error are crucial for accuracy.
• “Features” aren’t always the true computational primitives of a model, but that might be fine anyways — Features in mechanistic interpretability exist on a spectrum from memorization to computational primitives.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.