I built a tool that shows you what GPT-2 is "thinking" in real-time as it generates 3D graph of concept activations per token [R]

2026-05-19 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

AXON is a new tool designed to visualize GPT-2's internal processing in real-time, presenting its "thinking" as a live 3D force graph of concept activations per token. It operates by passing GPT-2's residual stream through Joseph Bloom's pretrained Sparse Autoencoder (SAE), which decomposes it into human-interpretable features such as "European geography" or "French language." These features are streamed via WebSocket to a browser, where they appear as nodes in a graph, with edges connecting co-activated features and node brightness indicating activation strength. A notable observation is how features like "geography" and "proper noun" activate before a word like "Paris" is generated. The tool's stack includes TransformerLens, SAELens, FastAPI WebSocket, Three.js, and 3d-force-graph, running at ~800ms/token on CPU or ~35ms on a 4050 GPU. It supports various models like GPT-2 variants, Pythia, and Gemma-2-2B, leveraging Neuronpedia's API for feature labels.

Key takeaway

For AI Scientists and Research Scientists focused on mechanistic interpretability, AXON provides a highly intuitive and dynamic approach to understanding LLM internal states. You should consider integrating this real-time 3D visualization tool to observe how concept activations evolve before token generation, offering tangible insights into hidden state dynamics. This interactive workflow can make complex interpretability research more accessible and potentially accelerate the discovery of causal links within models.

Key insights

AXON visualizes LLM internal states via real-time 3D graphs of SAE feature activations, revealing pre-token generation dynamics.

Principles

• Mechanistic interpretability benefits from dynamic visualization.
• SAEs decompose residual streams into interpretable features.
• Pre-generation feature activation reveals model's internal state.

Method

Pass LLM residual stream through a pretrained Sparse Autoencoder (SAE). Stream decomposed human-interpretable features via WebSocket. Visualize features as a live 3D force graph, updating per token with co-activation edges.

In practice

• Observe feature activation before token generation.
• Swap in different LLMs with available SAEs.
• Explore co-activation patterns between features.

Topics

• Mechanistic Interpretability
• Large Language Models
• Sparse Autoencoders
• Real-time Visualization
• GPT-2
• TransformerLens
• Three.js

Code references

• 09Catho/axon

Best for: AI Scientist, Research Scientist, AI Engineer

Related on AIssential

• [P] Visualizing token-level activity in a transformer — A developer is experimenting with a 3D visualization tool designed to make Large Language Model (LLM) inference more intuitive by animating activation paths across a network of components like attenti…
• Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics — Interpreting scientific foundation models reveals internal mechanisms are often inconsistent with known physics, posing evaluation challenges.
• Circuit Tracing in Autoregressive Protein Language Models — ProGenMech offers a mechanistic interpretability framework for generative protein language models, revealing underlying biological circuits.
• Nobody Knows What’s Inside a Neural Network. A Few People Are Trying to Find Out. — A small group of researchers is undertaking "biology on language models," dissecting them like organisms to uncover their internal mechanisms, a process revealing unexpected findings.
• Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification — A sparse-feature audit of GPT-2 small reveals task failure correlates, distinguishing causation from interpretability gains.
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models — Leveraging LLM internal causal mechanisms to guide data selection significantly boosts fine-tuning efficiency and performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.