Reading the mind of AI models: a Research Engineer’s analysis

2026-07-16 · Source: Naturallanguageprocessing on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

An analysis of Anthropic's Jacobian lens (J-Lens) for AI model monitoring reveals its production feasibility and associated costs. The J-Lens, which measures how earlier layers influence final outputs to access the model's "J-Space" of non-verbalized concepts, requires minutes-to-hours for fitting, with the cost determined by the earliest layer targeted. For GPT-2-medium (355M parameters, d=1024) on an L4 GPU, a 1000-prompt recipe takes approximately 6.7 hours, while 100 prompts achieve convergence in 40 minutes. Deployment overhead is under 2% of inference when monitoring a fixed concept dictionary (e.g., 1000 concepts over five layers). However, full-vocabulary readouts can cost up to 4x a forward pass for 24 layers on GPT-2-medium. The raw J-Lens underperforms the logit lens until a one-parameter correction (J + λI) is applied, which then surpasses logit lens faithfulness (0.294 vs 0.275 at layer 12). Middle layers are identified as optimal for both engineering efficiency and concept quality.

Key takeaway

For MLOps Engineers evaluating model monitoring solutions, the Jacobian lens (J-Lens) presents a highly efficient option for production. Deploying J-Lens with a fixed concept dictionary costs under 2% of inference, making it viable for real-time internal state monitoring. Consider applying the "J + λI" correction for smaller models to ensure faithfulness. Prioritize middle layers for both computational efficiency and concept quality, avoiding the higher fitting costs of earlier layers.

Key insights

The J-Lens offers a computationally feasible way to monitor internal AI model states, especially with fixed concept dictionaries.

Principles

• J-Lens fitting cost is determined by the earliest layer targeted.
• Fixed concept dictionaries enable near-free J-Lens monitoring.
• Earlier layers require more samples for Jacobian convergence.

Method

Compute Jacobian by running backprop once per output coordinate, stacking rows, and averaging across prompts (e.g., 100-1000). Apply at inference via matrix-vector products.

In practice

• Monitor specific concepts (e.g., violent content) at decode time.
• Apply "J + λI" correction for small models to improve faithfulness.
• Focus J-Lens application on middle layers for efficiency.

Topics

• Jacobian Lens
• AI Model Monitoring
• Transformer Circuits
• Inference Cost Optimization
• GPT-2-medium
• Internal Model States

Code references

• willkn/jlens_re

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Anthropic Just Read Claude’s Mind. Sort Of — The J-lens offers a novel window into LLMs' internal, unspoken conceptual states, revealing pre-output thoughts.
• Towards surfacing model algorithms with meta-tokens in the J-Space — J-lens can surface "meta-tokens" in LLMs, revealing internal algorithms and computational states.
• [AINews] not much happened today — AI inference efficiency, open model releases, and production-ready agentic workflows are rapidly advancing.
• Anthropic J-lens reveals hidden workspace inside Claude — Anthropic's Claude models spontaneously developed an internal "J-space" for reasoning, detectable via the Jacobian lens, impacting AI safety.
• The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked — Efficient small model inference requires both flexible model support and robust infrastructure for production deployment.
• No Space Like J-Space — The J-space is a functional global workspace in LLMs, containing verbalizable representations crucial for internal reasoning and output determination.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Naturallanguageprocessing on Medium.