Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Natural-language autoencoders, used for explaining hidden activations by reconstruction, are shown to be structurally insensitive to individual false claims. A Qwen-2.5-7B verbalizer reconstructs well, yet only ~2% of claims are reconstruction-dependent, indicating it captures gist over specific facts. Under synthetic ground truth, standard methods develop co-adapted private codes. To address this, two audit protocols (grounded-vs-true cross, evaluator swap) and RECAP (Readable Encodings via Co-trained Auxiliary Predictors) are introduced. RECAP trains linear heads alongside the target model to keep designated content decodable. On RECAP-trained sandbox models and Pythia-160M, designated content becomes reliably probe-decodable, improving truthfulness (0.44-0.46 vs near-zero control). For AI safety, RECAP makes internal content independently checkable. A probe achieves AUC 0.96 (vs 0.82 without RECAP) and flags lies against adversaries (AUC 0.95 vs 0.51 control).

Key takeaway

For AI Security Engineers evaluating or building interpretable AI systems, recognize that standard activation explanation methods are unreliable for specific claims. You should adopt RECAP and its audit protocols to ensure internal model content is independently verifiable, which is crucial for robust AI safety and adversarial resilience. This approach helps prevent models from gaming explanations.

Key insights

Current activation explanation methods are flawed; RECAP offers a verifiable alternative for internal model content.

Principles

• Reconstruction scores track gist, not specific facts.
• Co-adapted private codes can develop in explanation systems.
• Independent probes can verify internal model content.

Method

RECAP trains linear heads alongside the target model to keep designated content decodable, enabling independent verification.

In practice

• Use grounded-vs-true cross audit protocol.
• Employ evaluator swap audit protocol.
• Implement RECAP for verifiable internal content.

Topics

• Activation Explanations
• Model Interpretability
• AI Safety
• Verifiable AI
• Natural Language Autoencoders
• RECAP

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations — Reconstruction-based NLA faithfulness tests are unreliable, requiring target model training for verifiable activation explanations.
• Explainable AI in Production: A Neuro-Symbolic Model for Real-Time Fraud Detection — Integrating explanations directly into model architecture enables real-time, deterministic, and low-latency explainability.
• Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness — Agentic XAI requires explicit verification against model behavior to ensure explanation faithfulness and prevent user misinformation.
• Robust Explanations for User Trust in Enterprise NLP Systems — Decoder LLMs offer significantly more robust explanations than encoders, with stability increasing with model scale, crucial for enterprise trust.
• Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision — Language models can develop faithful self-explanations using fixed, potentially outdated, counterfactual training data.
• Natural Language Autoencoders: Turning Claude’s thoughts into text — Natural Language Autoencoders translate AI model internal activations into human-readable text, revealing hidden thoughts and motivations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.