Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

The paper "Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations" audits natural-language autoencoders (NLAs) used to explain hidden activations in large language models. It reveals that the standard reconstruction-based faithfulness test is structurally insensitive to false claims, often passing explanations based on "gist" rather than specific facts. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well (r̃=0.84) but only ~2% of their specific claims are reconstruction-dependent. In synthetic tests, co-trained verbalizer/reconstructor pairs consistently develop "co-adapted private codes"—false wording the reconstruction relies on. The authors propose RECAP (Readable Encodings via Co-trained Auxiliary Predictors), a method that trains linear heads alongside the target model to keep designated content decodable. RECAP-trained sandbox models yield truthful verbalizers, and on Pythia-160M, designated content becomes reliably probe-decodable at a minimal cost (+0.001-nat). This approach makes internal content independently checkable, improving lie detection (AUC 0.96 vs. 0.82 control) even against adversaries gaming reconstruction scores.

Key takeaway

For AI safety engineers or ML researchers developing interpretable models, relying solely on reconstruction scores for activation explanation faithfulness is insufficient and prone to gaming. You should implement RECAP by co-training auxiliary predictors with your target model to ensure designated internal content is independently probe-decodable. This enables robust, verifiable monitoring of verbalizer claims (e.g., AUC 0.96 on Pythia-160M), even against adversaries, making your model's explanations trustworthy.

Key insights

Reconstruction-based NLA faithfulness tests are unreliable, requiring target model training for verifiable activation explanations.

Principles

• Reconstruction scores track gist, not specific facts.
• Co-trained verbalizers develop private codes (false wording).
• Probe-based monitoring needs explicit recalibration.

Method

RECAP adds linear auxiliary predictors (heads) to the target model during training, predicting external ground truth from hidden states. This co-training ensures designated content remains decodable by independent probes.

In practice

• Audit NLA explanations using claim-level counterfactual flips.
• Implement RECAP to make internal model states verifiable.
• Use independent probes to monitor verbalizer truthfulness.

Topics

• Activation Explanations
• Model Interpretability
• RECAP Framework
• AI Safety
• Probe-Based Monitoring
• Verifiable AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations — Current activation explanation methods are flawed; RECAP offers a verifiable alternative for internal model content.
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — NLAs translate LLM activations into human-readable text, revealing internal thoughts and hidden motivations for auditing.
• Auditing Construct Overlap in Explainable Machine Learning: Evidence from Burnout-Depression Prediction Across Student Cohorts — XAI pipelines can produce misleadingly stable risk hierarchies due to construct overlap in mental health outcomes.
• Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision — Language models can develop faithful self-explanations using fixed, potentially outdated, counterfactual training data.
• When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles — Fine-tuned Activation Oracles can selectively fail to report internally decodable concepts, raising reliability concerns for learned interpretability.
• Project Ariadne: A Structural Causal Framework for Auditing Faithfulness in LLM Agents — LLM agent reasoning traces often act as "Reasoning Theater," not faithful generative drivers, due to Causal Decoupling.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.