Building Fast, Evaluating Slow: Pipeline Choices Dominate Autointerpretability Score Variance

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

This study reveals that autointerpretability scores, widely used for cross-paper comparison of sparse autoencoder (SAE) features, are significantly influenced by evaluation pipeline choices rather than stable feature properties. Through systematic experiments across four metrics (simulation, detection, fuzzing, purity) and two models (Pythia-160M, Apertus-8B), researchers found that methodological variance collectively exceeds architectural variance (R1). Each metric exhibits a distinct instability profile, with detection being the most stable and fuzzing unreliable (R2). Furthermore, top-$k$ feature rankings are inconsistent across conditions, masking per-feature instability despite stable mean scores (R3). These findings suggest that current comparisons may reflect pipeline differences, hindering progress in interpretability research. The authors contribute a variance decomposition approach, a Stability Check, and a Minimum Reporting Checklist to improve evaluation reliability.

Key takeaway

For Research Scientists evaluating Sparse Autoencoders (SAEs), you must critically assess the stability of your autointerpretability evaluation pipeline. Your cross-architecture comparisons may reflect methodological choices rather than true architectural differences. Implement the proposed Stability Check and Minimum Reporting Checklist to diagnose reliability before interpreting score variations. This ensures your findings contribute meaningfully to the SAE utility debate and advance interpretability research.

Key insights

Autointerpretability scores for SAEs are unstable, primarily reflecting evaluation pipeline choices, not inherent feature properties.

Principles

• Methodological variance exceeds architectural variance.
• Each interpretability metric has a distinct instability profile.
• Mean scores can mask per-feature instability.

Method

The study quantifies instability using Intraclass Correlation Coefficient (ICC), Top-$k$ Jaccard, and a random effects model for variance decomposition, proposing a Stability Check and Minimum Reporting Checklist.

In practice

• Use multi-metric evaluation for SAE interpretability.
• Implement a Stability Check for evaluation pipelines.
• Adhere to a Minimum Reporting Checklist for transparency.

Topics

• Sparse Autoencoders
• Autointerpretability
• Evaluation Reliability
• Language Models
• Metric Instability
• Variance Decomposition

Best for: AI Scientist, Research Scientist

Related on AIssential

• Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders — Reward model preference instability stems from brittle features, detectable and correctable via Sparse Autoencoders without retraining.
• Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations — A new framework evaluates Sparse Autoencoder interpretability by aligning latents with human concepts using synthetic data and perturbation scores.
• From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness — High correlational recovery in Sparse Autoencoders often masks causally inert features, necessitating deeper validation.
• Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE) — Aligned training improves Sparse Autoencoder feature quality and stability without adding parameters.
• From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability — A framework certifies SAE-based LM interpretability by bounding risk using proxy fidelity, reconstruction, mismatch, and complexity.
• Statistically Grounded Sparse-Feature Interventions for Activation-Space Control in Large Language Models — Statistically-grounded, optimization-free SAE-feature steering provides interpretable LLM control, but usable shifts are highly localized and quality-dependent.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.