“Features” aren’t always the true computational primitives of a model, but that might be fine anyways

2026-02-02 · Source: AI Alignment Forum · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

The concept of "feature" in mechanistic interpretability is often debated, with three primary definitions emerging: computational primitives, salient properties of the input, and features assumed to exist for theoretical discussions. Computational primitives are challenging to reverse-engineer, as illustrated by the modular addition network example where the MLP's function was progressively understood from sin/cos factors to a trigonometric integral. Salient properties are practical, referring to model-represented input characteristics that can be manipulated to alter behavior, as seen in SAEs. The article proposes that "features" exist on a spectrum from pure memorization to case analysis/equivalence partitioning, culminating in true computational primitives. This spectrum is demonstrated using a toy vision-language model classifying "bleggs vs rubes," showing how a model could represent two properties (redness, cubeness), four object types, or even memorize individual data points, each being a valid "feature" interpretation depending on the context.

Key takeaway

For research scientists grappling with feature definitions in mechanistic interpretability, recognize that "features" are not monolithic but exist on a spectrum from memorized data points to abstract computational primitives. This understanding should guide your interpretability efforts, allowing you to select the most appropriate feature interpretation for your specific analytical goal, whether it's debugging, formal verification, or understanding model generalization. Avoid rigid definitions and instead focus on what computations neural networks implement and how components contribute to a compact description of that computation.

Key insights

Features in mechanistic interpretability exist on a spectrum from memorization to computational primitives.

Principles

• No single "true" feature definition exists.
• Features are useful if they reduce loss.
• Interpretability benefits from case analysis.

In practice

• Consider feature spectrum when interpreting models.
• Use case analysis for debugging model behavior.
• Recognize memorized facts as valid features.

Topics

• Mechanistic Interpretability
• Neural Network Features
• Sparse Autoencoders
• Feature Spectrum
• Model Behavior Analysis

Best for: Research Scientist, AI Researcher, AI Scientist, AI Student

Related on AIssential

• Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation — Mechanism-aware evaluation using symbolic rules and interpretability better distinguishes true generalization from pattern exploitation.
• The most important features in generalized additive models might be groups of features — Analyzing feature groups, not just individual features, offers critical insights in interpretable machine learning.
• ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations — ConceptTracer interactively analyzes neural network representations using information-theoretic measures for concept saliency and selectivity.
• Alike Parts: A Feature-Informed Approach to Local and Global Prototype Explanations — Integrating feature importance into prototype selection enhances both local explanations and global prototype diversity without compromising model fidelity.
• From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts? — Correlational disentanglement metrics for interpretability methods do not guarantee independent concept manipulability during steering.
• Interview with Deepika Vemuri: interpretability and concept-based learning — Concept-based learning enhances AI interpretability by aligning model representations with human-understandable concepts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Alignment Forum.