The Representational Limit of Scalar Interactions: An Interventional Decomposition

2026-06-19 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new paper, "The Representational Limit of Scalar Interactions: An Interventional Decomposition," submitted on 17 Jun 2026, addresses the fundamental issue that signed pairwise interaction scores inherently conflate uniqueness (U), redundancy (R), and synergy (S). The authors demonstrate this limitation using a minimal 3-way XOR structural causal model, where faithful indices like Shapley-Taylor yield zero per pair, while projective indices such as Shapley Interaction incorrectly spread third-order effects. To overcome this, the paper introduces Stochastic Hi-Fi, a post-hoc, retraining-free predictability decomposition method. This estimator provides exact interventional semantics, finite-sample Monte Carlo bounds, strict variance reduction via coupled diamond sampling, and uniform finite-vocabulary convergence. Stochastic Hi-Fi successfully recovers structure missed by scalar baselines, showing up to 411x larger interaction-magnitude recovery ratios on tabular SCMs, and effectively separates redundant and synergistic heads within the GPT-2 IOI circuit. It also matches GradCAM on the Pointing Game and significantly improves Deletion AUC on NIH ChestX-ray14.

Key takeaway

For machine learning engineers and AI scientists focused on model interpretability, you should recognize that traditional scalar interaction scores often misrepresent feature contributions by conflating uniqueness, redundancy, and synergy. To gain a more accurate understanding of complex model behaviors, consider implementing advanced decomposition methods like Stochastic Hi-Fi. This approach offers superior insights into feature interactions, crucial for debugging models and ensuring reliability in critical applications.

Key insights

Scalar interaction scores inherently conflate uniqueness, redundancy, and synergy, necessitating advanced decomposition for accurate feature attribution.

Principles

• Scalar interaction scores conflate U/R/S.
• Interventional masked inference estimates U/R/S.
• Coupled diamond sampling reduces variance.

Method

Stochastic Hi-Fi estimates per-feature uniqueness, redundancy, and synergy profiles using interventional masked inference, providing a post-hoc, retraining-free predictability decomposition.

In practice

• Recover missed structure in tabular SCMs.
• Separate GPT-2 IOI circuit heads.
• Improve Deletion AUC on medical imaging.

Topics

• Machine Learning
• Feature Attribution
• Model Interpretability
• Causal Inference
• Stochastic Hi-Fi
• Predictability Decomposition

Best for: Research Scientist, NLP Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Identifying Interactions at Scale for LLMs — SPEX and ProxySPEX efficiently identify influential interactions in ML models by exploiting sparsity and hierarchy.
• Towards Rigorous Explainability by Feature Attribution — Existing SHAP scores for ML explainability are flawed and misleading; rigorous alternatives are necessary.
• Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale — Hierarchical data generation is key to understanding emergent mechanistic phenomena in Transformer language models.
• Interpretable Factor Decomposition for Decision Intelligence in Large-Scale Financial Markets: Evidence from China's A-Share Market — Interpretable ML reveals behavioral factors dominate Chinese A-share predictability, offering auditable financial insights.
• Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs — HETA improves LLM interpretability by combining semantic flow, Hessian-based sensitivity, and KL divergence for causally faithful token attribution.
• Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution — Influcoder enables scalable, cost-effective data attribution for LLMs by distilling decoder influence into an encoder.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.