Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics

2026-07-13 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A precise asymptotic characterization of leave-one-out influences in high-dimensional M-estimation, where the input dimension d grows proportionally to the sample size n (i.e., n ~ d), is presented. For convex M-estimation under Gaussian design, the study reveals that the marginal distribution of test error influences (IF_i) weakly converges to a limiting measure, sharply characterized as the pushforward of a four-dimensional Gaussian distribution through a non-linear map. Similarly, the empirical distribution of DFBETA metrics (DFBETA_i), which quantify changes in model parameters, concentrates to a limiting two-dimensional Gaussian pushforward. Numerical experiments on logistic regression (d=2000, alpha=2, lambda=0.05) and ridge regression (lambda=0.1) validate these theoretical predictions. The findings indicate that influential samples typically reside close to the decision boundary, providing analytical support for common active learning heuristics, a phenomenon also observed qualitatively in real-world datasets like MNIST and chest X-rays.

Key takeaway

For Machine Learning Engineers developing high-dimensional M-estimators or active learning strategies, understanding sample influence is critical. Your data selection policies should prioritize samples near the decision boundary, as these are analytically shown to be most influential in the n ~ d regime. Be aware that the impact of individual samples on model accuracy is maximal at intermediate sample complexities, suggesting a sweet spot for data informativeness.

Key insights

High-dimensional influences in M-estimation converge to predictable distributions, revealing influential samples cluster near decision boundaries.

Principles

• High-dimensional (n~d) influence depends on full dataset.
• Average influence peaks at intermediate sample complexities.
• Label noise flattens influence distributions.

Method

The paper characterizes influence distributions by deriving self-consistent functional equations for complex-valued resolvents, which define summary statistics (Q^(k), V^(k)) that parameterize Gaussian pushforward maps.

In practice

• Prioritize data points near the decision boundary for active learning.
• Consider sample complexity when assessing data informativeness.
• Evaluate DFBETA for empirical distribution of estimator changes.

Topics

• High-dimensional Statistics
• M-estimation
• Influence Diagnostics
• Leave-one-out Cross-validation
• Active Learning
• Gaussian Design

Code references

• HugoCui/HD_influences

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Active Subsampling for Measurement-Constrained M-Estimation of Individualized Thresholds with High-Dimensional Data — Active subsampling significantly accelerates individualized threshold estimation in high-dimensional, label-constrained settings.
• 5 Essential Approaches to Robust Outlier Detection — The choice of outlier detection method depends critically on data distribution and dimensionality.
• Universality of High-Dimensional Logistic Regression and a Novel CGMT under Dependence with Applications to Data Augmentation — The paper extends high-dimensional model risk characterization to dependent data, proving Gaussian universality and introducing a novel CGMT framework.
• Why Machine Learning Finds Order In A Million Dimensions — The manifold hypothesis states real-world data forms low-dimensional structures within high-dimensional spaces.
• Inference with non-differentiable surrogate loss in a general high-dimensional classification framework — A kernel-smoothed decorrelated score enables robust inference for high-dimensional linear decision rules using non-differentiable surrogate loss functions.
• Effective Covariance Dynamics in Solvable High-Dimensional GANs — Structured latent covariance in high-dimensional GANs dictates training dynamics and learnability through an effective second moment, enabling signal-boosting.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.