From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

2026-03-20 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

A paper by Karun Adusumilli, Maximilian Kasy, and Ashia Wilson, submitted on arXiv as 2603.20388v2 on July 10, 2026, derives the asymptotic risk function for regularized empirical risk minimization (ERM) estimators. These estimators are specifically tuned using n-fold cross-validation (CV). The research reveals that the out-of-sample prediction loss of these CV-tuned estimators converges in distribution to the squared-error loss of shrinkage estimators within the normal means model, which are tuned by Stein's unbiased risk estimate (SURE). This finding offers a more granular understanding of predictive performance, detailing how risk varies with the true parameter, unlike broader uniform bounds on worst-case regret. Key intermediate results include demonstrating that n-fold CV converges uniformly to SURE, and that SURE's global minimum is well-separated, ensuring the convergence of the tuning parameter chosen by CV to that chosen by SURE.

Key takeaway

For research scientists developing or evaluating regularized empirical risk minimization (ERM) models, this work provides a more precise framework for understanding predictive performance. You can now analyze the asymptotic risk of n-fold cross-validation-tuned estimators by mapping them to SURE-tuned shrinkage estimators. This allows for a fine-grained quantification of how risk changes with true parameters, moving beyond worst-case regret bounds and potentially guiding more robust model selection.

Key insights

Cross-validation-tuned regularized estimators exhibit asymptotic risk equivalent to SURE-tuned shrinkage estimators in the normal means model.

Principles

• N-fold cross-validation converges uniformly to Stein's unbiased risk estimate (SURE).
• SURE's global minimum is well-separated, ensuring tuning parameter convergence.

Method

The paper derives the asymptotic risk function for regularized ERM estimators tuned by n-fold CV, showing its convergence to the squared-error loss of SURE-tuned shrinkage estimators.

Topics

• Regularized Estimators
• Cross-Validation
• Stein's Unbiased Risk Estimate
• Asymptotic Risk
• Empirical Risk Minimization
• Shrinkage Estimators

Best for: AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction — The framework decomposes pre-hoc fine-tuning prediction risk into intrinsic and reducible components, revealing fundamental uncertainty limits.
• XMSE-Aware Adaptive Empirical Bayes Estimation — An XMSE-aware mixed estimator adaptively blends ML and EB shrinkage, using second-order risk to mitigate kernel misspecification.
• Double descent for least-squares interpolation on contaminated data: A simulation study — The least-squares interpolator exhibits double descent and can outperform robust methods on contaminated data with high overparameterization.
• Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure — No universal noisy-label learning method exists for frozen Vision Foundation Models; selection must be regime-aware.
• Stein’s method, learning and inference -or- how to really monitor convergence and thin chains — Stein's method, leveraging scores and operators, offers a robust approach for convergence monitoring and advanced probabilistic inference.
• K-Fold Cross-Validation: Every Point Tested Once — K-Fold Cross-Validation provides a robust model performance estimate by ensuring every data point is tested once across multiple splits.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.