Double descent for least-squares interpolation on contaminated data: A simulation study

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

A simulation study (April 15, 2026) investigated the "double descent" phenomenon in least-squares interpolation when applied to contaminated linear regression data. Contrary to classical statistical theory, which predicts overfitting with overparameterization, this study found that highly overparameterized least-squares interpolators can exhibit excellent generalization performance. Specifically, for a linear regression setting with n=100 instances and p varying from 10 to 200, and 10% contamination, the least-squares interpolator showed a double descent: test MSE initially increased, peaked at p=n, then decreased significantly for p>n. This performance surpassed several robust alternatives like Huber M-estimator, Tukey biweight M-estimator, sparse LTS, and RRBoost, which either showed less pronounced double descent or none at all. The study suggests that the double descent is a robust property, even with contaminated data, implying potential benefits for using simple least-squares in highly overparameterized scenarios.

Key takeaway

For Data Scientists designing models for potentially contaminated datasets, especially when high overparameterization is feasible, you should reconsider the default assumption that robust estimators are always superior. This research suggests that a simple least-squares interpolator can achieve better generalization performance than robust alternatives in the "double descent" regime (p>n), potentially simplifying model selection and reducing computational overhead. Explore its efficacy before defaulting to more complex robust methods.

Key insights

The least-squares interpolator exhibits double descent and can outperform robust methods on contaminated data with high overparameterization.

Principles

• "Double descent" indicates generalization error decreases after peak complexity.
• Overparameterized models can implicitly denoise contaminated data.
• Robust estimators may underperform simple interpolators in high overparameterization.

Method

The study compared least-squares interpolation with Huber, Tukey, sparse LTS, and RRBoost estimators on contaminated linear regression data, evaluating test MSE, training MSE, and coefficients across varying p.

In practice

• Consider least-squares interpolation for highly overparameterized models.
• Evaluate simple interpolators even with mild data contamination.

Topics

• Double Descent
• Least-Squares Interpolation
• Robust Statistics
• Contaminated Data
• Overparameterization
• Linear Regression

Best for: AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• When “Garbage In, Garbage Out” Gets It Wrong — Wide, messy tabular datasets can produce robust predictions by using redundant signals to recover underlying latent drivers, defying "garbage in, garbage out."
• A Regularization-Sharpness Tradeoff for Linear Interpolators — A regularization-sharpness tradeoff replaces bias-variance in overparameterized linear models, decomposing model selection into alignment and local perturbation effects.
• Adversarial Robustness of NTK Neural Networks — Early stopping is crucial for adversarial robustness in NTK neural networks, as overfitting leads to vulnerability.
• A Theoretical Analysis of Memory and Overfitting Phenomena in Stochastic Interpolation Models — Stochastic interpolation models inherently memorize training data, with generation deviations controlled by discretization and estimation errors.
• Robust Simulation-Based Inference Under Missing Data via Neural Processes — RISE jointly learns data imputation and parameter inference in SBI to robustly handle missing data and prevent biased posteriors.
• Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss — Optimizing for test-time performance, not just validation loss, is crucial for deep learning models with test-time feedback.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.