Which Regularizer Should You Actually Use? Lessons from 134,400 Simulations

2026-05-02 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

A study by Ahsaas Bajaj and Benjamin S Knight, grounded in eight real-world Instacart production ML models, simulated 134,400 scenarios to compare Ridge, Lasso, ElasticNet, and Post-Lasso OLS regularization for linear models. The research evaluated these methods across three objectives: predictive accuracy (test RMSE), variable selection (F1 score), and coefficient estimation (L2 error). Key findings indicate that for prediction, all three primary methods (Ridge, Lasso, ElasticNet) are nearly interchangeable with median RMSE differences of at most 0.3%, especially with adequate training data (n/p ≥ 78). For variable selection, ElasticNet significantly outperforms Lasso under high multicollinearity (kappa > ~10^4), achieving a 5x recall advantage. For coefficient estimation, ElasticNet is preferred at high multicollinearity, while at low multicollinearity, the choice depends on domain knowledge regarding model sparsity. The study emphasizes that sample size is the most critical factor, outweighing regularizer choice.

Key takeaway

For AI Engineers and Research Scientists selecting a linear model regularizer, prioritize increasing your sample-to-feature ratio (n/p) as it yields greater performance gains than regularizer choice. If n/p ≥ 78, use RidgeCV for its speed and comparable accuracy. When n/p < 78, use ElasticNetCV as a safe default for variable selection and coefficient estimation, particularly with high multicollinearity (kappa > ~10^4), and avoid Post-Lasso OLS entirely.

Key insights

Regularizer choice for linear models depends on optimization objective and data characteristics like sample size and multicollinearity.

Principles

• Sample size (n/p ratio) is the dominant driver of model performance.
• Multicollinearity (kappa) significantly impacts variable selection and coefficient estimation.
• Regularization strength (alpha) can proxy for signal-to-noise ratio (SNR).

Method

The study used 134,400 simulations across 960 configurations of a 7-dimensional parameter space, benchmarking four regularization frameworks against predictive accuracy, variable selection, and coefficient estimation objectives.

In practice

• Compute n/p ratio and condition number (kappa) before model fitting.
• Use RidgeCV for prediction if n/p ≥ 78 due to speed.
• Default to ElasticNetCV for variable selection, especially with high multicollinearity.

Topics

• Ridge, Lasso, ElasticNet
• Linear Models
• Machine Learning Simulation
• Predictive Accuracy
• Variable Selection

Best for: AI Engineer, Research Scientist, Data Scientist, Machine Learning Engineer, AI Scientist

Related on AIssential

• Elastic Net - Explained — Elastic Net combines L1 and L2 regularization for stable feature selection and grouping of correlated variables.
• Vector Norms - Explained — Vector norms offer diverse ways to quantify vector magnitude, each with distinct mathematical and geometric properties.
• A Regularization-Sharpness Tradeoff for Linear Interpolators — A regularization-sharpness tradeoff replaces bias-variance in overparameterized linear models, decomposing model selection into alignment and local perturbation effects.
• Machine Learning | Ridge Regression-L2 — Ridge Regression uses L2 regularization to stabilize coefficients, reduce overfitting, and manage multicollinearity in linear models.
• Theta-regularized Kriging: Modelling and Algorithms — Theta-regularized Kriging improves prediction accuracy and stability by penalizing the Gaussian stochastic process's theta hyperparameter.
• [D] Risk of using XGB models — Weak features in ensemble models can lead to overfitting and reduced robustness, despite aggregation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.