Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new asymptotic theory is proposed for online inference of stochastic gradient descent (SGD) iterates combined with dropout regularization in linear regression models. This theory establishes the geometric-moment contraction (GMC) for constant step-size SGD dropout iterates, demonstrating the existence of a unique stationary distribution for the dropout recursive function. Utilizing the GMC property and a functional dependence measure, the research provides quenched central limit theorems (CLT) for gradient descent iterates with dropout, as well as for Ruppert-Polyak averaged GD (AGD) and averaged SGD (ASGD) iterates. Furthermore, an online estimator for the long-run covariance matrix of ASGD dropout is introduced, enabling efficient recursive inference in terms of computational time and memory. Numerical experiments confirm that for large samples, the proposed confidence intervals for ASGD with dropout achieve the nominal coverage probability. This work was published in 2026, 27(83):1−78.

Key takeaway

For Machine Learning Engineers or Data Scientists implementing online inference with stochastic gradient descent and dropout regularization, this research provides critical theoretical backing and practical tools. You can now confidently apply ASGD with dropout, leveraging the established Central Limit Theorems and the proposed online estimator for the long-run covariance matrix. This enables robust inference with reliable confidence intervals, particularly for large datasets, improving the statistical validity and efficiency of your online learning systems.

Key insights

This paper provides a comprehensive asymptotic theory for SGD with dropout in linear models, enabling robust online inference.

Principles

• Geometric-Moment Contraction (GMC) ensures stationary distribution.
• Functional dependence measure yields Central Limit Theorems.
• Online covariance estimation facilitates efficient recursive inference.

Method

The method involves establishing GMC for SGD dropout, deriving CLTs for GD/AGD/ASGD iterates, and constructing an online estimator for the long-run covariance matrix of ASGD dropout for recursive inference.

In practice

• Use ASGD dropout for robust online inference.
• Apply proposed confidence intervals for large samples.
• Implement online covariance estimator for efficiency.

Topics

• Stochastic Gradient Descent
• Dropout Regularization
• Linear Models
• Asymptotic Theory
• Central Limit Theorem
• Online Inference
• Covariance Estimation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• On the Utility of Equal Batch Sizes for Inference in Stochastic Gradient Descent — Equal batch sizes can consistently estimate SGD asymptotic covariance with memory efficiency and bias correction.
• Understanding Dropout: How Randomly Removing Neurons Helps Neural Networks Generalize Better — Dropout combats neural network overfitting by randomly disabling neurons during training, forcing distributed learning and better generalization.
• Kernel-based guarantees for nonlinear parametric models in Bayesian optimization — A kernel-based framework provides theoretical guarantees for nonlinear parametric models in adaptive optimization.
• Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization — Connecting stochastic resetting to ridge regression offers a renewal perspective on spectral regularization.
• Information-Theoretic Lower Bounds for Bit-Constrained Stochastic Optimization via a Reduction to Compressed Gaussian Mean Estimation — The paper provides information-theoretic lower bounds for bit-constrained stochastic optimization, linking it to compressed Gaussian mean estimation.
• Stochastic Gradient Methods: Bias, Stability and Generalization — A new framework analyzes stability and generalization for biased stochastic gradient methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.