Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new analysis challenges the common modeling of Stochastic Gradient Descent (SGD) as a Langevin process, which assumes minibatch noise acts as Brownian motion. This work, published on 2026-05-21, proposes an alternative framework: SGD as deterministic dynamics within a fluctuating loss landscape caused by minibatch sampling. Starting from the discrete update, the authors derive a master equation and a discrete Fokker--Planck equation, which deviates from the standard Langevin form at order eta^2. Analyzing SGD near critical points, the framework reveals distinct behaviors along the mean Hessian's eigenbasis. Notably, nearly-flat directions exhibit variance growth over time, indicating effective diffusion along valleys with a learning rate-proportional coefficient. Empirical evidence supports these predictions on neural network models in computer vision and natural language processing, showing clear separation between confined and diffusive modes.

Key takeaway

For machine learning engineers optimizing neural network training, this research suggests re-evaluating traditional SGD models. You should consider the proposed discrete Fokker--Planck framework, which more accurately captures SGD's behavior, especially the effective diffusion along nearly-flat loss valleys. This understanding can inform hyperparameter tuning and architectural choices, potentially leading to more stable and efficient optimization strategies.

Key insights

SGD's dynamics are better modeled as deterministic in a fluctuating loss landscape, not Brownian motion.

Principles

• Nearly-flat directions do not admit a stationary distribution.
• Variance grows over time, proportional to the learning rate.

Method

Derive a master equation for parameter distribution and a discrete Fokker--Planck equation directly from discrete SGD updates.

In practice

• Analyze SGD dynamics near critical points of the loss.
• Observe confined and diffusive modes in neural networks.

Topics

• Stochastic Gradient Descent
• Langevin Dynamics
• Fokker-Planck Equation
• Loss Landscape
• Neural Network Optimization
• Computer Vision
• Natural Language Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Why Gradient Descent Feels Like a Particle Rolling Down a Hill — Gradient descent is fundamentally a physical process of energy minimization in high-dimensional parameter space.
• Neural Galerkin Normalizing Flows for Bayesian Inference of Diffusions with Inaccessible Boundaries — Neural Galerkin Normalizing Flows efficiently learn diffusion transition densities for Bayesian inference, bypassing complex analytical challenges.
• Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation — SGLRW improves SG-MCMC robustness by localizing stochastic noise to off-diagonal covariance elements via lattice-based updates.
• On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials — Nonlinear dynamics significantly impact GD/SGD stability, often contradicting linear approximations, especially with high-order derivatives and single batch instability.
• More Than Image Generators: A Science of Problem-Solving using Probability | Diffusion Models — Diffusion models combine deep learning with Langevin sampling to generate images by iteratively following noisy gradients of an image's probability distribution.
• Training Neural Networks with Optimal Double-Bayesian Learning — A double-Bayesian framework theoretically derives optimal SGD hyperparameters, showing superior generalization over Adam.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.