Fisher-Geometric Diffusion in Stochastic Gradient Descent: Optimal Rates, Oracle Complexity, and Information-Theoretic Limits

2026-03-04 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper introduces a Fisher-geometric theory for Stochastic Gradient Descent (SGD), asserting that mini-batch noise is an intrinsic, loss-induced matrix rather than an exogenous scalar. Under exchangeable sampling, the mini-batch gradient covariance is primarily determined by the projected covariance of per-sample gradients, equating to projected Fisher information for well-specified likelihood losses and the projected Godambe matrix for general M-estimation losses. This identification leads to a diffusion approximation with Fisher/Godambe-structured volatility, where the effective temperature is defined as $\tau=\eta/b$. The theory yields an Ornstein-Uhlenbeck linearization whose stationary covariance is given in closed form by a Fisher-Lyapunov equation. The authors prove matching minimax upper and lower bounds of order $\Theta(1/N)$ for Fisher/Godambe risk under a total oracle budget $N$, and derive oracle-complexity guarantees for $\varepsilon$-stationarity in the Fisher dual norm that depend on an intrinsic effective dimension and a Fisher/Godambe condition number, rather than ambient dimension or Euclidean conditioning. Experiments validate the Lyapunov predictions and demonstrate that scalar temperature matching fails to reproduce directional noise structure.

Key takeaway

For AI Researchers and Research Scientists optimizing models with SGD, understanding that mini-batch noise has an intrinsic Fisher/Godambe geometry is crucial. This means your batch size decisions directly control the "temperature" of a diffusion process, shaping the anisotropic noise and influencing convergence in statistically meaningful directions. You should consider optimizing for Fisher/Godambe risk, which scales with intrinsic effective dimension and condition number, rather than relying solely on Euclidean metrics or scalar variance assumptions, especially in operations research settings where sampling effort is a key design variable.

Key insights

Mini-batch noise in SGD possesses an intrinsic, loss-induced matrix geometry, not scalar variance.

Principles

• Noise covariance is determined by sampling mechanism and loss function.
• Fisher/Godambe metric is the natural measure for SGD convergence.
• Batch size controls diffusion "temperature" but not noise shape.

Method

The paper identifies mini-batch noise covariance as $G^{\star}(\theta)/b$, leading to a diffusion approximation and an Ornstein-Uhlenbeck linearization whose stationary covariance solves a Fisher-Lyapunov equation.

In practice

• Evaluate variance reduction by its effect on Fisher-metric risk.
• Monitor effective dimension $d_{\operatorname{eff}}$ for tighter bounds.
• Regulate batch size based on local curvature for adaptive control.

Topics

• Stochastic Gradient Descent
• Fisher Information
• Godambe Information
• Diffusion Approximation
• Oracle Complexity

Best for: AI Researcher, AI Scientist, Research Scientist

Related on AIssential

• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics — SGD's dynamics are better modeled as deterministic in a fluctuating loss landscape, not Brownian motion.
• Fisher Width: A Geometric Measure of Complexity on Statistical Manifolds — Fisher width extends Gaussian width to statistical manifolds by incorporating the local Fisher information metric.
• Convergence Rates for Non-Log-Concave Sampling and Log-Partition Estimation — Optimal rates for non-log-concave sampling and log-partition estimation can sometimes surpass optimization rates.
• stochastic gradient descent #maths #statistics #datascience #machinelearning #dataanlysis — Faster iteration with noisy gradients often outperforms slow, precise convergence in machine learning.
• Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser — VMPO minimizes log importance weight variance for diffusion alignment, unifying existing methods and suggesting new designs.
• On the Utility of Equal Batch Sizes for Inference in Stochastic Gradient Descent — Equal batch sizes can consistently estimate SGD asymptotic covariance with memory efficiency and bias correction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.