Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

2026-07-09 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The paper "Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima" extends the theory of gradient descent (GD) dynamics, particularly when large step sizes violate classical analysis conditions in deep neural network training. Published on 2026-07-09, this research generalizes prior work on overparameterized least-squares with a single scalar output near an isolated flat minimum. The authors expand this to include vector-valued outputs, encompassing regression with arbitrarily many observations, and to neighborhoods of a manifold of flat minima, which is crucial for applications like matrix factorization. They generalize the normal form and three convergence theorems, overcoming technical challenges by solving a singular partial differential equation with a novel method. The framework also applies to deep matrix factorization, revealing that flat minima form a fibre bundle over a product of spheres and sharpness is Morse-Bott along this manifold.

Key takeaway

For AI scientists optimizing deep neural networks with large learning rates, this research provides a more robust theoretical framework for understanding gradient descent dynamics. You should consider how the extended analysis for vector-valued outputs and manifolds of flat minima impacts your model design and convergence guarantees, especially in applications like deep matrix factorization. This work suggests new avenues for analyzing and potentially stabilizing training processes where classical step size conditions are violated.

Key insights

The paper extends gradient descent theory for large step sizes to vector outputs and manifolds of flat minima, crucial for deep learning.

Principles

• Gradient descent dynamics can be analyzed with large step sizes.
• Manifolds of flat minima are essential for matrix factorization.
• Sharpness can be Morse-Bott along flat minima manifolds.

Method

The paper describes generalizing a normal form and convergence theorems, solving a singular partial differential equation via a novel method.

In practice

• Apply extended GD theory to deep matrix factorization.
• Analyze vector-valued output models with large step sizes.
• Understand flat minima structure in deep learning.

Topics

• Gradient Descent Dynamics
• Large Step Sizes
• Flat Minima Manifolds
• Deep Matrix Factorization
• Overparameterized Models
• Convergence Theory

Best for: Research Scientist, AI Scientist

Related on AIssential

• Stability of Low-Rank Implicit Regularization in Perturbed Deep Matrix Factorization — Low-rank implicit regularization in deep matrix factorization is robust to bounded additive noise.
• The Day the Machine Learned to Doubt Improvement — Machine learning optimization requires understanding landscape curvature to distinguish true global minima from deceptive local minima or plateaus.
• Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning — Gradient descent in Deep Learning dynamically truncates data clusters, leading to neural collapse and zero loss.
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — A new model tracks average trajectory and oscillation covariance, revealing an effective free energy for Edge of Stability dynamics.
• Riemannian Deep Learning:Modules, Networks, and Geometries — A unified framework for Riemannian deep learning generalizes core components and introduces efficient geometric representations.
• Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks — A variational formulation for shallow neural networks allows global optimization via a linear system, bypassing iterative training.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.