Natural gradient descent with momentum

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Anthony Nouy and Agustín Somacal introduce a novel approach called "natural gradient descent with momentum" (NGDM) to enhance the optimization of loss functions for approximating functions on nonlinear manifolds. This method extends the traditional natural gradient descent (NGD), which acts as a preconditioned gradient descent using the Gram matrix of the tangent space to the approximation manifold. While NGD offers locally optimal updates in function space, both NGD and standard gradient descent can become trapped in local minima, especially with nonlinear model classes like neural networks or tensor networks, or when loss functions are ill-conditioned (e.g., KL-divergence, PDE residuals). NGDM integrates classical inertial dynamics, such as Heavy-Ball or Nesterov methods, into the natural gradient framework to improve the learning process and overcome these limitations in nonlinear model optimization.

Key takeaway

For research scientists developing or applying machine learning models on nonlinear manifolds, incorporating natural gradient descent with momentum (NGDM) could significantly improve optimization. If your current natural gradient methods struggle with local minima or ill-conditioned loss functions, consider implementing NGDM to achieve more robust and efficient learning, particularly with neural networks or tensor networks.

Key insights

Natural gradient descent with momentum improves optimization for nonlinear models by incorporating inertial dynamics.

Principles

• NGD uses Gram matrix for locally optimal updates.
• Inertial dynamics can overcome local minima.
• Nonlinear manifolds benefit from momentum-enhanced NGD.

Method

NGDM integrates classical inertial dynamic methods (Heavy-Ball, Nesterov) into the natural gradient descent framework, using the Gram matrix of the tangent space for preconditioning updates in parameter space.

In practice

• Apply NGDM to neural networks.
• Use NGDM for tensor networks.
• Improve density estimation with KL-divergence.

Topics

• Natural Gradient Descent
• Momentum Optimization
• Nonlinear Manifolds
• Neural Networks
• Tensor Networks

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Natural gradient descent with momentum — Natural gradient descent with momentum improves optimization for nonlinear models by overcoming local minima and suboptimal directions.
• Optimizers in Deep Learning: From Gradient Descent to Adam — Optimizers guide neural network training by efficiently updating parameters to minimize loss, evolving from basic GD to adaptive Adam.
• Guaranteed Nonconvex Low-Rank Tensor Estimation via Scaled Gradient Descent — ScaledGD offers provably fast, condition-number-independent convergence for low-rank tensor estimation using t-SVD.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient — Batch size, momentum, and variance reduction critically shape the implicit bias of stochastic steepest descent.
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — A new model tracks average trajectory and oscillation covariance, revealing an effective free energy for Edge of Stability dynamics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.