Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates

2026-07-08 · Source: Machine Learning · Field: Science & Research — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Engineering & Applied Sciences · Depth: Expert, quick

Summary

A new optimization method, Riemann-normal-coordinate Levenberg-Marquardt (RNC-LM), is proposed to enhance nonlinear least-squares optimization. This method addresses a limitation of the standard Levenberg-Marquardt (LM) approach, which applies tangent-space steps as straight updates in parameter coordinates, leading to inconsistencies due to parameter-effects curvature. RNC-LM extends geodesic acceleration by reformulating the geodesic equation, enabling arbitrary-order corrections and constructing finite-step updates with higher reparameterization consistency. It employs a line search along the RNC curve to control distance while maintaining cost close to standard LM. The technique eliminates the tangential component of residual acceleration order by order, making objective reduction more consistent with LM's linear model. RNC-LM demonstrates improved convergence and robustness on classical benchmarks, reduces relative L2 error to 1e-3 on a reaction-diffusion PINN failure-mode benchmark, and achieves a 34-fold speedup on a large-scale machine-learning potential-energy-surface fitting task.

Key takeaway

For research scientists optimizing nonlinear least-squares problems, particularly in physics-informed neural networks or large-scale machine learning, you should consider RNC-LM. This method offers significantly improved convergence and robustness over standard Levenberg-Marquardt, reducing L2 error to 1e-3 and achieving a 34-fold speedup in specific tasks. Evaluate RNC-LM to overcome parameter-effects curvature limitations and achieve more physically meaningful solutions.

Key insights

RNC-LM improves nonlinear least-squares optimization by using higher-order geometric updates for enhanced reparameterization consistency and performance.

Principles

• Parameter-effects curvature limits standard LM.
• Higher-order geometric updates improve optimization consistency.
• Riemann-normal-coordinates extend geodesic acceleration.

Method

RNC-LM reformulates the geodesic equation to extend geodesic acceleration, constructing finite-step updates with higher reparameterization consistency, and uses a line search along the RNC curve.

In practice

• Apply RNC-LM to regression problems.
• Use RNC-LM for physics-informed neural networks.
• Improve large-scale potential-energy-surface fitting.

Topics

• Nonlinear Least-Squares
• Levenberg-Marquardt Method
• Riemann Normal Coordinates
• Physics-Informed Neural Networks
• Optimization Algorithms
• Geodesic Acceleration

Best for: AI Scientist, Research Scientist

Related on AIssential

• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — A curvature-aware optimization framework improves PINN training by adaptively correcting first-order optimizers using secant information.
• Laplace Approximation for Bayesian Tensor Network Kernel Machines — LA-TNKM provides scalable, principled uncertainty estimates for tensor network kernel machines via Laplace approximation.
• Neural Legendre-Fenchel transform with Hessian Preconditioning — Hessian-based preconditioning improves neural Legendre-Fenchel transform accuracy for ill-conditioned functions.
• On the Convergence of Self-Improving Online LLM Alignment — Regularization with reverse KL divergence ensures global convergence for online LLM alignment algorithms.
• Generalization in Nonlinear Least Squares via Learned Feature Geometry — New generalization bounds for nonlinear least-squares models depend on learned feature geometry and data-dependent effective dimension, not just parameter count.
• Fast Computation of Superquantile-Constrained Optimization Through Implicit Scenario Reduction — Superquantile-constrained optimization can be significantly accelerated using a specialized second-order computational framework.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.