An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

2026-07-02 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Engineering & Applied Sciences · Depth: Expert, quick

Summary

DSGNAR, a Doubly-Sketched Gauss-Newton with Adaptive Ratio, is a new scalable second-order optimization framework designed to improve Physics-Informed Neural Networks (PINNs). PINNs have faced limitations in precision due to ill-conditioned loss landscapes during training. DSGNAR tackles this by integrating a doubly-sketched Gauss-Newton model with a novel strategy that precisely manages both regularization and step length. This framework demonstrates unprecedented accuracy and speed across diverse problems, including nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes equations. It achieves relative ℓ₂ errors as low as 3×10⁻¹⁶ in double precision, improves contemporary results by five orders of magnitude on Burgers' equation, and by eight orders on a high-dimensional Poisson problem. Furthermore, it solves Burgers' equation to ℓ₂ᵗᵉˡ = 4.75 × 10⁻⁷ in under ten seconds using single precision, proving robust across different architectures and hyperparameters.

Key takeaway

For Research Scientists or Machine Learning Engineers developing Physics-Informed Neural Networks, you should consider adopting the DSGNAR optimization framework. This framework directly addresses the critical ill-conditioning problem, enabling significantly higher precision. You can achieve ℓ₂ errors down to 3×10⁻¹⁶ and faster convergence for complex PDE solutions. Integrate DSGNAR to overcome current PINN precision limitations and accelerate your computational physics simulations.

Key insights

PINNs' precision issues are solvable via DSGNAR, a second-order optimization framework achieving unprecedented accuracy and speed by addressing ill-conditioning.

Principles

• Ill-conditioned loss landscapes hinder PINN precision.
• Second-order optimization can resolve PINN training issues.
• Robustness to architecture and hyperparameters is achievable.

Method

DSGNAR couples a doubly-sketched Gauss-Newton model with a novel strategy to carefully control both regularization and step length, confronting ill-conditioning in PINN training.

In practice

• Use DSGNAR for high-precision PINN solutions.
• Apply DSGNAR to nonlinear, chaotic, multi-scale PDEs.
• Explore DSGNAR for faster single-precision PINN results.

Topics

• Physics-Informed Neural Networks
• DSGNAR
• Second-Order Optimization
• Gauss-Newton Method
• Partial Differential Equations
• Numerical Analysis

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• Statistical Learning Analysis of Physics-Informed Neural Networks — PINNs can be understood as a statistical learning problem minimizing KL divergence with physics as indirect data.
• Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle — This guide demystifies PINN training by exposing the full algebraic cycle, from forward pass to gradient descent.
• Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks — Evolutionary two-stage hyperparameter optimization significantly enhances PINN accuracy and robustness under fixed computational budgets.
• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.