Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks

2026-06-19 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A two-stage hyperparameter optimization framework significantly improves the accuracy and robustness of Physics-Informed Neural Networks (PINNs) for solving Partial Differential Equations (PDEs). This method addresses PINNs' sensitivity to hyperparameters and unstable convergence by combining exploration and exploitation phases. The first stage uses evolutionary algorithms like JADE, LSHADE, Grey Wolf, and WOA for low-fidelity training with truncated epochs to rapidly screen candidate configurations. The second stage fully trains only the most promising candidates using standard gradient-based optimizers. Evaluated on Advection, Klein–Gordon, and Helmholtz equations, the approach consistently outperforms standard training, achieving 28% to 77% error reduction and approximately 40% average improvement over baseline error within fixed computational budgets, with JADE showing superior stability. An exploration budget of about 10% of full training epochs proved optimal.

Key takeaway

For Machine Learning Engineers or Research Scientists developing Physics-Informed Neural Networks, you should adopt a two-stage evolutionary hyperparameter optimization strategy. This approach systematically identifies superior configurations, reducing reliance on manual tuning. Specifically, allocate approximately 10% of total training epochs for an exploration phase using algorithms like JADE to rapidly identify promising configurations, then fully train only the best. This can reduce error by 28-77% and significantly improve robustness compared to traditional methods.

Key insights

Evolutionary two-stage hyperparameter optimization significantly enhances PINN accuracy and robustness under fixed computational budgets.

Principles

• PINN performance is highly sensitive to hyperparameters.
• Evolutionary algorithms excel in noisy, non-convex search spaces.
• RMSE correlates with total PINN loss for hyperparameter selection.

Method

A two-stage process: low-fidelity training with truncated epochs for rapid screening (exploration), followed by full training of promising candidates with gradient-based optimizers (exploitation).

In practice

• Use JADE for superior PINN hyperparameter optimization.
• Allocate ~10% of full training epochs for exploration.
• Optimize for total loss when reference solutions are unavailable.

Topics

• Physics-Informed Neural Networks
• Hyperparameter Optimization
• Evolutionary Algorithms
• JADE
• Partial Differential Equations
• Computational Efficiency

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks — A two-stage evolutionary approach effectively optimizes PINN hyperparameters, improving accuracy and robustness under computational constraints.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle — This guide demystifies PINN training by exposing the full algebraic cycle, from forward pass to gradient descent.
• GRAPE: Guided Parameter-Space Evolution for Compact Adversarial Robustness — Guided evolution of a neural network's parameter space during adversarial training can yield more compact and robust models.
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — A new model tracks average trajectory and oscillation covariance, revealing an effective free energy for Edge of Stability dynamics.
• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.