Physics-Informed Neural Network with Squeeze-Excitation-like Attention

2026-06-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Engineering & Applied Sciences · Depth: Expert, quick

Summary

SEA-PINN is a novel architecture that integrates a Squeeze-Excitation-like attention mechanism into physics-informed neural networks (PINNs), dynamically recalibrating neuron importance across layers. A key feature of SEA-PINN is its highly stable initialization, which resulted in nearly negligible variance and significantly reduced initial loss on 17 out of 20 benchmark problems, establishing a quasi-deterministic starting point for optimization. Without relying on Fourier feature embeddings or periodic activation functions, SEA-PINN achieved competitive accuracy, demonstrating an 83% improvement relative to FNN-PINN on the high-frequency case 7, compared to TSA-PINN's 90%. Furthermore, its integration into TSA-PINN boosted performance by 42.49%. These results position SEA-PINN as a lightweight, plug-in module that enhances nonlinear representation power, promotes robust and efficient convergence, and strengthens the overall reliability of physics-informed learning.

Key takeaway

For Machine Learning Engineers developing physics-informed neural networks, if you are struggling with unstable training or suboptimal accuracy, consider integrating SEA-PINN. Its Squeeze-Excitation-like attention mechanism provides highly stable initialization and boosts performance by enhancing nonlinear representation. You should evaluate SEA-PINN as a lightweight plug-in to achieve more robust and efficient convergence in your PINN applications.

Key insights

SEA-PINN uses Squeeze-Excitation-like attention for dynamic neuron recalibration and stable initialization in physics-informed neural networks.

Principles

• Dynamic neuron importance recalibration enhances PINN performance.
• Stable initialization significantly reduces initial loss and variance.
• Attention mechanisms improve nonlinear representation in PINNs.

Method

Incorporates a Squeeze-Excitation-like attention mechanism into physics-informed neural networks to dynamically recalibrate neuron importance across layers.

In practice

• Integrate SEA-PINN as a lightweight plug-in module.
• Apply SEA-PINN for robust and efficient PINN convergence.
• Utilize SEA-PINN to enhance nonlinear representation power.

Topics

• Physics-Informed Neural Networks
• Squeeze-Excitation Attention
• Neural Network Architectures
• Model Initialization
• Nonlinear Representation

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.
• Statistical Learning Analysis of Physics-Informed Neural Networks — PINNs can be understood as a statistical learning problem minimizing KL divergence with physics as indirect data.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• Attention as Frustrated Synchronization — Frustrated synchronization enables attention to predict context continuation, not just retrieve consensus.
• Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle — This guide demystifies PINN training by exposing the full algebraic cycle, from forward pass to gradient descent.
• Using AI, Mathematicians Find Hidden Glitches in Fluid Equations — Bespoke PINNs can precisely identify previously unobserved unstable fluid singularity candidates across diverse fluid models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.