Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

A new study explains the origin of near-zero Hessian eigenvalues in neural networks, arguing they stem from weakly lifted pseudo-Goldstone modes of continuous symmetries inherent in network parametrization. While deep linear networks exhibit exact symmetries leading to flat directions and exact zero modes, introducing a ReLU nonlinearity weakly breaks these symmetries. The research demonstrates that high-curvature directions are orthogonal to the symmetry subspace, with the bulk of the Hessian spectrum lying almost entirely within it. This mechanism is validated using a two-layer ReLU student-teacher model, a network trained on CIFAR-10, and a convolutional example, extending the diagnostic beyond fully connected layers. These findings clarify the origin of near-zero modes by linking the Hessian bulk to weakly broken symmetries.

Key takeaway

For AI Scientists analyzing neural network optimization landscapes, understanding that near-zero Hessian eigenvalues arise from weakly broken continuous symmetries is crucial. This insight suggests that many flat directions are not random but structured by network parametrization. You should consider how these approximate symmetries influence training dynamics and generalization, potentially guiding the development of new regularization strategies or optimizers that explicitly account for these low-curvature subspaces to improve model stability and performance.

Key insights

Near-zero Hessian eigenvalues in neural networks stem from weakly broken continuous symmetries.

Principles

• Continuous symmetries generate flat directions.
• ReLU nonlinearities weakly break these symmetries.
• High-curvature directions are orthogonal to symmetry.

Method

Analyze Hessian spectrum by resolving eigenvectors, identifying symmetry subspace, and observing orthogonality of high-curvature directions.

In practice

• Diagnose near-zero modes in fully connected layers.
• Extend diagnostic to convolutional networks.

Topics

• Hessian Eigenvalues
• Neural Network Symmetries
• Loss Landscape
• Optimization Theory
• Deep Learning Theory
• ReLU Networks

Best for: Research Scientist, AI Scientist

Related on AIssential

• How the Hessian-Spectrum of Neural Networks Depends on Data — The Hessian spectrum of linear neural networks is derivable and its sharpness relates to data class imbalance.
• Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima — The paper extends gradient descent theory for large step sizes to vector outputs and manifolds of flat minima, crucial for deep learning.
• Curvature-Aligned Probing for Local Loss-Landscape Stabilization — Curvature-aligned probing in top Hessian eigenspaces offers a more efficient and accurate way to assess loss-landscape stabilization.
• Group-Equivariant Poincaré Convolutional Networks — Combining hyperbolic geometry with discrete symmetry groups in neural networks drastically improves optimization and convergence.
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization — LLM training implicitly transfers data symmetries into structured geometric configurations within model parameters.
• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective — Neural network generalization is geometrically characterized by the Representation Gap, scaling with intrinsic data dimension and model equivariance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.