Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

2026-07-27 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This research analyzes the Bayes-optimal generalization error in extensive-width shallow neural networks using a teacher-student model. Focusing on 2-layer networks where width k and input dimension d are large and proportional, and training data n scales quadratically with d (i.e., n ~ d^2), the study computes the error for any activation function near the interpolation threshold. A discontinuous phase transition is uncovered for binary weights, separating a "universal" phase from a "specialization" phase. In the universal phase, generalization error is independent of weight distribution and decays slowly, with the student learning only non-linear combinations of teacher weights. The specialization phase shows faster error decay due to student alignment with the teacher network, but this highly predictive solution is algorithmically hard to find without informative initialization. The work also extends the GAMP-RIE algorithm to generic activations, matching universal phase performance.

Key takeaway

For AI Scientists and Research Scientists investigating neural network generalization, understanding the n/d^2 sampling rate is crucial. This work reveals that while a "universal" learning phase offers predictable performance, a "specialization" phase promises faster error decay but is computationally challenging to achieve. You should consider the trade-offs between algorithmic tractability (e.g., using GAMP-RIE for universal performance) and the potential for superior, but harder-to-find, specialized solutions, especially when dealing with non-Gaussian weight priors.

Key insights

Extensive-width shallow neural networks exhibit a phase transition between universal and specialization learning, impacting generalization and algorithmic tractability.

Principles

• Generalization error in shallow NNs can be independent of weight distribution in a "universal" phase.
• Exploiting non-Gaussian priors can lead to a "specialization" phase with faster error decay.
• Optimal solutions can be algorithmically hard to reach without informative initialization.

Method

The study employs a statistical mechanics framework, specifically the replica method, with a Gaussian ansatz on replicated post-activations and Hermite polynomial expansion for activation functions.

In practice

• GAMP-RIE can achieve universal phase performance for generic activations.
• Monte Carlo sampling may fail to find optimal solutions in the specialization phase.
• Consider the n/d^2 ratio (sampling rate alpha) to predict learning phase behavior.

Topics

• Neural Network Generalization
• Shallow Neural Networks
• Statistical Mechanics of Learning
• Phase Transitions
• Bayes-Optimal Learning
• GAMP-RIE Algorithm

Code references

• Minh-Toan/extensive-width-NN
• SPOC-group/ExtensiveWidthQuadraticSamples

Best for: AI Scientist, Research Scientist

Related on AIssential

• Statistical mechanics of extensive-width Bayesian neural networks near interpolation — Extensive-width Bayesian neural networks exhibit distinct learning phases, including "specialization" where student weights align with teacher features.
• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime — A universal theoretical framework for neural scaling laws and weight spectra in shallow networks is established via sparse estimation.
• Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks — Flattest interpolators in homogeneous neural networks generalize well for multi-index models despite symmetries.
• Training Neural Networks with Optimal Double-Bayesian Learning — A double-Bayesian framework theoretically derives optimal SGD hyperparameters, showing superior generalization over Adam.
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation — Statistical physics reveals deep learning's feature learning limits and optimal specialization in the interpolation regime.
• Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization — Radial inflation of hidden representations drives memorization-generalization delays in neural networks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.