Canonical Regularisation of Wide Feature-Learning Neural Networks

2026-05-19 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This research introduces a novel framework for understanding and regularizing wide neural networks in the feature-learning regime, a critical but less-studied area compared to kernel-regime networks. The authors demonstrate that traditional regularization methods like standard and anchored ridge regularization, which are effective in the kernel regime, bias gradient flow in feature-learning networks due to the curved geometry of their parameter space. This bias, which persists even with vanishing regularization, can degrade the inductive bias of pretrained networks. To address this, the paper axiomatizes a "canonical regularizer" as a regime-agnostic function-space energy, leading to the derivation of "geodesic ridge" for feature-learning networks and identifying the corresponding function-space prior as a "Riemannian Gibbs Process." For practical application, the authors propose "arc ridge," a scalable, minimax-robust surrogate to geodesic ridge, which is equivalent to early stopping under exact gradient flow. Empirical results on image processing (UTKFace with ResNet18) and NLP transfer-learning (Yelp Review with DistilBERT) problems validate the theory, showing that arc ridge preserves pretrained priors better than traditional methods at high regularization strengths.

Key takeaway

For research scientists developing or fine-tuning wide neural networks in feature-learning regimes, you should re-evaluate your regularization strategies. Traditional anchored ridge or weight decay can pathologically degrade pretrained representations by biasing gradient flow along output fibers. Instead, consider implementing "arc ridge" regularization, which is a scalable, minimax-robust surrogate for the theoretically sound "geodesic ridge." This approach respects the intrinsic geometry of feature learning, preserves inductive biases, and offers a principled alternative to early stopping, improving generalization, especially in transfer learning scenarios.

Key insights

Traditional ridge regularization biases feature-learning networks due to curved parameter-space geometry, necessitating a geodesic-based approach.

Principles

• Network implicit prior is governed by gradient flow trajectory geometry.
• Anchored ridge fails in feature-learning due to manifold curvature.
• Geodesic ridge generalizes canonical regularization to curved geometries.

Method

The proposed method axiomatizes a canonical function-space energy, lifts it to parameter space, and derives "geodesic ridge" and its scalable surrogate "arc ridge" by studying Riemannian geometry.

In practice

• Use arc ridge for unbiased regularization in feature-learning networks.
• Arc ridge is computationally efficient, requiring only path length tracking.
• Consider arc ridge as a principled alternative to early stopping.

Topics

• Canonical Regularisation
• Feature-Learning Networks
• Geodesic Ridge
• Riemannian Gibbs Process
• Arc Ridge

Code references

• GWhittle110/canonical-regularization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Geodesics with Unified Tangent-constrained Priors and Curvature Regularization — Integrating tangent-constrained priors with curvature penalization in an orientation-lifted space improves geodesic image segmentation robustness and shape fidelity.
• Elastic Net - Explained — Elastic Net combines L1 and L2 regularization for stable feature selection and grouping of correlated variables.
• Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization — Connecting stochastic resetting to ridge regression offers a renewal perspective on spectral regularization.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• Regularisation in Neural Networks — Regularization techniques are crucial for neural networks to prevent overfitting and improve generalization on unseen data.
• A Function-Centric Perspective on Flat and Sharp Minima — Sharpness in neural network minima is function-dependent, not solely indicative of poor generalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.