Generalization in Nonlinear Least Squares via Learned Feature Geometry

2026-06-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

The paper "Generalization in Nonlinear Least Squares via Learned Feature Geometry" investigates generalization in ridge-regularized nonlinear least-squares models. It derives error bounds for local minimizers using on-average algorithmic stability, introducing a data-dependent effective dimension. This dimension captures the gradient model's geometry at trained parameters, incorporating the empirical Jacobian Gram matrix and a residual-curvature term. Unlike neural tangent kernel analyses, this effective dimension is evaluated at the trained model. The authors further bound this dimension through gradient feature covering complexity, providing guarantees based on learned geometry rather than parameter count. For manifold-supported data, bounds scale with intrinsic dimension, and for one-hidden-layer ReLU networks, the mechanism involves activation-stable regions. Experiments confirm trained-Jacobian compression and the bounds' agreement with observed generalization gaps. The derivation relies on first principles using the Brascamp--Lieb inequality.

Key takeaway

For AI Scientists evaluating nonlinear models, this research suggests moving beyond simple parameter counts for generalization guarantees. Your analysis should consider data-dependent effective dimensions that reflect learned feature geometry, such as the empirical Jacobian Gram matrix and residual-curvature terms. This approach offers a more nuanced understanding of model performance, especially for complex architectures like ReLU networks, by linking generalization directly to the data's intrinsic dimension and activation stability.

Key insights

New generalization bounds for nonlinear least-squares models depend on learned feature geometry and data-dependent effective dimension, not just parameter count.

Principles

• Generalization bounds can reflect learned geometry.
• Effective dimension can be data-dependent.
• Algorithmic stability yields error bounds.

Method

Error bounds are derived via on-average algorithmic stability, defining an effective dimension from the empirical Jacobian Gram matrix and a residual-curvature term. This dimension is bounded using gradient feature covering complexity.

Topics

• Generalization Theory
• Nonlinear Least Squares
• Algorithmic Stability
• Effective Dimension
• Jacobian Matrix
• ReLU Networks

Best for: Research Scientist, AI Scientist

Related on AIssential

• Separating Geometry from Probability in the Analysis of Generalization — Generalization can be analyzed deterministically via sensitivity of optimization solutions to data perturbations.
• The Impact of Dimensionality on the Stability of Node Embeddings — Node embedding stability varies with dimensionality, but optimal stability does not always align with peak performance.
• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective — Neural network generalization is geometrically characterized by the Representation Gap, scaling with intrinsic data dimension and model equivariance.
• Generalization at the Edge of Stability — Generalization at the "edge of stability" is linked to the "sharpness dimension" of chaotic optimization attractors.
• Quantifying and Optimizing Simplicity via Polynomial Representations — Polynomial representations offer a quantitative simplicity metric and regularizer for deep network generalization.
• The paper that killed deep learning theory — Deep neural networks can memorize random data, challenging traditional generalization theories based on hypothesis class simplicity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.