Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks

2026-06-03 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The paper "Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks" addresses the challenged heuristic that "flat interpolators generalize well." Previous work by Dinh et al. (2017) showed network symmetries could alter flatness without changing loss, making the heuristic seem vacuous. This research establishes a direct link between flatness and generalization for learning unknown multi-index models using 2-layer non-convex homogeneous neural networks, specifically focusing on "flattest" interpolators. It demonstrates that a class of non-generalizing interpolators exists whose flatness cannot be minimized by symmetries. Furthermore, for data generated by a sum of single-index models with low approximation error and label noise, any flattest interpolator achieves small population loss, consistently generalizing. This connection applies across various activations and realistic data distributions.

Key takeaway

For AI scientists evaluating neural network generalization, this research clarifies that "flattest" interpolators in 2-layer homogeneous networks reliably generalize for multi-index models under specific data conditions. You should consider the "flattest" interpolator concept when analyzing model performance, especially when dealing with low noise and approximation errors. This refines the understanding of flatness as a generalization predictor.

Key insights

Flattest interpolators in homogeneous neural networks generalize well for multi-index models despite symmetries.

Principles

• Flattest interpolators generalize reliably.
• Symmetries do not always minimize flatness.
• Low noise and error enable generalization.

Topics

• Flatness
• Generalization
• Multi-Index Models
• Homogeneous Neural Networks
• Interpolators
• Gradient Methods

Best for: Research Scientist, AI Scientist

Related on AIssential

• A Sharper Picture of Generalization in Transformers — Sparse Fourier spectra and low-sharpness constructions improve Transformer generalization on boolean domains.
• The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity — The γ-graph dimension characterizes learnability, and a median-based aggregation of three interpolating hypotheses is optimal.
• Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective — Generative Cross-Entropy improves DNN accuracy and calibration by regularizing class-level confidence from a generative perspective.
• A Regularization-Sharpness Tradeoff for Linear Interpolators — A regularization-sharpness tradeoff replaces bias-variance in overparameterized linear models, decomposing model selection into alignment and local perturbation effects.
• Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin — Gradient flow dynamics of homogeneous neural networks beyond the origin preserve sparsity and characterize saddle points.
• A Function-Centric Perspective on Flat and Sharp Minima — Sharpness in neural network minima is function-dependent, not solely indicative of poor generalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.