Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Neural collapse (NC) is a phenomenon observed during the terminal phase of deep neural network training, where features within the same class converge to their respective sample means, forming a simplex equiangular tight frame. Previous research primarily used the unconstrained feature model (UFM), which inadequately explains how network architecture and dataset properties influence NC. This work by Wanli Hong and Shuyang Ling, published in 2026, investigates NC in shallow ReLU neural networks, specifically two and three-layer architectures. The authors characterize NC's emergence based on network width, depth, data dimension, and training dataset statistical properties. They found that for two-layer ReLU networks, NC's occurrence depends on data dimension, sample size, and signal-to-noise ratio (SNR), rather than network width. For three-layer networks, NC emerges if the first layer is sufficiently wide. Crucially, generalization performance is strongly tied to data SNR; even with NC, poor generalization can result from low SNR. This research significantly advances the theoretical understanding of NC beyond UFM.

Key takeaway

For AI Scientists designing or analyzing shallow ReLU neural networks, understanding neural collapse (NC) requires considering data properties and specific layer widths. Your generalization performance will heavily depend on the data's signal-to-noise ratio (SNR), even if NC occurs. Prioritize data quality and analyze the first layer's width in deeper shallow networks, as these factors are more critical for NC and subsequent generalization than overall network width.

Key insights

Neural collapse in shallow ReLU networks depends on data SNR and specific architectural widths, not always overall width.

Principles

• Neural collapse in two-layer ReLU networks depends on data dimension, sample size, and SNR.
• Sufficient first-layer width enables neural collapse in three-layer networks.
• Generalization performance is strongly correlated with data's signal-to-noise ratio.

Topics

• Neural Collapse
• Shallow Neural Networks
• ReLU Networks
• Signal-to-Noise Ratio
• Generalization Theory
• Network Architecture

Best for: Research Scientist, AI Scientist

Related on AIssential

• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime — A universal theoretical framework for neural scaling laws and weight spectra in shallow networks is established via sparse estimation.
• Deep Neural Regression Collapse — Deep Neural Regression Collapse extends beyond the last layer, revealing structured feature learning in deep networks.
• Generalization in Nonlinear Least Squares via Learned Feature Geometry — New generalization bounds for nonlinear least-squares models depend on learned feature geometry and data-dependent effective dimension, not just parameter count.
• Learning with Shallow Neural Networks on Cluster-Structured Features — Input feature correlations significantly reduce sample complexity for shallow neural networks under specific conditions.
• The Evolution of Language Modeling — Recurrent Neural Networks process variable-length sequential data using feedback loops and shared weights.
• [P] Graph Representation Learning Help — Representation collapse in JEPA-style models often stems from predictor capacity or insufficient objective diversity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.