A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

2022-10-28 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, short

Summary

The paper "A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks" by Chen, Vanden-Eijnden, and Bruna, submitted on October 28, 2022 (v1) and last revised July 7, 2026 (v2), extends mean-field theory to analyze three-layer neural networks. Specifically, it focuses on networks where the first-layer weights are randomly sampled and untrained, a departure from prior studies on two-layer networks. The authors rigorously define the limiting model by lifting neuron representation from Euclidean to functional spaces. This approach allows them to establish mean-field training dynamics as a functional gradient flow with a time-varying, positive-definite kernel, proving a linear-rate convergence of its training loss. Furthermore, the work defines novel function spaces containing solutions from mean-field training and derives Rademacher complexity bounds for these spaces. The analysis covers various scaling choices, revealing two distinct mean-field limit regimes that both exhibit feature learning.

Key takeaway

For AI Scientists researching neural network training dynamics, this work provides a foundational theoretical framework. You should consider how functional-space mean-field theory, particularly its application to three-layer networks with untrained first layers, could inform your understanding of convergence and feature learning. This approach offers new tools for analyzing complex network architectures beyond traditional two-layer models, potentially guiding future model design and optimization strategies.

Key insights

Extending mean-field theory to functional spaces enables rigorous analysis of three-layer neural network training dynamics and feature learning.

Principles

• Functional spaces enable rigorous mean-field analysis.
• Feature learning occurs in specific mean-field regimes.

Method

The paper extends mean-field theory by lifting neuron representation to functional spaces, defining training dynamics as a functional gradient flow with a time-varying kernel, and proving linear-rate convergence.

Topics

• Neural Network Theory
• Mean-Field Theory
• Functional Spaces
• Training Dynamics
• Feature Learning
• Three-Layer Neural Networks

Best for: Research Scientist, AI Scientist

Related on AIssential

• Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods — Dynamical Mean Field Theory (DMFT) can exactly solve coupled inference and training dynamics in high-dimensional Neural ODEs.
• How does feature learning reshape the function space? — Feature learning in two-layer neural networks deforms function space via a data-adaptive, target-dependent kernel.
• Neural Field Tokenizations with Hierarchy and Spatial Locality Priors — LH-NeF efficiently learns general-purpose neural field representations by integrating locality and hierarchy priors via a single forward pass.
• Uniform-in-time concentration in two-layer neural networks via transportation inequalities — The study quantifies neural network prediction discrepancies from mean-field limits using transportation inequalities.
• Weighted universal approximation of differentiable maps on infinite-dimensional manifolds — The universal approximation theorem for FNNs is generalized to differentiable maps, including derivative approximation, via a weighted Nachbin theorem.
• Conservation Laws for Modern Neural Architectures — A unified framework characterizes gradient descent conservation laws for modern neural architectures, including attention and MoE models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.