A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Zhengdao Chen, Eric Vanden-Eijnden, and Joan Bruna's 2026 paper introduces a functional-space mean-field theory for partially-trained three-layer neural networks. This work extends prior mean-field studies on two-layer networks by lifting neuron representations from Euclidean to functional spaces to define the infinite-width limit. The authors establish mean-field training dynamics as a functional gradient flow with a time-varying, positive-definite kernel, proving linear-rate convergence of its training loss. They also define novel function spaces containing the solutions and derive Rademacher complexity bounds for these spaces. Their analysis covers various scaling choices, revealing two distinct mean-field limit regimes that both demonstrate feature learning during training.

Key takeaway

For AI scientists and theoretical ML researchers exploring advanced neural network architectures, this work provides a critical theoretical foundation. You should consider how the functional-space mean-field theory extends understanding of training dynamics and generalization for deeper networks, particularly those with partially untrained layers. This framework offers new avenues for analyzing feature learning and convergence beyond traditional two-layer models.

Key insights

Functional-space mean-field theory extends neural network analysis to three-layer models, proving convergence and feature learning.

Principles

• Infinite-width limits apply to three-layer NNs.
• Functional spaces enable rigorous MF theory.
• Feature learning occurs with untrained first layers.

Method

The method extends mean-field theory by lifting neuron representations to functional spaces, defining training dynamics as a functional gradient flow with a time-varying kernel.

Topics

• Neural Networks
• Mean-Field Theory
• Functional Spaces
• Training Dynamics
• Rademacher Complexity
• Feature Learning

Best for: Research Scientist, AI Scientist

Related on AIssential

• Neural Field Tokenizations with Hierarchy and Spatial Locality Priors — LH-NeF efficiently learns general-purpose neural field representations by integrating locality and hierarchy priors via a single forward pass.
• Universality of Shallow and Deep Neural Networks on Non-Euclidean Spaces — TFNNs extend universal approximation to non-Euclidean spaces, even with width constraints, by leveraging topological properties.
• Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods — Dynamical Mean Field Theory (DMFT) can exactly solve coupled inference and training dynamics in high-dimensional Neural ODEs.
• Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics — Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.
• Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks — A new framework unifies representation costs and native function spaces for parametric data-fitting methods, revealing deep neural network inductive biases.
• Uniform-in-time concentration in two-layer neural networks via transportation inequalities — The study quantifies neural network prediction discrepancies from mean-field limits using transportation inequalities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.