Quantitative Gaussian-Process limits of Tensor Programs

2026-07-08 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper presents a quantitative convergence theory for the infinite-width Gaussian-process limit of random neural networks, utilizing the tensor program framework. The main result establishes explicit finite-width error bounds, of order inverse square-root of the widths (1/√n), in Wasserstein distance between finite-network executions and their Gaussian-process limits. This architecture-agnostic framework encompasses feed-forward models and weight-sharing schemes relevant for recurrent and transformer-type architectures. The work also introduces "Netsor^K", an extension supporting scalar kernel variables and scalar-parametric nonlinearities, crucial for attention layers. Numerical experiments validate these quantitative wide-limit convergences across shallow and deep MLPs, time-unrolled RNNs, and Residual Networks, demonstrating output-law convergence visible beyond sampling noise.

Key takeaway

For AI Scientists designing or analyzing large-scale neural networks, this work provides critical quantitative bounds on the approximation quality of finite-width models to their infinite-width Gaussian Process limits. You should consider the established 1/√width error rates when evaluating the trade-off between computational efficiency and theoretical guarantees. This framework is particularly relevant for architectures employing weight sharing, such as RNNs, or attention mechanisms, enabling more precise predictions of model behavior at various scales.

Key insights

Quantitative bounds (1/√width) for neural network Gaussian Process limits are established via tensor programs.

Principles

• Finite-width NNs converge to Gaussian Processes with 1/√width error bounds.
• Tensor programs unify quantitative analysis across diverse NN architectures.
• Weight-sharing in RNNs necessitates conditional Gaussian structure analysis.

Method

A line-by-line inductive coupling method, combined with a quantitative kernel LLN, establishes convergence by tracking conditional Gaussian laws.

In practice

• "Netsor^K" extends analysis to attention layers and normalization.
• Framework applies to MLPs, RNNs, and Residual Networks.
• Sliced Wasserstein-1 distance quantifies empirical output-law convergence.

Topics

• Gaussian Processes
• Tensor Programs
• Neural Network Theory
• Wasserstein Distance
• Recurrent Neural Networks
• Attention Mechanisms

Best for: Research Scientist, AI Scientist

Related on AIssential

• Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection — Finite neural networks can be approximated by Gaussian process mixtures with provable error bounds.
• ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit — ResNet training dynamics converge to a quantifiable large-scale limit across depth, width, and embedding dimension.
• The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours — NNGP/GPnn offer scalable, principled Gaussian process regression with minimax-optimal rates and hyperparameter robustness.
• Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime — The paper proposes an equivalent Wishart Ansatz and kernel renormalization to predict Bayesian DNN generalization in the proportional limit.
• Upper Bounds on the Generalization Error of Deep Learning Models via Local Robustness and Stability — A new generalization bound improves accuracy by locally scaling robustness based on stable and unstable samples.
• Uniform-in-time concentration in two-layer neural networks via transportation inequalities — The study quantifies neural network prediction discrepancies from mean-field limits using transportation inequalities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.