Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new algorithmic framework approximates finite neural networks (NNs) with a mixture of Gaussian processes (GPs), providing provable bounds on the approximation error. This work addresses the limitation that NNs and GPs are equivalent only in the infinite width/depth limit, offering a method for finite NNs with non-i.i.d. parameters. The approach quantifies closeness using the Wasserstein distance and iteratively approximates each NN layer's output distribution as a GP mixture. Crucially, for any NN and $\epsilon >0$, the framework returns a GP mixture that is $\epsilon$-close to the NN at a finite set of input points. The differentiability of the error bound also allows tuning NN parameters to mimic a given GP's functional behavior, useful for prior selection in Bayesian inference. Empirical investigations on regression and classification problems with various NN architectures demonstrate the method's effectiveness.

Key takeaway

For research scientists developing or analyzing neural networks, this framework offers a novel way to formally quantify NN uncertainty and align NN behavior with Gaussian processes. You should explore integrating this method to establish provable error bounds for finite NNs, particularly when rigorous uncertainty quantification or specific prior matching is critical for your models.

Key insights

Finite neural networks can be approximated by Gaussian process mixtures with provable error bounds.

Principles

• NNs and GPs are equivalent only in the infinite limit.
• Wasserstein distance quantifies probabilistic model closeness.

Method

The method iteratively approximates each NN layer's output distribution as a mixture of Gaussian processes, using optimal transport and Wasserstein distance to bound approximation error.

In practice

• Quantify NN uncertainty formally.
• Tune NN parameters for Bayesian prior selection.

Topics

• Finite Neural Networks
• Gaussian Processes
• Mixture Models
• Wasserstein Distance
• Optimal Transport

Best for: Research Scientist, AI Scientist

Related on AIssential

• The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours — NNGP/GPnn offer scalable, principled Gaussian process regression with minimax-optimal rates and hyperparameter robustness.
• Generalization error bounds for two-layer neural networks with Lipschitz loss function — Generalization error bounds for two-layer neural networks can be derived without bounded loss functions using Wasserstein distance and SGM moment bounds.
• Probabilistic Verification of Neural Networks via Efficient Probabilistic Hull Generation — The framework verifies neural networks by computing a guaranteed safe probability range for probabilistic inputs.
• Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks? — Deep ensembles for GNNs suffer "epistemic collapse," limiting their uncertainty quantification benefits due to lack of diversity.
• An Asymptotically Optimal Coordinate Descent Algorithm for Learning Bayesian Networks from Gaussian Models — A new coordinate descent algorithm offers provable optimality guarantees for learning Bayesian networks from Gaussian data.
• Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime — The paper proposes an equivalent Wishart Ansatz and kernel renormalization to predict Bayesian DNN generalization in the proportional limit.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.