Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Bayesian Deep Kernel Networks (BDKN) is a novel, efficient Bayesian deep ensemble method for predictive regression, designed to offer enhanced interpretability, competitive predictive performance, and computational efficiency. It integrates the statistical rigor of Bayesian inference with the scalability of deep ensembles, providing calibrated uncertainty estimates. BDKN achieves this through a low-dimensional ensemble representation, where predictions combine a small number of neural predictors, and a closed-form Bayesian aggregation using Bayesian linear regression for interpretable posterior weights. Additionally, independent training of multiple neural networks ensures diverse predictive representations, improving robustness and uncertainty calibration. Empirical evaluations on standard regression benchmarks demonstrate BDKN's competitive predictive performance and reliable uncertainty estimates, often outperforming standard Deep Ensembles (DE) and maintaining stability across varying optimization settings, with a computational cost comparable to DE.

Key takeaway

For Machine Learning Engineers building predictive regression models in safety-critical applications, BDKN offers a robust solution for calibrated uncertainty. You should consider adopting BDKN to achieve competitive predictive performance and interpretable uncertainty estimates without the high computational cost of traditional Bayesian methods. This approach provides stability across diverse optimization settings and allows you to scale ensemble size for improved accuracy and calibration.

Key insights

BDKN combines independent neural networks with exact Bayesian aggregation for efficient, interpretable, and robust uncertainty quantification.

Principles

• Bayesian aggregation improves robustness over uniform averaging.
• Separating representation learning from inference simplifies implementation.
• Finite-rank GP interpretation enables principled Bayesian aggregation.

Method

BDKN trains H independent neural networks to generate feature maps. These maps form a design matrix for a Bayesian linear regression layer, which uses a Jeffreys prior for noise variance and an isotropic Gaussian prior for coefficients, yielding a closed-form Student-t posterior predictive distribution.

In practice

• Use BDKN for safety-critical regression requiring calibrated uncertainty.
• Scale ensemble size to improve accuracy and uncertainty calibration.
• Apply BDKN in systems needing interpretable posterior weights.

Topics

• Bayesian Deep Ensembles
• Uncertainty Quantification
• Predictive Regression
• Gaussian Processes
• Neural Networks
• Machine Learning Efficiency

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Efficient Bayesian Deep Ensembles via Analytic Predictive Inference — Efficient Bayesian deep ensembles combine low-dimensional representation, closed-form aggregation, and independent training for calibrated uncertainty and performance.
• Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks? — Deep ensembles for GNNs suffer "epistemic collapse," limiting their uncertainty quantification benefits due to lack of diversity.
• Variance Computation for Weighted Model Counting with Knowledge Compilation Approach — Computing WMC variance is tractable for structured d-DNNFs but intractable for broader DNNF variants.
• Laplace Approximation for Bayesian Tensor Network Kernel Machines — LA-TNKM provides scalable, principled uncertainty estimates for tensor network kernel machines via Laplace approximation.
• Reliable mechanistic operator recovery with biologically-informed neural networks: principles for architecture and optimisation design — Reliable BINN mechanistic inference requires balancing architectural, optimization, and data informativeness, not maximizing single aspects.
• py/cuTAGI: An Open-Source Library for Tractable Approximate Gaussian Inference in Bayesian Neural Networks — py/cuTAGI enable analytic Bayesian learning in neural networks by treating all quantities as Gaussian random variables, bypassing gradient descent.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.