A Bayesian Framework for Built-in Input Dimension Reduction for Gaussian Process Modeling

2026-07-23 · Source: stat.ML updates on arXiv.org · Field: Science & Research — Mathematics & Computational Sciences, Engineering & Applied Sciences, Research Methodology & Innovation · Depth: Expert, extended

Summary

A new Bayesian framework integrates dimensionality reduction directly into Gaussian Process (GP) modeling, addressing the curse of dimensionality in high-dimensional inputs. This approach, built on a hierarchical Bayesian model, uses priors on the Stiefel manifold to enforce orthonormality on the projection matrix and performs posterior inference via Hamiltonian Monte Carlo with geodesic flow. The framework extends to Deep Gaussian Processes (DGP) with two or three layers, offering enhanced flexibility for complex datasets. Numerical studies, including synthetic benchmarks, a stochastic elliptic PDE, and the ONERA–M6 wing design problem, demonstrate that this method, while computationally intensive, significantly improves predictive performance and uncertainty quantification compared to traditional two-stage or non-BDR methods, especially in small-data, high-dimensional regimes.

Key takeaway

For research scientists developing surrogate models for high-dimensional computer experiments, you should consider this Bayesian framework for built-in dimension reduction. It offers superior predictive accuracy and more reliable uncertainty quantification by jointly learning the low-dimensional input structure and the Gaussian Process surrogate. This approach is particularly beneficial in small-data regimes where propagating projection uncertainty is critical, leading to more robust and interpretable models.

Key insights

Integrating Bayesian dimension reduction directly into GP modeling improves predictive accuracy and uncertainty quantification in high-dimensional settings.

Principles

• Jointly inferring projection and GP parameters propagates uncertainty.
• Matrix Langevin priors enforce orthonormality on projection matrices.
• Deeper GP layers capture nonstationary and nonlinear relationships.

Method

The framework uses a hybrid MCMC scheme combining Metropolis–Hastings, Elliptical Slice Sampling for latent variables, and Hamiltonian Monte Carlo with geodesic flows on the Stiefel manifold for the projection matrix.

In practice

• Apply to high-dimensional computer experiments with limited data.
• Use for stochastic PDE problems with random coefficients.
• Optimize aerodynamic designs like the ONERA–M6 wing.

Topics

• Gaussian Processes
• Bayesian Dimension Reduction
• Deep Gaussian Processes
• Hamiltonian Monte Carlo
• Stiefel Manifold
• Uncertainty Quantification

Code references

• UCStat/Bayesian_BDR_GP

Best for: AI Scientist, Research Scientist

Related on AIssential

• A Bayesian Framework for Built-in Input Dimension Reduction for Gaussian Process Modeling — A Bayesian framework seamlessly integrates input dimension reduction with Gaussian Process modeling, enhancing predictive performance.
• Bayesian Optimization with Gaussian Processes to Accelerate Stationary Point Searches — GPR with active learning and inverse-distance kernels significantly accelerates stationary point searches on potential energy surfaces.
• A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems — A convex framework for neural likelihood approximation improves theoretical foundations and practical efficiency in inverse problems.
• Deep Gaussian Processes on Directed Acyclic Graphs — Deep Gaussian Processes on DAGs model compositional functions, addressing complex inference and uncertainty propagation challenges.
• The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networks — Corrected stochastic MALA (csMALA) improves Gibbs-posterior approximation for scalable uncertainty quantification in neural networks.
• Probabilistic ML - 17 - Deep Learning — Deep neural networks can be reinterpreted as Gaussian Processes, enabling probabilistic functionality like uncertainty quantification and continual learning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.