Probabilistic ML - 18 - Probabilistic Deep Learning

2025-07-02 · Source: Tübingen Machine Learning - YouTube · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This lecture explores Probabilistic Deep Learning, focusing on Laplace approximations to treat deep neural networks as Gaussian Processes. It details a four-step method to construct a posterior, which serves as a second-order approximation to a trained neural network's maximum a posteriori estimate. The discussion addresses the computational cost and scalability of these methods, comparing them to alternatives like model ensembles. Key functionalities enabled by probabilistic deep learning include fixing asymptotic overconfidence in ReLU networks, facilitating hyperparameter optimization via the evidence framework, allowing the imposition of meaningful priors and mechanistic knowledge, and enabling continuous learning to combat catastrophic forgetting. The speaker argues that incorporating uncertainty provides essential capabilities for modern deep learning, particularly for foundation models, by offering a sensitivity map of the model's knowledge.

Key takeaway

For Machine Learning Engineers developing foundation models, you should integrate probabilistic methods like Laplace approximations. This approach provides essential uncertainty quantification, enabling robust out-of-distribution detection and mitigating issues like catastrophic forgetting in continuous learning. Consider advocating for industry standards that include approximate kernel data structures alongside model weights to enable these critical functionalities for downstream fine-tuning and analysis.

Key insights

Probabilistic deep learning, even with simple approximations, endows deep neural networks with critical, transferable functionalities beyond point estimates.

Principles

• Deep neural networks are implicitly Gaussian Processes.
• Point estimates inherently lead to overconfidence.
• Posterior covariance acts as a model sensitivity map.

Method

Train a deep neural network, then compute the loss function's curvature (Generalized Gauss-Newton) to derive an approximate Gaussian Process posterior with a Laplace tangent kernel.

In practice

• Mitigate ReLU network overconfidence with minimal cost.
• Optimize hyperparameters using marginal likelihood evidence.
• Incorporate mechanistic priors for scientific applications.

Topics

• Probabilistic Deep Learning
• Laplace Approximation
• Gaussian Processes
• Uncertainty Quantification
• Catastrophic Forgetting
• Foundation Models

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Probabilistic ML - 17 - Deep Learning — Deep neural networks can be reinterpreted as Gaussian Processes, enabling probabilistic functionality like uncertainty quantification and continual learning.
• Probabilistic ML - 25 - Revision — Probabilistic inference, though often intractable, provides a foundational framework for machine learning through structured approximations.
• SPDE Methods for Nonparametric Bayesian Posterior Contraction and Laplace Approximation — Extending diffusion-based methods to SPDEs enables robust analysis of nonparametric Bayesian posteriors.
• Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection — Finite neural networks can be approximated by Gaussian process mixtures with provable error bounds.
• Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning — Bayesian deep learning uncertainty estimation can be made efficient and accurate without expensive sampling.
• Laplace Approximation for Bayesian Tensor Network Kernel Machines — LA-TNKM provides scalable, principled uncertainty estimates for tensor network kernel machines via Laplace approximation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tübingen Machine Learning - YouTube.