Probabilistic ML - 17 - Deep Learning

2025-07-02 · Source: Tübingen Machine Learning - YouTube · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Advanced, extended

Summary

The content defines deep learning as a differentiable program and links its training to Bayesian inference, where loss functions are log-likelihoods and regularizers are log-priors, leading to a Maximum A Posteriori (MAP) estimate. It highlights deep learning's advantages, such as O(1) training complexity through mini-batching, array-centric programming (SIMD), and the economic benefit of proprietary models. Conversely, it identifies key disadvantages, including challenges in quantifying uncertainty, enabling continual learning without catastrophic forgetting, and selectively deleting data. The central argument is that any deep neural network can be conceptualized as a Gaussian Process by training the network, linearizing it around the trained weights, applying a Laplace approximation to the loss function for a Gaussian posterior over parameters, and then pushing this measure to function space, yielding a Laplace Tangent Kernel. The LLAX library is introduced as a tool to implement this framework, offering a method to imbue deep networks with probabilistic functionality.

Key takeaway

For AI Scientists and Machine Learning Engineers building foundation models, you should consider integrating Laplace approximations to imbue deep neural networks with crucial probabilistic capabilities. This approach allows for quantifying uncertainty, enabling continual learning without catastrophic forgetting, and even implementing data deletion, addressing critical limitations of current deep learning paradigms. Explore libraries like LLAX to leverage existing trained models and enhance their robustness and interpretability.

Key insights

Deep neural networks can be reinterpreted as Gaussian Processes, enabling probabilistic functionality like uncertainty quantification and continual learning.

Principles

• Deep learning loss functions are log-likelihoods.
• Regularizers serve as log-priors in Bayesian inference.
• Nonlinearity in parameters creates core algorithmic challenges.

Method

The proposed method involves four steps: 1) Train the neural network to find optimal weights. 2) Linearize the network around these trained weights. 3) Apply Laplace approximation to the loss for a Gaussian posterior over parameters. 4) Push this measure to function space to derive a Gaussian Process.

In practice

• Use the LLAX library for Laplace approximation.
• Approximate curvature matrices for scalability.

Topics

• Deep Learning
• Bayesian Inference
• Gaussian Processes
• Laplace Approximation
• Uncertainty Quantification
• Continual Learning

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Probabilistic ML - 18 - Probabilistic Deep Learning — Probabilistic deep learning, even with simple approximations, endows deep neural networks with critical, transferable functionalities beyond point estimates.
• Probabilistic ML - 25 - Revision — Probabilistic inference, though often intractable, provides a foundational framework for machine learning through structured approximations.
• Laplace Approximation for Bayesian Tensor Network Kernel Machines — LA-TNKM provides scalable, principled uncertainty estimates for tensor network kernel machines via Laplace approximation.
• Top 20 Bayesian Regression Interview Questions and Answers (Part 1 of 2) — The Laplace prior promotes sparsity in Bayesian regression, improving interpretability and reducing overfitting when few features are important.
• Bayesian 3D Steerable CNNs: Enabling Equivariance and Uncertainty Quantification Simultaneously — Bayesian Steerable-CNNs unify SE(3)-equivariance with uncertainty quantification via stochastic kernels and variational inference.
• A Bayesian Framework for Built-in Input Dimension Reduction for Gaussian Process Modeling — Integrating Bayesian dimension reduction directly into GP modeling improves predictive accuracy and uncertainty quantification in high-dimensional settings.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tübingen Machine Learning - YouTube.