Probabilistic ML - 16 - Inference in Linear Models

2025-07-02 · Source: Tübingen Machine Learning - YouTube · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Software Development & Engineering · Depth: Advanced, extended

Summary

This lecture explores probabilistic inference in linear models, particularly logistic regression, serving as a conceptual bridge to deep learning. It highlights that the softmax function, often seen as an arbitrary choice in deep learning, is deeply motivated by Bayesian inference. While statistical machine learning focuses on empirical risk minimization, a probabilistic perspective frames learning as prior assumptions conditioned on data, leading to challenges with general likelihoods and non-Gaussian posteriors. The Laplace approximation is introduced to describe these complex posteriors. The discussion then contrasts stochastic gradient descent (SGD) with Newton optimization, demonstrating Newton's quadratic convergence for convex problems like logistic regression, completing in 7 steps versus SGD's 10,000. The Hessian matrix, crucial for Newton's method, also quantifies posterior uncertainty. However, Newton optimization faces scalability issues in deep learning due to massive parameter counts (cubic complexity) and large datasets (linear cost per step), alongside numerical stability concerns. The lecture also touches on classifiers as generative models and MacKay's approximation for predictive probabilities with uncertainty.

Key takeaway

For Machine Learning Engineers working with smaller datasets or convex problems, you should prioritize Newton optimization over stochastic gradient descent. Its quadratic convergence significantly reduces training steps (e.g., 7 vs. 10,000), and the computed Hessian directly provides posterior covariance for quantifying model uncertainty. This allows you to move beyond point estimates, offering richer, more reliable predictions, especially for critical applications like medical diagnostics where "I don't know" is a valuable output.

Key insights

Probabilistic methods offer deeper insights and more efficient optimization for linear models, bridging to deep learning's uncertainty challenges.

Principles

• Softmax is Bayes' theorem for categorical outputs.
• Hessian quantifies posterior precision.
• Convex problems benefit from Newton optimization.

Method

Newton optimization iteratively updates parameters using the gradient and inverse Hessian of the log posterior, converging quadratically. This Hessian also provides posterior covariance for uncertainty quantification.

In practice

• Use Newton for small-scale convex problems.
• Leverage Hessian for posterior uncertainty.
• Consider MacKay's approximation for predictive probabilities.

Topics

• Probabilistic ML
• Logistic Regression
• Bayesian Inference
• Newton Optimization
• Laplace Approximation
• Uncertainty Quantification

Best for: AI Student, AI Scientist, Machine Learning Engineer

Related on AIssential

• Probabilistic ML - 15 - Logistic Regression — Bayes' theorem and exponential families provide a probabilistic foundation for logistic regression and softmax functions.
• A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization — Integrating L-SVRG and stochastic L-BFGS yields a linearly convergent quasi-Newton method for nonsmooth convex optimization.
• Probabilistic ML - 18 - Probabilistic Deep Learning — Probabilistic deep learning, even with simple approximations, endows deep neural networks with critical, transferable functionalities beyond point estimates.
• Bayesian Linear Regression and Maximum Likelihood Estimates — Least squares regression is the maximum likelihood estimate assuming Gaussian noise, enabling Bayesian integration.
• Top 20 Bayesian Regression Interview Questions and Answers (Part 1 of 2) — The Laplace prior promotes sparsity in Bayesian regression, improving interpretability and reducing overfitting when few features are important.
• Bayesian Magnetic Resonance Joint Image Reconstruction and Uncertainty Quantification using Sparsity Prior Models and Markov Chain Monte Carlo Sampling — Bayesian MCMC with sparsity priors enables superior MR image reconstruction and uncertainty quantification, outperforming optimization and deep learning methods.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tübingen Machine Learning - YouTube.