Probabilistic ML - 15 - Logistic Regression

2025-07-02 · Source: Tübingen Machine Learning - YouTube · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

This lecture explores the probabilistic foundations of classification problems, particularly logistic regression, connecting common deep learning components to established statistical principles. It demonstrates how the logistic function and softmax naturally arise from Bayes' theorem when dealing with discrete, finitely many hypotheses. The content further motivates linear classifier models by assuming class-conditional distributions are exponential families. The lecture then introduces the Laplace approximation as a method for parameter inference in these models, especially when non-Gaussian likelihoods prevent closed-form solutions, highlighting its use in approximating intractable posterior distributions by finding the mode and measuring curvature. A code demonstration using a toy "two moons" dataset illustrates the training of a Gaussian classifier and the generation of class-predictive distributions.

Key takeaway

For AI Scientists and Machine Learning Engineers developing classification models, understanding the probabilistic origins of logistic and softmax functions, derived from Bayes' theorem, is crucial. This perspective, combined with the Laplace approximation for parameter inference, offers a robust framework for interpreting model outputs and quantifying uncertainty, moving beyond simple point estimates. You can also leverage this to generate class-predictive distributions from your existing classifiers.

Key insights

Bayes' theorem and exponential families provide a probabilistic foundation for logistic regression and softmax functions.

Principles

• Bayes' theorem naturally yields logistic (binary) and softmax (multi-class) functions for classification.
• Linear classifiers can be motivated by assuming class-conditional distributions are exponential families.

Method

The Laplace approximation infers parameters by finding the posterior mode via optimization and then approximating the posterior with a Gaussian using the Hessian (curvature) at that mode.

In practice

• Classifiers can generate class-predictive distributions by reinterpreting Bayes' theorem outputs.
• Laplace approximation offers a tractable, differential approach to approximate posterior distributions.

Topics

• Logistic Regression
• Bayes' Theorem
• Exponential Families
• Laplace Approximation
• Softmax Function
• Probabilistic Deep Learning

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Probabilistic ML - 14 - Exponential Families — Exponential families simplify probabilistic inference by enabling automatic property derivation from their log partition function.
• Probabilistic ML - 16 - Inference in Linear Models — Probabilistic methods offer deeper insights and more efficient optimization for linear models, bridging to deep learning's uncertainty challenges.
• Logistic Regression: The Simplest Explanation — Logistic Regression converts linear model outputs into valid probabilities for binary classification using the Sigmoid function.
• Naïve Bayes: The Algorithm That’s Wrong About Everything But Right About Results — Naïve Bayes effectively classifies by combining prior class probabilities with feature likelihoods, despite a "naïve" independence assumption.
• A Brief History of Softmax: What It Is, Where It Came From, and How It Became Essential — The softmax function's universal mathematical form emerged independently across physics, psychology, and machine learning due to fundamental principles.
• Practical estimation of the optimal classification error with soft labels and calibration — Isotonic calibration enables consistent Bayes error estimation even with corrupted soft labels, provided their order is preserved.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tübingen Machine Learning - YouTube.