Online Bernstein-von Mises theorem

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Jeyong Lee, Junhyeok Choi, and Minwoo Chae, in their 2026 paper "Online Bernstein-von Mises theorem," introduce a method for online Bayesian learning that addresses the computational intractability of recursive Bayesian updating. This approach processes data in sequential mini-batches, updating posterior distributions incrementally. The core innovation involves using a variational approximation at each step, leveraging the Bernstein-von Mises theorem to approximate the updated posterior with a normal distribution when the model is regular. The authors demonstrate that, under mild assumptions, the accumulated approximation error becomes negligible once the mini-batch size surpasses a specific threshold determined by the parameter dimension. Consequently, the sequentially updated posterior distribution is asymptotically indistinguishable from the full posterior distribution that would be obtained from processing the entire dataset at once.

Key takeaway

For research scientists developing online learning algorithms, you should consider integrating variational approximations into your Bayesian updating schemes. This approach allows for computationally tractable sequential inference, and by ensuring your mini-batch sizes are adequately large relative to parameter dimensions, you can achieve asymptotic equivalence to full batch processing, improving efficiency without sacrificing accuracy.

Key insights

Sequential variational Bayesian updates can asymptotically match full posterior distributions with sufficient mini-batch sizes.

Principles

• Online learning updates parameters incrementally.
• Variational approximation can enable recursive Bayesian updating.

Method

The method involves sequentially updating posterior distributions using variational approximations, where the updated posterior from one step serves as the prior for the next, ensuring asymptotic equivalence to full posterior.

In practice

• Use variational methods for online Bayesian inference.
• Ensure mini-batch size exceeds parameter dimension threshold.

Topics

• Online Learning
• Bayesian Inference
• Variational Approximation
• Bernstein-von Mises Theorem
• Posterior Distribution

Code references

• JmlrOrg/v27

Best for: Research Scientist, AI Researcher, AI Scientist, AI Student

Related on AIssential

• Probabilistic ML - 23 - Variational Inference — Variational inference approximates intractable posteriors by iteratively maximizing the ELBO within a tractable family of distributions.
• SPDE Methods for Nonparametric Bayesian Posterior Contraction and Laplace Approximation — Extending diffusion-based methods to SPDEs enables robust analysis of nonparametric Bayesian posteriors.
• On the Convergence of Self-Improving Online LLM Alignment — Regularization with reverse KL divergence ensures global convergence for online LLM alignment algorithms.
• Efficient Online Conformal Selection with Limited Feedback — The ACI update rule ensures valid and efficient conformal selection even with limited bandit feedback.
• Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means — Bayes-assisted confidence sequences leverage prior information for efficiency without sacrificing anytime-valid coverage.
• XMSE-Aware Adaptive Empirical Bayes Estimation — An XMSE-aware mixed estimator adaptively combines ML and Empirical Bayes to mitigate kernel misspecification risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.