Smoothness-Based Derandomization of PAC-Bayes Bounds

2026-06-18 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

PAC-Bayes derandomization for smooth loss functions is studied to obtain high-probability generalization bounds for deterministic predictors. The research reveals that transitioning from a Gibbs predictor to a deterministic predictor at the posterior mean has a precise cost, defined by the generalization gap of the Jensen gap class. This class is controlled via its Rademacher complexity, leading to bounds for deterministic predictors that involve flatness quantities expressed as parameter Jacobians and Hessians of the score map. The framework applies to both bounded and unbounded smooth loss functions, with specialized results for linear predictors and smooth neural networks. The derived Jacobian and Hessian quantities motivate a practical regularizer, which is computed for BatchNorm networks by folding BatchNorm transformations into adjacent affine weights. Experiments on CIFAR-10 illustrate this regularizer's behavior under varying batch sizes.

Key takeaway

For machine learning engineers developing robust models, understanding the generalization costs of derandomized PAC-Bayes bounds is crucial. You should consider incorporating smoothness-based regularizers, derived from parameter Jacobians and Hessians, to improve deterministic predictor performance. Specifically, if you use BatchNorm networks, explore computing this regularizer by folding BatchNorm transformations into affine weights, as demonstrated on CIFAR-10, to potentially enhance model stability and generalization.

Key insights

Derandomizing PAC-Bayes bounds for deterministic predictors leverages loss and predictor smoothness.

Principles

• The cost of passing from a Gibbs to a deterministic predictor equals the Jensen gap class generalization gap.
• Flatness quantities (Jacobians, Hessians) are key to deterministic predictor bounds.

Method

Control the Jensen gap class via Rademacher complexity to derive bounds involving parameter Jacobians and Hessians, motivating a practical regularizer.

In practice

• Apply the framework to linear predictors and smooth neural networks.
• Compute the regularizer for BatchNorm networks by folding transformations into affine weights.

Topics

• PAC-Bayes
• Derandomization
• Generalization Bounds
• Smoothness
• Neural Networks
• Regularization
• BatchNorm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Smoothness-Based Derandomization of PAC-Bayes Bounds — PAC-Bayes derandomization for smooth loss functions quantifies the cost of deterministic predictors using flatness metrics.
• Separating Geometry from Probability in the Analysis of Generalization — Generalization can be analyzed deterministically via sensitivity of optimization solutions to data perturbations.
• PAC-Bayesian Generalization Guarantees for Fairness on Stochastic and Deterministic Classifiers — PAC-Bayesian bounds can guarantee fairness for both stochastic and deterministic classifiers.
• Upper Bounds on the Generalization Error of Deep Learning Models via Local Robustness and Stability — A new generalization bound improves accuracy by locally scaling robustness based on stable and unstable samples.
• Neural Legendre-Fenchel transform with Hessian Preconditioning — Hessian-based preconditioning improves neural Legendre-Fenchel transform accuracy for ill-conditioned functions.
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning — Robustness techniques converge on estimating and regularizing encoder Jacobians against label-preserving deployment nuisance covariance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.