Bound to Disagree: Generalization Bounds via Certifiable Surrogates

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Data Science & Analytics · Depth: Expert, extended

Summary

This paper introduces novel disagreement-based generalization bounds for deep learning models, addressing common issues like vacuousness, computability, and restrictions to specific model classes. The framework certifies the true risk of any predictor by leveraging a simpler surrogate model that possesses tight generalization guarantees. This is achieved by evaluating a disagreement bound on an unlabeled dataset, crucially without modifying the target model or its training procedure. Empirical demonstrations showcase the tightness of these certificates and the approach's versatility, applying it with sample compression, model compression, and PAC-Bayes theory. Experiments cover CNNs on MNIST, ResNet18 on CIFAR10, and DistilBERT and GPT2 on Amazon polarity, providing non-vacuous bounds for complex models.

Key takeaway

For AI Scientists and Machine Learning Engineers concerned with model reliability, these disagreement-based generalization bounds offer a practical solution. You can now obtain non-vacuous, computable risk certificates for complex deep neural networks without modifying their architecture or retraining. Consider integrating this framework to validate model performance, especially for quantized versions or those undergoing distillation, leveraging readily available unlabeled data.

Key insights

New disagreement-based bounds certify deep learning models without modification, using simpler surrogates and unlabeled data.

Principles

• Generalization bounds can apply to any ML predictor.
• Surrogate models enable tight generalization guarantees.
• Unlabeled data suffices for disagreement evaluation.

Method

Bound a target model's true risk by linking it to a simpler, certifiable surrogate model, measuring their prediction disagreement on an unlabeled dataset. This avoids altering the target model or its training.

In practice

• Certify complex models like DistilBERT and GPT2.
• Minimize disagreement via model distillation.
• Bound true risk gap for quantized models.

Topics

• Generalization Bounds
• Surrogate Models
• Model Compression
• PAC-Bayes Theory
• Sample Compression
• Model Distillation

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Upper Bounds on the Generalization Error of Deep Learning Models via Local Robustness and Stability — A new generalization bound improves accuracy by locally scaling robustness based on stable and unstable samples.
• Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards — First non-vacuous generalization bounds for billion-parameter RLVR LLMs improve understanding and enable more compressible models.
• PAC-Bayesian Generalization Guarantees for Fairness on Stochastic and Deterministic Classifiers — PAC-Bayesian bounds can guarantee fairness for both stochastic and deterministic classifiers.
• Model Agreement via Anchoring — Anchoring to model averages can effectively reduce disagreement between independently trained machine learning models.
• Separating Geometry from Probability in the Analysis of Generalization — Generalization can be analyzed deterministically via sensitivity of optimization solutions to data perturbations.
• Sound Probabilistic Safety Bounds for Large Language Models — A new framework computes rigorous, sound lower bounds for LLM harmfulness by actively guiding generation towards unsafe outputs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.