Extra #11 - Advanced Bagging: seven experiments

2026-05-20 · Source: Machine Learning Pills · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

Extra #11 - Advanced Bagging: seven experiments explores the complexities and limitations of Bootstrap Aggregating (Bagging) beyond its basic "Bootstrap → Train → Average" principle. The article investigates the variance floor, demonstrating that the ensemble's variance is bounded below by ρσ², where ρ is the correlation between individual model predictions and σ² is their variance. This is illustrated through a simulation using artificial predictions with chosen correlation values (e.g., ρ = 0.0, 0.1, 0.3, 0.6, 0.9) and varying numbers of averaged predictions (B = 1, 10, 100, 200). It also touches upon cardinality bias, the superior performance of Random Forests over plain Bagging, the trustworthiness of Out-of-Bag (OOB) error, and challenges in naive from-scratch implementations.

Key takeaway

For Machine Learning Engineers optimizing ensemble models, understanding the variance floor is crucial. You should analyze the correlation (ρ) between your base models, as this directly limits the achievable variance reduction, even with many estimators. If your models exhibit high correlation, consider techniques like feature subsampling, as seen in Random Forests, to decorrelate them and improve ensemble performance beyond plain Bagging's limits.

Key insights

Bagging's variance reduction is limited by model correlation, setting a floor for ensemble performance.

Principles

• Ensemble variance is bounded by ρσ².
• Higher model correlation limits Bagging gains.
• Random Forests reduce correlation better.

Method

Simulate ensemble variance by mixing shared and individual noise components in predictions, then calculate empirical and theoretical variance for varying B and ρ.

In practice

• Evaluate ensemble variance with varying B and ρ.
• Consider Random Forests for lower correlation.
• Use OOB error for validation if trustworthy.

Topics

• Bagging
• Ensemble Learning
• Variance Reduction
• Random Forests
• Model Correlation
• Out-of-Bag Error

Best for: Machine Learning Engineer, AI Scientist, Data Scientist

Related on AIssential

• Issue #132 - Bagging: Why Training on Broken Copies of Your Data Works — Bagging reduces model variance by averaging predictions from models trained on bootstrapped data, offering free out-of-bag validation.
• Bagging vs Boosting: Variance vs Bias — Bagging and Boosting are opposite ensemble methods that combine weak learners into strong models, with Bagging reducing variance and Boosting reducing bias.
• Random Forests as Statistical Procedures: Design, Variance, and Dependence — Random forests are finite-sample statistical designs with irreducible predictive variability due to inherent design-induced dependence.
• Simplex-Constrained Sparse Bagging: Transitioning from Uniform Priors to Sparse Posteriors in Ensemble Learning — SCSB compresses and calibrates bagging ensembles by optimizing sparse voting weights, improving efficiency and confidence.
• Bagging vs Boosting: The Complete Beginner’s Guide to Ensemble Learning in Machine Learning — Ensemble Learning combines multiple models to improve prediction, with Bagging and Boosting as core techniques.
• Starting Off on the Wrong Foot: Pitfalls in Data Preparation — Statistically informed data preparation significantly improves model robustness and efficiency for imbalanced, real-world datasets.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning Pills.