Issue #132 - Bagging: Why Training on Broken Copies of Your Data Works

2026-05-17 · Source: Machine Learning Pills · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

Bootstrap Aggregating, or Bagging, is a machine learning ensemble technique introduced by Leo Breiman in 1996 that trains multiple copies of the same model on slightly different versions of the training data and then averages their predictions. This method directly addresses high model variance, a common issue with models like fully-grown decision trees that tend to memorize training data and are highly sensitive to small changes in the dataset. Bagging utilizes Efron's 1979 bootstrap resampling technique, where each model is trained on a sample of n rows drawn with replacement from the original n-row dataset, resulting in approximately 63% unique rows and 37% "out-of-bag" (OOB) rows. Averaging these models reduces prediction variance, though correlation between individual bagged models sets a floor on this reduction. The OOB samples provide a free, reliable validation score, approximating test error without needing a separate validation set. While effective for variance reduction, Bagging cannot fix systematic model bias or issues arising from highly correlated base models.

Key takeaway

For Machine Learning Engineers building robust predictive models, understanding Bagging's mechanics is crucial. You should implement Bagging to significantly reduce model variance, especially with high-variance base learners like decision trees, and leverage the out-of-bag score for efficient, built-in validation. Be aware that increasing the number of trees beyond 100-200 typically yields diminishing returns and that Bagging will not correct inherent model bias or issues from highly correlated base models.

Key insights

Bagging reduces model variance by averaging predictions from models trained on bootstrapped data, offering free out-of-bag validation.

Principles

• High model variance is reduced by averaging independent predictions.
• Bagging only fixes variance, not systematic model bias.
• Correlation between base models limits variance reduction.

Method

Train B models on B bootstrap samples (n rows with replacement). Average their predictions for regression or use majority vote for classification. Utilize out-of-bag samples for validation.

In practice

• Use OOB score for free validation without a separate test set.
• Target 100-200 trees for optimal performance in most tabular datasets.
• Apply Bagging to high-variance base models like decision trees or k-NN.

Topics

• Bagging
• Ensemble Learning
• Model Variance
• Out-of-Bag Error
• Decision Trees

Best for: AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Extra #11 - Advanced Bagging: seven experiments — Bagging's variance reduction is limited by model correlation, setting a floor for ensemble performance.
• Bagging vs Boosting: Variance vs Bias — Bagging and Boosting are opposite ensemble methods that combine weak learners into strong models, with Bagging reducing variance and Boosting reducing bias.
• Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well — Cross-validation significantly reduces benchmarking variance, improving reliability of machine learning performance evaluation through virtual data augmentation.
• Simplex-Constrained Sparse Bagging: Transitioning from Uniform Priors to Sparse Posteriors in Ensemble Learning — SCSB compresses and calibrates bagging ensembles by optimizing sparse voting weights, improving efficiency and confidence.
• Bagging vs Boosting: The Complete Beginner’s Guide to Ensemble Learning in Machine Learning — Ensemble Learning combines multiple models to improve prediction, with Bagging and Boosting as core techniques.
• An Information-Theoretic Criterion for Efficient Data Synthesis — Synthetic data is effective only when external signals keep the training loop "information-open."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning Pills.