Bagging vs boosting — how do they differ, and when does each help?
Bagging trains many independent models in parallel on bootstrap samples and averages them, which mainly reduces variance; boosting trains models sequentially so each corrects its predecessor's errors, which mainly reduces bias. Use bagging (e.g. random forests) when your base learner is high-variance and overfits; use boosting (e.g. gradient boosting) when you need to squeeze out bias and maximize accuracy, accepting more tuning and overfitting risk.
How to think about it
The crisp answer
Bagging (bootstrap aggregating) trains many models independently and in parallel on random bootstrap samples, then averages or votes — this reduces variance. Boosting trains models sequentially, each focusing on the mistakes of the previous ones — this reduces bias. The split maps cleanly onto the bias-variance decomposition.
Why they differ
Because bagged models are independent and identically distributed, averaging them cuts variance without raising bias — so you pair bagging with strong, low-bias, high-variance learners like fully grown trees (that’s a random forest, which also randomizes features per split). Boosting instead combines weak learners like shallow trees, each correcting residual errors, driving down bias. The scikit-learn ensembles guide frames this exact contrast.
When each helps
- Bagging / random forest: base model overfits, you want a robust, low-tuning model that’s parallelizable and resistant to noise and outliers.
- Boosting / gradient boosting (XGBoost, LightGBM, CatBoost): you want the highest accuracy on structured/tabular data and are willing to tune learning rate, depth, and number of trees. Often state of the art on tabular problems in 2026.
Concrete example
On noisy tabular data, a random forest gives a strong, stable baseline out of the box; a carefully tuned gradient-boosted model usually edges it out on accuracy but is more sensitive to hyperparameters and overfitting.
The common trap
Saying boosting always wins — it’s more prone to overfitting noisy data and outliers (it keeps focusing on hard, possibly mislabeled points) and can’t be parallelized across rounds. Follow-up: “Which reduces bias vs variance?” — bagging reduces variance, boosting reduces bias; stacking blends diverse models via a meta-learner.