What is the difference between bagging and boosting, and how do Random Forest and gradient boosting work?

Both are ensemble methods combining weak learners, but they build them differently.

Bagging (Bootstrap Aggregating) trains many models independently and in parallel on bootstrap samples of the data, then averages or votes. Because the models are independent, averaging reduces variance without increasing bias.

Random Forest is bagging with decision trees plus one addition: at each split, only a random subset of features is considered. This decorrelates the trees, which is what makes the averaging effective. It is robust, needs little tuning, handles mixed data types, and rarely overfits badly.

Boosting trains models sequentially, each one focused on the errors of those before it. It reduces bias, building a strong learner from weak ones.

Gradient boosting fits each new tree to the residuals — more precisely, the negative gradient of the loss — of the current ensemble. XGBoost, LightGBM, and CatBoost are optimised implementations, and they remain the strongest general-purpose approach for tabular data.

The trade-offs:

• Random Forest is harder to overfit, trains in parallel, and needs less tuning.
• Gradient boosting usually achieves better accuracy but is sensitive to hyperparameters — particularly learning rate and tree depth — and will overfit if pushed too far. Use early stopping on a validation set.

Note: For most tabular business problems, a well-tuned gradient boosting model beats a neural network. Saying so demonstrates practical rather than academic judgement.