Combine many learners

Ensembles combine multiple models so the final prediction is not determined by a single learner.

The idea

Instead of trusting one opinion, combine several independent or sequentially improved opinions into a final decision.

Bagging-style methods aggregate many fitted learners; boosting-style methods build learners sequentially to correct remaining error under a defined objective.

How it works

1. Choose a base learner family.
2. Create diversity by resampling/features/randomization or create sequential corrections depending on the ensemble method.
3. Fit multiple learners.
4. Aggregate by vote, average, weighted sum, or additive score depending on the algorithm.

Two common aggregation patterns

Bagging-style prediction averages/votes across members. Boosting builds an additive predictor stage by stage with a learning rate η.

Bagging: ŷ = (1/M)Σₘ fₘ(x) Boosting: F_M(x)=F₀(x)+Σₘ η fₘ(x)

Check your understanding

Which statement is true of boosting?

• Learners are added sequentially to improve remaining error
• It always uses one tree only
• It cannot use weights
• It removes the target

A common misconception

Random forest, AdaBoost, gradient boosting, XGBoost, and Extra Trees are not interchangeable names for the same method.

Open Gradient Boosting visualizer