Gradient Boosting
Add weak trees sequentially and watch each stage correct the current ensemble.
Gradient Boosting builds an additive predictor stage by stage. ModelOrigen fits genuine shallow trees to the current error signal and records residual magnitude, weak contributions, boundaries, and held-out metrics.
Step by step
- Start from an initial prediction.
- Measure the current residual/negative-gradient signal.
- Fit a shallow tree to that signal.
- Scale the tree by the learning rate.
- Add it to the ensemble.
- Repeat and inspect how metrics change by stage.
Core formulas
Additive update
Fₘ(x) = Fₘ₋₁(x) + η hₘ(x)Each stage adds a learning-rate-scaled weak learner.
Squared-error residual
rᵢ = yᵢ − Fₘ₋₁(xᵢ)For squared loss, residuals are the correction target.
When to use Gradient Boosting
- A sequence of small corrections can model complex structure.
- Prediction quality matters more than a single-tree explanation.
- You want to study additive boosting behavior.