XGBoost
See gradients, Hessians, regularized split gain, leaf scores, and eta-scaled trees.
XGBoost is a regularized gradient-boosted tree method. ModelOrigen implements a compact educational tree-booster subset in a Web Worker so students can inspect the core equations that distinguish it from generic Gradient Boosting.
Step by step
- Start from a base raw prediction.
- Calculate first-order gradients g and second-order Hessians h.
- Search feature thresholds using regularized split gain.
- Assign each leaf the regularized score −G/(H+λ).
- Scale the new tree by eta.
- Add it to the ensemble and repeat while inspecting objective and held-out metrics.
Core formulas
Regularized objective
Obj = Σ l(yᵢ, ŷᵢ) + Σ Ω(fₖ)Data fit and model complexity are optimized together.
Gradient / Hessian
gᵢ = ∂l/∂ŷᵢ, hᵢ = ∂²l/∂ŷᵢ²Second-order loss information guides tree construction.
Leaf score
w* = −G / (H + λ)Lambda shrinks large leaf corrections.
Split gain
½[GL²/(HL+λ)+GR²/(HR+λ)−G²/(H+λ)]−γGamma requires enough regularized improvement before a split is accepted.
When to use XGBoost
- You want to understand modern regularized tree boosting.
- Complex nonlinear interactions are likely.
- You want explicit control over shrinkage and tree regularization.