Gradient boosting and XGBoost

If random forest is the workhorse, gradient boosting is the scalpel. Every public tabular-data competition for the past decade has been won by some variant of boosted trees — XGBoost, LightGBM, or CatBoost. They routinely beat neural nets on structured data, often by a lot, and they remain the default choice for fraud detection, credit scoring, ad ranking, and demand forecasting.

Boosting in 30 seconds

Bagging (random forest) trains trees in parallel, each independent, then averages. Boosting trains trees sequentially, each one targeting the errors left by the ensemble so far.

The simplest version (gradient boosting on squared error):

1. Start with a constant prediction: the mean of y.
2. Compute residuals — the errors left by the current prediction: r = y - current_prediction.
3. Fit a small tree to predict those residuals.
4. Add a fraction (learning_rate) of that tree’s prediction to the ensemble.
5. Repeat for n_estimators rounds.

Each tree corrects the previous ensemble’s mistakes. You end up with hundreds or thousands of tiny trees whose summed predictions are extremely accurate.

Watch the fit improve round by round — each new tree nudges the ensemble closer to the data:

XGBoost, LightGBM, CatBoost — when each shines

These are three flavors of gradient-boosted trees, all roughly equivalent in accuracy but with different strengths.

A senior practitioner usually picks LightGBM for new projects (speed matters), reaches for CatBoost when categoricals dominate, and uses XGBoost when an existing system already depends on it.

Fit a boosted model on tabular features

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 2000
df = pd.DataFrame({
    "age":           rng.integers(18, 75, n),
    "credit_score":  rng.integers(300, 850, n),
    "income":        rng.gamma(2, 30000, n).round(0),
    "debt_ratio":    rng.beta(2, 5, n).round(3),
    "open_accounts": rng.integers(1, 15, n),
    "delinquencies": rng.poisson(0.3, n),
    "loan_amount":   rng.gamma(2, 5000, n).round(0),
})
# Synthetic default rule with strong nonlinearities + noise
risk = (
    -0.005 * df["credit_score"]
    + 1.5  * df["debt_ratio"]
    + 0.4  * df["delinquencies"]
    - 0.00001 * df["income"]
    + rng.normal(0, 0.5, n)
)
df["defaulted"] = (risk > risk.median()).astype(int)

X, y = df.drop(columns=["defaulted"]), df["defaulted"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

model = HistGradientBoostingClassifier(
    max_iter=400,
    learning_rate=0.05,
    max_depth=5,
    min_samples_leaf=20,
    random_state=0,
).fit(X_tr, y_tr)

proba = model.predict_proba(X_te)[:, 1]
print(f"test accuracy: {model.score(X_te, y_te):.3f}")
print(f"test AUC     : {roc_auc_score(y_te, proba):.3f}")

The API mirrors XGBoost and LightGBM almost line-for-line. The same hyperparameters carry over.

The four hyperparameters that actually matter

Boosting libraries expose dozens of knobs. In practice you tune four:

The classic rule of thumb: lower learning_rate, more n_estimators, use early stopping. Tiny step size + many corrective trees + automatic stopping is the recipe.

Early stopping — the lever you really want

Don’t pre-pick n_estimators — let validation loss tell you when to stop. You set a large n_estimators (say 2000) and tell the trainer to halt when the validation score hasn’t improved for early_stopping_rounds trees in a row.

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=4000, n_features=20, n_informative=12, random_state=0,
)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

model = HistGradientBoostingClassifier(
    max_iter=2000,
    learning_rate=0.05,
    early_stopping=True,
    validation_fraction=0.15,
    n_iter_no_change=20,
    random_state=0,
).fit(X_tr, y_tr)

print(f"stopped at iteration: {model.n_iter_}")
print(f"test accuracy       : {model.score(X_te, y_te):.3f}")

The model picks its own size. You’ll see it stop hundreds of iterations before the max — because the validation curve flattened out and adding more trees would overfit.

Why boosted trees dominate tabular data

A few reasons all stack:

• Nonlinear interactions for free. Trees naturally model if income > X and debt_ratio < Y then ... without manual feature engineering.
• Mixed feature types. Numeric, ordinal, low-cardinality categoricals all work without scaling.
• Robust to feature scale and outliers. Splits are threshold-based.
• Gradient information. Each tree is fit to the gradient of the loss, meaning the ensemble takes targeted corrective steps — far more efficient than independent random trees.
• Strong regularization knobs. learning_rate, min_child_weight, lambda (L2), gamma (split penalty) all let you tune capacity precisely.

The result: on most structured datasets, a well-tuned XGBoost beats a random forest by 1-3 points and beats a tabular neural net (TabNet, FT-Transformer) by similar margins, while training 10x faster.

A note on Kaggle-style tuning

In competitions, people tune dozens of hyperparameters with Optuna or hyperopt over thousands of trials. In production, don’t. Tune the four parameters above with CV, accept the result, and move on. The 0.001 AUC gain from heroic tuning is rarely worth the maintenance burden and the overfitting risk.

In one breath

• Boosting trains trees sequentially, each fitting the residuals of the ensemble so far and adding a learning_rate fraction — hundreds of tiny corrective trees that sum to an extremely accurate model.
• XGBoost / LightGBM / CatBoost are three flavors: XGBoost (classic, safe), LightGBM (fast, large data), CatBoost (native high-cardinality categoricals). Modern tree libs handle raw categoricals — stop one-hot-encoding for trees.
• Tune four knobs: n_estimators (with early stopping), learning_rate (small = better generalization, more trees), max_depth (shallow, 3–8), min_child_weight/min_samples_leaf.
• Early stopping auto-sizes the ensemble: set a big n_estimators and halt when validation hasn’t improved for N rounds.
• Boosted trees dominate tabular data (free nonlinear interactions, mixed types, scale/outlier-robust, gradient-guided), beating RF by 1–3 points and tabular NNs while training fast. Senior workflow: RF baseline first, then boosting; skip Kaggle-style mega-tuning in production.

Quick check