Random forest

A single decision tree is a high-variance estimator — change a few training rows and the whole tree restructures. Random forest is the elegant fix: train many trees on slightly different versions of the data, average their predictions, and watch the variance collapse. It’s been state-of-the-art on tabular data for two decades, and even today it’s the model most senior practitioners reach for first before bothering with gradient boosting.

Bagging, in 30 seconds

Bagging = Bootstrap AGGregatING. The recipe:

1. Draw B bootstrap samples from the training data — each is the same size as the original, sampled with replacement.
2. Train one decision tree on each bootstrap sample.
3. To predict: classification → majority vote across trees; regression → average of the trees’ predictions.

Each tree sees a slightly different dataset, so each tree makes slightly different mistakes. When you average uncorrelated errors, they cancel (positive and negative deviations wash out); the shared signal does not. Variance plummets, bias stays roughly the same.

Random forest’s extra twist

Plain bagging alone isn’t enough — the trees end up too correlated because the same dominant features get picked at the top of every tree. Random forest adds one more piece of randomness:

At each split, only consider a random subset of features.

That decorrelates the trees. The default subset size is sqrt(n_features) for classification and n_features (all features) for regression in current sklearn. This is the single change that makes random forest beat plain bagged trees.

Fit one on a churn problem

import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
n = 1500
df = pd.DataFrame({
    "tenure_months":        rng.integers(1, 72, n),
    "monthly_spend":        rng.gamma(2, 50, n).round(2),
    "support_tickets":      rng.integers(0, 12, n),
    "logins_per_week":      rng.integers(0, 30, n),
    "days_since_last_login":rng.integers(0, 90, n),
    "plan_tier":            rng.integers(1, 4, n),
    "promo_emails_opened":  rng.integers(0, 20, n),
})
# Synthetic churn rule with noise — multiple interacting features
score = (
    -0.05 * df["tenure_months"]
    + 0.3  * df["support_tickets"]
    + 0.1  * df["days_since_last_login"]
    - 0.1  * df["logins_per_week"]
    + rng.normal(0, 1.5, n)
)
df["churned"] = (score > 1.5).astype(int)

X, y = df.drop(columns=["churned"]), df["churned"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

# A single deep tree vs a forest
tree = DecisionTreeClassifier(random_state=0).fit(X_tr, y_tr)
forest = RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1).fit(X_tr, y_tr)

print(f"single tree  -> train: {tree.score(X_tr, y_tr):.3f}  test: {tree.score(X_te, y_te):.3f}")
print(f"forest (200) -> train: {forest.score(X_tr, y_tr):.3f}  test: {forest.score(X_te, y_te):.3f}")

The single deep tree memorizes training data and tests worse. The forest fits the training data nearly as well but generalizes much better. That gap is the variance the forest just averaged away.

The hyperparameters that matter

The “obvious” instinct is to tune n_estimators first. Don’t. Random forest’s accuracy is fairly flat past 200-300 trees — beyond that you’re burning compute. max_features is the parameter that actually moves the needle.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=10,
    random_state=0,
)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)

print(f"{'max_features':>14}  {'test acc':>9}")
for mf in [1, 2, 5, "sqrt", "log2", 10, 20]:
    rf = RandomForestClassifier(n_estimators=200, max_features=mf,
                                random_state=0, n_jobs=-1).fit(X_tr, y_tr)
    print(f"{str(mf):>14}  {rf.score(X_te, y_te):>9.3f}")

The sweet spot is typically 'sqrt' or a small integer. Set max_features=20 (all of them) and you’ve turned off the decorrelation trick — the forest collapses to plain bagging and accuracy drops.

Out-of-bag (OOB) score — free validation

Here’s a beautiful property of random forest: because each tree only sees ~63% of the training data (the bootstrap sample), the other ~37% is “out-of-bag” for that tree. For every training row, you can average predictions from only the trees that didn’t see it — and that gives you an honest validation estimate without holding any data out.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=2000, n_features=15, n_informative=8, random_state=0)

rf = RandomForestClassifier(
    n_estimators=300, oob_score=True, random_state=0, n_jobs=-1
).fit(X, y)

print(f"OOB score: {rf.oob_score_:.3f}")
print(f"in-sample: {rf.score(X, y):.3f}")

The OOB score is a strong proxy for cross-validated test accuracy and costs you nothing extra to compute. Useful when you’re iterating on hyperparameters fast.

When gradient boosting beats RF

Random forest builds trees independently and averages. Gradient boosting builds trees sequentially, each correcting the residual errors of the previous ensemble. On most tabular benchmarks, well-tuned XGBoost or LightGBM beats random forest by 1-3 percentage points. That’s huge in a Kaggle competition. In a production system where the model is one piece of a larger pipeline, it’s often not worth the tuning effort.

Use random forest when you want a strong, robust baseline with almost no tuning. Use gradient boosting when squeezing the last few points of accuracy actually matters for the business.

In one breath

• A random forest bags decision trees — trains many on bootstrap samples and averages/votes — which collapses the variance of a single high-variance tree.
• Its extra twist: at each split consider only a random subset of features (sqrt(n) for classification), which decorrelates the trees — the change that beats plain bagging.
• The knob that matters is max_features (decorrelation), not n_estimators (accuracy plateaus past ~200–300 trees); leave max_depth=None, set n_jobs=-1.
• The OOB score is free validation: each tree skips ~37% of rows, so you score each row using only the trees that didn’t see it — no held-out set needed.
• It’s the default first model (few knobs, no scaling, robust, free importances + OOB); gradient boosting wins the last 1–3 points when accuracy truly matters.

Quick check