Train, test, and cross-validation

A model that gets 99% accuracy on the data it was trained on tells you almost nothing. Of course it remembers — it just saw those answers. The only number that matters is how well it does on data it has never seen. That sentence is the entire reason train/test splits exist. Generalization is the ability to perform well on data the model has never seen — it’s the only form of accuracy that matters in production.

Misconception to kill early: high training accuracy does not mean a good model. A model that memorizes the training set gets 100% train accuracy and may be useless in production.

The overfitting demo

Overfitting is when a model memorises the training data so thoroughly that it loses the ability to generalise — it’s fitting noise, not signal. Watch it happen:

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

# Simulated churn data: 200 customers, 5 features, binary churn label
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 5))
y = (X[:, 0] + 0.3 * rng.normal(size=200) > 0).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

# An unconstrained tree will memorise the training data
model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)

print("train accuracy:", model.score(X_train, y_train))
print("test  accuracy:", model.score(X_test,  y_test))

You should see exactly 1.00 on train and a good bit lower — somewhere around 0.80–0.90 — on test. The gap is the model’s generalization error (train score minus test score) — the only number anyone outside your team cares about.

One corollary that trips up beginners: never tune your model based on test-set results. The moment you look at the test score and adjust a hyperparameter, the test set becomes training data in disguise — you’ve used its information to make a decision. Cross-validation is the right tool for tuning; the test set is only touched once, at the very end.

train_test_split — the basics

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,        # 20% goes to test
    random_state=42,      # reproducible split
    stratify=y,           # preserve class proportions (classification)
)

stratify=y is the line that catches everyone. Without it, a 5%-churn dataset can end up with a test set that’s 8% churn and a training set that’s 4% — your metrics will jitter for no good reason. Always pass stratify=y for classification.

import numpy as np
from sklearn.model_selection import train_test_split

# 1000 customers, only 5% churn — heavily imbalanced
rng = np.random.default_rng(0)
y = (rng.random(1000) < 0.05).astype(int)
X = rng.normal(size=(1000, 3))

# Without stratify
_, _, _, y_test_a = train_test_split(X, y, test_size=0.2, random_state=1)

# With stratify
_, _, _, y_test_b = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)

print("no stratify — churn rate in test:", y_test_a.mean().round(3))
print("with stratify — churn rate in test:", y_test_b.mean().round(3))
print("true churn rate:", y.mean().round(3))

Run it and the pattern is always the same: the with-stratify test churn rate sits right on the true rate, while the no-stratify one drifts above or below it — the exact digits depend on the seed, but the gap between the two is the whole point.

Cross-validation — beyond a single split

A single 80/20 split is sensitive to which 20% landed in test. With small data, that variance is large enough to mislead you. K-fold cross-validation averages over K different splits.

fold 1: [test | train train train train]
fold 2: [train | test | train train train]
...
fold 5: [train train train train | test]

Each row gets to be in the test set exactly once. You average the K scores. The default k=5 is a good trade-off between bias and runtime — folds large enough to be representative, few enough to fit quickly.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 4))
y = (X[:, 0] - X[:, 1] + 0.5 * rng.normal(size=500) > 0).astype(int)

# StratifiedKFold preserves class balance in each fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(LogisticRegression(), X, y, cv=cv, scoring="accuracy")

print("per-fold scores:", scores.round(3))
print(f"mean: {scores.mean():.3f}  ±  {scores.std():.3f}")

Always report mean ± std from cross-validation, not a single number. A 0.80 ± 0.02 model and a 0.80 ± 0.15 model are very different products.

The time-series gotcha

Random splits assume your rows are exchangeable — that shuffling them doesn’t change anything fundamental. For time series (sales, sensor logs, clicks), that’s wildly false. If you randomly split sales data, future rows leak into your training set and you’ll score wonderfully on the test set and abysmally in production.

Use TimeSeriesSplit instead — train on the past, test on the future:

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

# 60 days of data, one row per day
X = np.arange(60).reshape(-1, 1)

tscv = TimeSeriesSplit(n_splits=5)
for i, (train_idx, test_idx) in enumerate(tscv.split(X), 1):
    print(f"fold {i}: train days {train_idx[0]}–{train_idx[-1]},"
          f" test days {test_idx[0]}–{test_idx[-1]}")

# Then score as usual: cross_val_score(LinearRegression(), X, y, cv=tscv, scoring="r2")
# (R² per fold depends on the data; the split — shown below — is the point.)

fold 1: train days 0–9, test days 10–19
fold 2: train days 0–19, test days 20–29
fold 3: train days 0–29, test days 30–39
fold 4: train days 0–39, test days 40–49
fold 5: train days 0–49, test days 50–59

Each fold’s training window expands; the test window always sits after it. That’s how a real backtest looks.

The decision tree

In one breath

• Training accuracy is a lie — a model that memorizes scores ~100% on train; only performance on unseen data (generalization) counts.
• train_test_split holds out a test set; pass stratify=y for classification to preserve class proportions (essential when imbalanced), and touch the test set only once, at the end — tuning on it leaks.
• A single split has variance, so K-fold cross-validation averages over K splits; report mean ± std, and use StratifiedKFold for classification.
• For time series, random splits leak the future into training — use TimeSeriesSplit (train on past, test on future, expanding window).
• For grouped rows (the same patient across rows), use GroupKFold so a group never spans train and test.

Quick check