Train, test, and cross-validation

At 3 a.m., your churn model is scoring 99% accuracy in the dashboard. The team celebrates. Then the first production batch arrives and it scores 78%.

One common explanation is that the dashboard measured training or tuning data. Other possibilities include distribution shift, where production cases differ from development data; label or pipeline mismatches; and changes in data quality. The symptom alone does not identify the cause.

A model is useful when it performs well on unseen data, meaning examples that played no part in fitting the model or choosing its settings. This ability is called generalization. A train/test split and cross-validation estimate it before production gives you an expensive answer.

The overfitting demo

Suppose you have 200 customers, five features, and a binary target: churned or stayed. We will use this small churn dataset throughout the ordinary tabular examples.

Overfitting means fitting the training examples so closely that the model also learns noise and fails to generalize. An unconstrained decision tree illustrates this: it can keep splitting until each training row has its own rule.

import numpy as np

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

# Simulated churn data: 200 customers, 5 features, binary churn label
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 5))
y = (X[:, 0] + 0.3 * rng.normal(size=200) > 0).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0
)

# An unconstrained tree can memorise the training data
model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)

print("train accuracy:", model.score(X_train, y_train))
print("test  accuracy:", model.score(X_test, y_test))

The training score will be 1.0 for this data: the tree can separate the training rows perfectly. The test score will be lower, with the exact value depending on the generated sample and library version. The difference is the generalization gap. Generalization error is the model’s expected error on new examples from the real task.

The test score is useful because those 60 rows were not used to fit the tree. It is still only an estimate: a different random 30% could contain easier or harder customers. With 140 training rows and 60 test rows, a handful of unusual customers can move the score noticeably.

There are three different jobs:

• Training data fits model parameters, such as logistic-regression coefficients.
• Validation data helps choose models or settings, such as tree depth or regularization.
• Test data gives the final estimate after those choices are finished.

With a small dataset, cross-validation is a more efficient validation process. At the end, fit the chosen approach on all development data and evaluate it once on the untouched test set.

One random split

For independent tabular rows, the basic tool is train_test_split:

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,        # 20% goes to test
    random_state=42,      # make this split repeatable
    stratify=y,           # preserve class proportions
)

test_size=0.2 puts roughly 80% of the rows in development and 20% in test. random_state does not make the split better; it makes it repeatable, which is essential when comparing models fairly.

For classification, stratify=y tries to preserve target-class proportions. If 1,000 customers include 50 churners, a 20% test set should contain about 10 churners. Without stratification, one split might contain 6 and another 14, changing accuracy or recall partly because the split changed.

Stratification does not fix class imbalance. A classifier that predicts “no churn” for every customer still gets 95% accuracy in that example and catches zero churners. Choose a metric that represents the cost of mistakes, such as recall, precision, balanced accuracy, or an appropriate area-under-curve measure. See metrics and class imbalance.

There is an important priority rule:

Preserve the deployment structure first. Stratification is useful only when it does not violate time or group boundaries.

If one patient has 20 rows, stratify=y does not stop some rows from landing in training and the rest in test. If rows are future events, stratification does not stop future information from entering training. Those problems require group-aware or time-aware splitting.

Cross-validation: several honest rehearsals

A single split answers, “How did the model do on this particular holdout?” With 500 rows, that answer may be mostly luck.

K-fold cross-validation divides development data into K parts called folds. For each round, one fold is held out for validation and the other K minus 1 folds are used for training. Every row is held out exactly once.

With five folds:

fold 1: [validation | train train train train]
fold 2: [train | validation | train train train]
fold 3: [train train | validation | train train]
fold 4: [train train train | validation | train]
fold 5: [train train train train | validation]

The validation scores are averaged. The average estimates typical performance across held-out portions; the spread shows how sensitive the result is to which rows were held out.

import numpy as np

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 4))
y = (X[:, 0] - X[:, 1] + 0.5 * rng.normal(size=500) > 0).astype(int)

# Each fold has a similar class balance
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(
    LogisticRegression(),
    X, y,
    cv=cv,
    scoring="accuracy",
)

print("per-fold scores:", scores.round(3))
print(f"mean: {scores.mean():.3f}  ±  {scores.std():.3f}")

A result such as 0.80 ± 0.02 is more reassuring than 0.80 ± 0.15. The first model behaved similarly across folds; the second may be excellent for some data slices and poor for others.

The standard deviation is the spread of fold scores, not a confidence interval for production performance. Folds reuse many training rows, so their scores are not independent experiments. Treat mean ± std as a description of evaluation stability.

Five folds are a common compromise. More folds cost more computation; fewer folds reduce that cost but leave each validation estimate based on less data. There is no universally correct K.

For classification, use StratifiedKFold so each fold has approximately the same class proportions:

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

shuffle=True helps when source data is ordered by class or collection batch. It does not make time-series data safe: shuffling a timeline hides the leak.

A class must have enough examples for the requested number of folds. If there are only three positive examples, five stratified folds cannot each contain one. Use fewer folds, collect more data, or rethink the evaluation question.

For ordinary regression, use KFold; there is no general rule that preserves every continuous target distribution.

Leakage inside preprocessing

A split can look correct while evaluation is still contaminated.

Data leakage is information from outside a training fold influencing the model or a preprocessing decision. If you standardize every row before cross-validation, the validation rows help determine the mean and scale used to represent the training data. The leak may be small, but it is the wrong experiment.

Other examples include imputing with full-dataset statistics, selecting features after looking at all target labels, and oversampling before splitting.

Use a Pipeline so each learned preprocessing step is fitted inside each training fold:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)

scores = cross_val_score(
    pipe, X, y, cv=cv, scoring="accuracy"
)

During one fold, StandardScaler learns its mean and scale from that fold’s training rows only. It transforms the validation rows using those values. The validation rows influence the score, but not the fitted transformation.

The same principle applies to the final test set: fit preprocessing on development data, then apply it to test data. Never call fit on the test set.

Cross-validation protects you from one particular split, not from unlimited experimentation. If you try 200 settings and select the best score, some will look good partly by chance. Nested cross-validation uses an inner loop for tuning and an outer loop for evaluation. In a normal product workflow, a separately held-out test set often provides the final check. See model selection and hyperparameter tuning.

The time-series gotcha

Random splitting assumes rows are exchangeable, meaning their order and relationships do not change the experiment’s validity. Daily revenue rows are not exchangeable: trends, promotions, and market conditions connect nearby dates.

If you randomly split 60 days of sales, training may contain days 41 through 60 while test contains day 12. That is not how the model will be used. A prediction on day 61 cannot use information that arrives on day 62.

Use TimeSeriesSplit for an expanding-window backtest:

import numpy as np

from sklearn.model_selection import TimeSeriesSplit

# 60 days of data, one row per day
days = np.arange(60).reshape(-1, 1)

tscv = TimeSeriesSplit(n_splits=5)
for i, (train_idx, test_idx) in enumerate(tscv.split(days), 1):
    print(
        f"fold {i}: train days {train_idx[0]}–{train_idx[-1]}, "
        f"test days {test_idx[0]}–{test_idx[-1]}"
    )

Each test window comes after its training window. That is a backtest: a rehearsal of the information available at each historical prediction date.

The default expanding window is not always the right production simulation. If the real model trains only on the latest 90 days, use max_train_size. If labels arrive two days after an event, leave a gap with gap. If you forecast the next seven days, make test windows and feature construction reflect that horizon.

Time leakage often hides in features. “Customer’s purchases in the next 30 days” is a powerful predictor of churn and an impossible feature at prediction time. A timestamp split cannot rescue a feature whose value would not yet exist.

Related rows and groups

Rows are not independent when several belong to the same entity. A hospital dataset might contain 12 visits per patient. A random split can put 10 visits in training and 2 in test, letting the model learn patient-specific signatures. The score then measures recognition of known patients rather than performance on new ones.

Whether groups must stay disjoint depends on deployment. Keep an entity on one side when generalizing to new patients or customers. If production predicts future rows for entities already known during training, such as future visits from known patients, GroupKFold may answer the wrong question. Use a time-aware split that permits the intended history. The split must reproduce how predictions will actually arrive.

A group is the entity whose rows may need to stay together: patient, household, merchant, device, or customer. Use GroupKFold and pass group labels separately:

from sklearn.model_selection import GroupKFold, cross_val_score

# Synthetic IDs for the current 500-row X and y: five rows per customer
customer_ids = np.arange(len(y)) // 5
assert len(customer_ids) == len(y)  # one group ID is required per row

groups = customer_ids
cv = GroupKFold(n_splits=5)

scores = cross_val_score(
    pipe,
    X, y,
    groups=groups,
    cv=cv,
    scoring="accuracy",
)

No customer appears in both training and validation in a fold. If you also need class balance, use StratifiedGroupKFold, but group separation remains the non-negotiable constraint. A balanced leak is still a leak.

Choosing the split

Failure modes you can spot

High cross-validation, low production: check for duplicate entities, leaked post-outcome features, preprocessing fitted before the split, distribution shift, label or pipeline mismatches, and degraded data quality.

Scores swing between random seeds: the data may be small, rare classes may be too sparse, or rows may contain rare groups. Report the spread and use a split that represents deployment.

A feature records a later event: it is unavailable at prediction time. Remove it or rebuild it using only information available at the cutoff.

In one breath

• Training performance is usually optimistic because the model has already seen those rows.
• A holdout test set estimates unseen-data performance, but must remain untouched while choices are made.
• stratify=y preserves class proportions for ordinary classification; it does not solve imbalance, groups, or time order.
• Cross-validation provides an average and a sense of evaluation noise.
• Put learned preprocessing inside a pipeline.
• Use TimeSeriesSplit for future prediction and group-aware splitting when entities contribute multiple rows.
• The honest split reproduces how a genuinely new prediction will arrive.

Quick check