Data leakage — the silent model killer

You train a churn model. Cross-validation AUC: 0.97. You ship it. In production, AUC drops to 0.62 and the business stops trusting your team.

Welcome to data leakage — the bug where information about the held-out set or the future sneaks into training. Because the model has already “seen” what it’s being evaluated on, validation scores are artificially inflated: the model looks brilliant on paper and useless in production. It’s the most common silent failure in ML, it bites senior practitioners regularly, and the cure is the Pipeline pattern.

This lesson is about the three forms it takes and how to lock them out.

The same illegitimate shortcut wears three disguises — and each is closed by one line of discipline.

Form 1: Target leakage

A feature in your training data is derived from the label — sometimes indirectly, sometimes obviously. The model learns a near-perfect rule involving that feature, which doesn’t exist at prediction time.

A classic example: predicting churn with last_billing_status. If last_billing_status='cancelled' only ever appears after the customer has churned, your model “learns” that this feature predicts churn perfectly — but at the moment you’d actually need the prediction, the billing status hasn’t been set yet.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 1500
df = pd.DataFrame({
    "tenure_months":   rng.integers(1, 72, n),
    "monthly_spend":   rng.gamma(2, 50, n).round(2),
    "support_tickets": rng.integers(0, 12, n),
    "logins_per_week": rng.integers(0, 30, n),
})
# True churn signal
score = -0.05 * df["tenure_months"] + 0.3 * df["support_tickets"] + rng.normal(0, 1, n)
df["churned"] = (score > 0.5).astype(int)

# The LEAKY feature — set only AFTER churn happened, but accidentally available at training time
df["last_billing_status"] = np.where(df["churned"] == 1, "cancelled", "active")

# Train WITH the leaky feature
df_enc = df.copy()
df_enc["last_billing_status"] = (df_enc["last_billing_status"] == "cancelled").astype(int)

X_leaky = df_enc.drop(columns=["churned"])
y = df_enc["churned"]
X_tr, X_te, y_tr, y_te = train_test_split(X_leaky, y, test_size=0.25, stratify=y, random_state=0)
clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("With leaky feature   - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

# Train WITHOUT the leaky feature
X_clean = df_enc.drop(columns=["churned", "last_billing_status"])
X_tr, X_te, y_tr, y_te = train_test_split(X_clean, y, test_size=0.25, stratify=y, random_state=0)
clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("Without leaky feature - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

Run it and the leaky model lands at an AUC near 1.0 — effectively perfect — while the clean model sits in the high 0.7s. The reason isn’t subtle: last_billing_status is the label wearing a disguise (it is cancelled exactly when churned == 1), so the classifier finds a flawless separating rule. That rule does not exist at prediction time, when the billing status hasn’t been written yet.

In production, the leaky model collapses: last_billing_status is unknown at prediction time — or, worse, is always active because the prediction happens before any cancellation. The ordinary-looking clean model is the one that survives contact with reality.

How to catch this: Any feature with suspiciously high importance is your first suspect. Ask: “Was this feature value available at the moment I’d want to make the prediction?” If not, drop it.

Form 2: Train-test contamination

You fit a StandardScaler (or imputer, or PCA) on the full dataset before splitting. The scaler now “knows” the mean and standard deviation of the test data. Your test set isn’t held-out anymore — its statistics have been baked into the preprocessing.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=2000, n_features=15, n_informative=8, random_state=0)

# --- LEAKY: scale on the full dataset, then split ---
scaler = StandardScaler().fit(X)
X_scaled = scaler.transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y, test_size=0.25, stratify=y, random_state=0)
clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("Leaky scaling    - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

# --- CLEAN: split first, fit scaler on train only ---
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
scaler = StandardScaler().fit(X_tr)         # ONLY train
X_tr_s = scaler.transform(X_tr)
X_te_s = scaler.transform(X_te)             # apply same transform to test
clf = LogisticRegression(max_iter=1000).fit(X_tr_s, y_tr)
print("Clean scaling    - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

The gap is small here because StandardScaler leaks only mean/std — but with more aggressive preprocessing (PCA, target encoding, SMOTE) it can be huge. The rule is: any preprocessing step that learns parameters from data must be fit only on the training set. The transform is then applied to test data using those train-derived parameters.

The robust fix isn’t to remember this every time — it’s to put your preprocessing inside a Pipeline.

Form 3: Time leakage

You have time-series data (customer events, financial transactions, sensor readings). You do a random train/test split — and your training set contains rows from June while your test set contains rows from May.

The model has effectively seen the future when predicting the past. Performance is unrealistically good because future patterns leak backward.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, TimeSeriesSplit
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 1200
df = pd.DataFrame({
    "day":     np.arange(n),                           # day 0 to day 1199
    "feature": rng.normal(size=n),
})
# The relationship between feature and label drifts over time
df["churned"] = ((df["feature"] + 0.001 * df["day"] + rng.normal(0, 0.5, n)) > 0.8).astype(int)

# --- LEAKY: random split ignores time ---
X = df[["day", "feature"]]
y = df["churned"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("Random split       - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

# --- CLEAN: time-ordered split — train on the past, test on the future ---
df_sorted = df.sort_values("day").reset_index(drop=True)
cutoff = int(len(df_sorted) * 0.75)
X_tr, X_te = df_sorted[["day", "feature"]].iloc[:cutoff], df_sorted[["day", "feature"]].iloc[cutoff:]
y_tr, y_te = df_sorted["churned"].iloc[:cutoff], df_sorted["churned"].iloc[cutoff:]
clf = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("Time-ordered split - AUC:", round(roc_auc_score(y_te, clf.predict_proba(X_te)[:, 1]), 3))

The random split’s AUC overstates real performance because the model implicitly knew the drift direction. The time-ordered split is honest — that’s the number you’ll actually see in production.

For time-series cross-validation, use TimeSeriesSplit instead of plain KFold. It generates folds where each training set strictly precedes its test set.

The fix: Pipeline + fit only on train

Wrap every preprocessing step inside a scikit-learn Pipeline. When cross-validation or fit is called, every step’s fit runs only on the current fold’s training data, and transform is applied separately to the validation data. Leakage becomes structurally impossible.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=2000, n_features=15, n_informative=8, random_state=0)

# Inject some missing values to require imputation
rng = np.random.default_rng(0)
mask = rng.random(X.shape) < 0.05
X_miss = X.copy()
X_miss[mask] = np.nan

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale",  StandardScaler()),
    ("clf",    LogisticRegression(max_iter=1000)),
])

scores = cross_val_score(pipe, X_miss, y, cv=5, scoring="roc_auc")
print(f"CV AUC: {scores.mean():.3f} ± {scores.std():.3f}")
print("(each fold's imputer + scaler fit only on the fold's training data — no leakage)")

This is the production-grade pattern. cross_val_score calls pipe.fit(X_train_fold, y_train_fold) for each fold, which trains the imputer, scaler, and classifier only on that fold’s training portion. The transform is then applied to the fold’s validation portion. There is no way for test statistics to bleed into training. Trust this and build everything on top.

A few less-obvious leakage sources

• Group leakage. Same user appears in train and test (multiple rows per user). Use GroupShuffleSplit or GroupKFold keyed on user_id.
• Target encoding without folding. Encoding category → mean(y) on the full data leaks the target into the feature. Use out-of-fold target encoding.
• Information from external joins. Joining in an aggregated table (“user’s average purchase over all time”) that was computed using rows that include the future. Always re-aggregate per-fold or per time window.
• Hyperparameter tuning on the test set. Tuning on what you call “test” is a form of leakage too — that’s why you need a separate validation set or nested CV (covered in hyperparameter-tuning.mdx).

In one breath

• Data leakage is when information from the held-out set or the future sneaks into training, so scores look brilliant in development and collapse in production.
• Target leakage: a feature is secretly the label in disguise, or recorded after it. Ask of every feature — “was this value available at the moment I’d predict?”
• Train–test contamination: preprocessing fit on all the data before the split. Fit on the training portion only.
• Time leakage: a random split on time-stamped data lets the model peek at the future. Split by time instead.
• One discipline closes most of it — put every learned preprocessing step inside a Pipeline, so each fold fits on its own training data and leakage becomes structurally impossible.

Quick check