Walk me through how you'd select between competing models without fooling yourself with data leakage.

How to think about it

I would freeze a representative test set before experimentation, use only the remaining development data for preprocessing, feature selection, tuning, and model comparison, then refit the chosen pipeline on all development data and evaluate it on the test set once. The split must also match production: random folds for independent rows, group folds for repeated entities, and forward-chaining folds for time-ordered data.

Why leakage changes the question

A validation score is meant to estimate how the model will perform on future cases. Data leakage makes the validation case easier than the production case by allowing information from the future, the label, or the held-out rows to influence training.

That influence can be obvious. A churn model might use cancellation_reason, even though that field is filled in only after the customer cancels. The model has not learned churn behaviour. It has learned a receipt for the answer.

It can also be subtle. Suppose I calculate a median, standard deviation, vocabulary, selected feature set, or target encoding using the complete dataset before cross-validation. The validation rows have now influenced the representation used to judge them. The model may never receive their labels directly, but the boundary between training and validation is already contaminated.

There is a second trap: model selection itself is a form of learning. If I try 40 algorithms, feature sets, random seeds, and thresholds, then choose whichever validation score is highest, I have used the validation set repeatedly. Some of that winning score may be luck. The validation set has gradually become another training signal.

The test set exists to detect this optimism. It cannot do that if I keep checking it while making decisions.

A concrete example

Suppose I am predicting whether 60,000 subscribers will cancel within 30 days. Each row is one account snapshot taken on 1 June, and the label says whether that account cancelled by 1 July. The positive rate is 12 percent.

Because there is one snapshot per account and the deployment population is similar to this one, I could reserve 12,000 accounts as a stratified test set. A stratified split preserves the class ratio, so the test set contains roughly the same proportion of churners. The other 48,000 rows form the development set.

I would then use five-fold cross-validation on those 48,000 rows. Every candidate model sees the same folds. The test labels remain locked away.

Here is the important implementation pattern:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["tenure_days", "monthly_spend", "failed_payments_30d"]
categorical = ["plan", "region"]

preprocess = ColumnTransformer(
    [
        (
            "numeric",
            Pipeline(
                [
                    ("impute", SimpleImputer(strategy="median")),
                    ("scale", StandardScaler()),
                ]
            ),
            numeric,
        ),
        (
            "categorical",
            Pipeline(
                [
                    ("impute", SimpleImputer(strategy="most_frequent")),
                    ("onehot", OneHotEncoder(handle_unknown="ignore")),
                ]
            ),
            categorical,
        ),
    ]
)

pipeline = Pipeline(
    [
        ("preprocess", preprocess),
        ("model", LogisticRegression(max_iter=2000)),
    ]
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)

search = GridSearchCV(
    pipeline,
    param_grid={"model__C": [0.1, 1.0, 10.0]},
    scoring="roc_auc",
    cv=cv,
    refit=True,
    n_jobs=-1,
)

search.fit(X_dev, y_dev)
test_auc = roc_auc_score(y_test, search.predict_proba(X_test)[:, 1])

The pipeline matters. For each fold, the imputer, scaler, and one-hot encoder are fitted on that fold’s training portion, then applied to its validation portion. They are not fitted once on all 48,000 development rows.

After cross-validation chooses the value of C, refit=True fits the selected pipeline on all development data. That is safe because the test set has still not been used.

Imagine two candidates produce these illustrative fold scores:

Candidate	Fold AUCs	Mean AUC
Logistic regression	0.77, 0.79, 0.78, 0.80, 0.77	0.782
Gradient-boosted trees	0.78, 0.80, 0.79, 0.79, 0.78	0.788

I would not declare the second model the winner merely because its mean is higher by 0.006. I would inspect the paired fold differences, variation across folds, calibration, inference cost, and performance in important customer segments. A tiny offline improvement is not automatically worth a slower or less stable service.

The split must match production

A random split is not a universal default. It is a claim about how future data will arrive.

Data situation	Appropriate evaluation	Why
Independent rows	Stratified random split or shuffled folds	Each row is a reasonable stand-in for a future row
Multiple rows per customer or device	Grouped split	Related rows cannot give the model an accidental identity shortcut
Forecasting or changing behaviour	Forward-chaining split	Training uses only information available before validation
New sites, hospitals, or regions	Site or region holdout	Tests whether the model travels to a new group

A group split keeps every row belonging to one entity in the same partition. If the same customer’s January and February records land in different random folds, the model may recognise the customer rather than learn a general pattern.

Whether that is leakage depends on the product. Predicting next month’s churn for existing customers may legitimately use the customer’s earlier history. Predicting churn for brand-new customers is a different task and needs an unseen-customer evaluation.

For time-dependent churn data, I might use forward-chaining splits such as:

• Train on January through March, validate on April.
• Train on January through April, validate on May.
• Train on January through May, validate on June.

I would also check label availability. If the label covers the next 30 days, a late-May snapshot may not have a resolved label when June predictions are supposedly made. Overlapping outcome windows or delayed data may require a gap, sometimes called a purge, between training and validation.

The same rule applies to features. A “last 30 days” spend feature must query events up to the prediction timestamp, not up to the date when the dataset was assembled. A scikit-learn pipeline cannot repair a warehouse query that has already read tomorrow’s transactions.

The protocol I would follow

Before fitting anything, I would write down:

1. The prediction unit: account, transaction, session, or something else.
2. The prediction timestamp and the exact label window.
3. The metric and the decision threshold.
4. The split strategy and the test-set boundary.
5. The candidate models and tuning budget.

The metric must reflect the decision. With 12 percent churn, ROC AUC measures ranking quality, but it does not tell the business how many of the top 1,000 contacted customers will actually churn. I might also report precision at the contact capacity, recall, calibration, and expected campaign value. Choosing the metric after seeing results is another way to overfit.

All learned steps belong inside the fold: imputation, scaling, encoding, feature selection, dimensionality reduction, oversampling, and target encoding. Feature generation from event tables needs the same discipline, even if it happens before Python. I would build features as-of each prediction timestamp and version that logic.

For comparison, I would use identical folds and scoring rules for every candidate. I would record fold-level scores rather than only one average. Once the model, features, threshold, and operating point are frozen, I would refit on the full development set and score the test set once.

If the test result is disappointing, I would not quietly tune against it. I would treat it as new evidence, revise the protocol, and obtain a fresh untouched evaluation set.

When nested cross-validation is the better answer

Nested cross-validation is useful when I need an honest estimate of the entire tuning process and do not have a genuinely untouched test set.

The outer loop holds out data for evaluation. Inside each outer training portion, an inner loop tunes hyperparameters and selects features. The chosen pipeline is then fitted on the outer training portion and evaluated on the outer holdout. Because the outer holdout never participates in tuning, its score estimates the full selection procedure rather than one already-optimised model.

With a large, representative external test set, development cross-validation plus one final test evaluation is usually simpler. Nested cross-validation costs many more model fits, so I would use it when the estimate matters more than the extra compute.

What failure looks like first

A classic symptom is an offline AUC of 0.99 followed by an online AUC near 0.71 during the first month. I would inspect features with suspiciously high importance and ask whether each was available at the prediction timestamp. In the churn example, cancellation_reason or a post-cancellation support ticket would be immediate suspects.

Other warning signs include:

• A target-encoded feature produces an implausibly large validation improvement.
• Random cross-validation looks excellent, but a time-based backtest collapses.
• Performance changes dramatically when the random seed changes.
• Near-duplicate customers or transactions appear in both training and validation.
• The test score is checked repeatedly until it becomes acceptable.

A clean test score is also only an estimate for the population and time period it represents. If pricing, customer behaviour, or data collection changes, the old test set may be clean but no longer representative.

What they’ll ask next

“Is fitting a scaler before the split always a serious leak?”

It is technically leakage because the scaler learns statistics from held-out rows. The numerical impact may be tiny when the dataset is large and the distributions are stable, but the safe and reproducible rule is to fit it inside the pipeline. The same rule is much more important for target encoding, feature selection, and oversampling because those steps can carry stronger information about the labels.

“How would you compare two models whose AUCs are 0.782 and 0.788?”

I would use the same folds, compare paired fold-level differences, inspect uncertainty and calibration, and check the business metric and operational cost. I would not treat six thousandths as meaningful without evidence that the difference is stable and useful.

“When would you avoid a random split?”

Whenever rows share an entity, time, location, or other dependency that will not be independent in production. I would use grouped or forward-chaining evaluation and make sure every feature is constructed from information available at the relevant cutoff.

For a deeper treatment, see model selection and nested cross-validation.

Say this in the interview: “I would lock a representative test set, run every preprocessing and tuning step inside identical development folds, choose using a production-relevant metric, and evaluate the frozen pipeline on the test set once with a split that matches how the model will actually be used.”