Why do we split data into train, validation, and test sets, and what are the typical proportions?

How to think about it

The direct answer is that the three sets protect three different jobs: the train set fits model parameters, the validation set helps us choose the model and its settings, and the untouched test set estimates how the finished system will perform on new data. A common starting point is 70 percent train, 15 percent validation, and 15 percent test, but the right proportions depend on dataset size, class balance, repeated entities, and whether the data changes over time.

Why the barrier matters

Suppose a model gets 99 percent accuracy on the examples it trained on. That tells us very little. It may have learned useful patterns, or it may have memorised quirks in those particular rows.

What we care about is generalization, meaning performance on examples the model did not see during development. A held-out set approximates that situation. The split creates an information barrier: the model and the person building it are not allowed to learn from the final evaluation examples.

The train set is where the model learns its internal parameters. For a linear model, these might be coefficients. For a neural network, they are millions or billions of weights. For a decision tree, they include the chosen split points and leaf predictions.

The validation set is different. It does not usually update those parameters directly, but it influences the decisions around them. We use it to choose:

• the model family;
• learning rate and regularization strength;
• tree depth or number of trees;
• input features;
• a classification threshold;
• an early-stopping checkpoint.

These are hyperparameters, which are settings chosen by the practitioner rather than learned as ordinary model parameters from each training example.

That distinction causes a common beginner mistake. A model may never run a gradient update on the validation rows, yet the validation rows still shape the final model because we make decisions after seeing validation scores.

The test set has one narrower job: estimate the performance of the final, frozen recipe. It should be used after model choice, feature design, preprocessing, threshold selection, and tuning are complete.

If we inspect test results, change the model, inspect the test results again, and repeat, the test set has become another validation set. The final score may still look respectable, but it is no longer a clean estimate. We have selected the model partly because it performed well on those particular test examples.

A concrete example

Imagine a fraud classifier built from 100,000 card transactions. Fraud occurs in 2 percent of them, so there are about 2,000 positive examples.

A 70 / 15 / 15 split gives:

Set	Rows	Approximate fraud cases	Job
Train	70,000	1,400	Fit model parameters
Validation	15,000	300	Select model and settings
Test	15,000	300	Report final performance

For an imbalanced problem such as fraud, I would usually use a stratified split so that each set has approximately the same fraud rate. Otherwise, a validation set with unusually few fraud cases could make a model look better or worse by accident.

Suppose we try three candidates on the training set:

• logistic regression with regularization;
• a random forest;
• gradient-boosted trees.

The validation results might look like this:

Candidate	Validation average precision
Logistic regression	0.41
Random forest	0.47
Gradient-boosted trees	0.51

These numbers are illustrative, but the decision process is real. We might choose gradient-boosted trees, tune its depth and learning rate, and select a probability threshold that catches 70 percent of fraud while keeping the manual-review queue manageable.

Only after those choices are frozen do we evaluate on the 15,000 test transactions. Suppose the test average precision is 0.49. That is the number we report as the best estimate of performance on similarly sampled future transactions.

The validation score of 0.51 was useful for choosing the system. The test score of 0.49 is useful for estimating how that chosen system generalizes. They answer different questions.

A two-step split in scikit-learn can produce roughly this arrangement:

from sklearn.model_selection import train_test_split

X_dev, X_test, y_dev, y_test = train_test_split(
    X,
    y,
    test_size=0.15,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_dev,
    y_dev,
    test_size=0.18,
    random_state=42,
    stratify=y_dev,
)

The first operation reserves 15 percent for testing. The second takes 18 percent of the remaining 85 percent for validation, which is about 15.3 percent of the original data. The result is approximately 69.7 percent train, 15.3 percent validation, and 15 percent test.

Typical proportions are starting points, not laws

For a medium-sized, independently sampled dataset, 70 / 15 / 15 or 80 / 10 / 10 are sensible starting points. The validation and test sets need enough examples to make their metrics reasonably stable, while the training set needs enough examples to learn the pattern.

With millions of rows, percentages can shrink. A 98 / 1 / 1 split on one million examples still leaves 10,000 examples in validation and 10,000 in test. Giving two percent back to training may be more valuable than reserving 20 percent for evaluation.

With a small dataset, the opposite problem appears. On 1,000 rows, a 10 percent test set contains only 100 examples. If the positive class rate is 1 percent, that test set may contain just one positive example. A single prediction then changes the apparent recall by a large amount.

For small datasets, I would often use cross-validation on the development data and keep a final test set if enough examples remain. In k-fold cross-validation, we divide the development data into k parts, train k times, and use a different part for validation each time. Five-fold or ten-fold cross-validation gives every development example a turn in validation, which is more data-efficient than throwing away a fixed 20 percent.

Cross-validation does not make the test set unnecessary when the test set is affordable. Repeatedly tuning against cross-validation scores can also overfit to those scores. For especially careful small-data evaluation, nested cross-validation uses an inner loop for tuning and an outer loop for estimating performance.

The absolute number of important cases matters more than the percentage. A one-percent test set with 10,000 fraud cases is useful. A 20-percent test set with three fraud cases is not magically reliable.

What happens after model selection?

Once we have selected the model and frozen the development decisions, there are two common approaches.

The conservative approach is to train on the training set and evaluate on the test set. This keeps the procedure simple and makes the roles obvious.

The data-efficient approach is to refit the selected model on the combined training and validation sets, then evaluate that final model once on the test set. This is valid because the validation data has finished its selection job. More training examples can help, particularly when the dataset is not large.

The final recipe must include more than model weights. It includes the feature definitions, preprocessing, vocabulary, imputation rules, hyperparameters, and decision threshold. If any of those are changed after looking at the test result, the test score becomes part of development.

The split must match how the system will be used

A random split is appropriate only when rows are close to independent and identically distributed, meaning future rows resemble the sampled rows and one row does not reveal another row’s identity.

That assumption often fails.

If several rows belong to the same customer, patient, device, or product, put the same entity in multiple sets and the model may recognise the entity rather than learn a general pattern. Use a group-aware split so that all rows for one entity stay together.

For forecasting or any prediction made about the future, split by time. Train on earlier observations, validate on a later period, and test on the latest period. A random split can put a customer’s December purchase in training while using their November purchase for testing. That is an easy exam score and a bad simulation of deployment.

The test set should also represent the population that matters. If a medical model will serve five hospitals but the test set contains only one, the score may not describe the real operating conditions. Subgroup performance and rare cases can matter more than one overall average.

The preprocessing trap

Preprocessing must follow the same barrier.

For example, a standard scaler computes a mean and standard deviation. If it computes them using all rows before the split, the training features contain information derived from validation and test rows. The leakage may be small for a large dataset, but it is still leakage. Target encoding, feature selection, imputation, vocabulary construction, and synthetic oversampling can leak information in the same way.

Fit each learned preprocessing step on the training data only. Apply the fitted transformation to validation and test data. After development is frozen, fit the final preprocessing pipeline on training plus validation if you are refitting the final model. Never use test rows to fit it.

A pipeline is useful because it keeps these operations together and makes it harder to accidentally transform the full dataset first.

Failure mode you would see first

A classic symptom is an excellent offline score followed by a sharp drop on the next month’s data or in production.

For the fraud example, random splitting may place transactions from the same card in both train and test. A feature such as a card-specific history can then make the test score look unusually strong. In production, a new card or a new time period may not resemble the training data. The first clues are often:

• validation and test scores are suspiciously close to training performance;
• performance collapses when evaluated on a later time window;
• one customer or device appears in several splits;
• a metric swings dramatically when the random seed changes.

The fix is not automatically “use a larger test set.” First check duplicate entities, timestamps, feature availability, preprocessing fit scope, label construction, and subgroup counts. A larger contaminated test set is still contaminated; it merely gives the wrong answer with more decimal places.

What they’ll ask next

Why not use only train and test?
You can, but then every hyperparameter choice and model comparison must be made without looking at the test score. In practice, once you use that score to choose between candidates, it is functioning as validation. A separate test set preserves a final evaluation after development.

Should the final model be trained on the validation set too?
Usually yes, after all choices are frozen. Combining train and validation gives the final model more data, then the untouched test set estimates its performance. The preprocessing and model must be refit on the combined development data, not on the test data.

How would you split a time-series dataset?
Use chronological splits: earlier data for training, a later window for validation, and the latest window for testing. Random splitting can let future information influence the past and produces an evaluation that deployment will not reproduce.

What if the dataset has only 1,000 rows?
Keep a final holdout if possible, but use cross-validation on the remaining development data rather than trusting one tiny validation slice. Report uncertainty, because a single score from a small or rare-event test set can be noisy.

Say this in the interview

“We split data to separate fitting from model selection and final evaluation: train learns the parameters, validation guides development, and a never-used test set estimates generalization; 70/15/15 is a common starting point, but the split must respect sample size, class balance, groups, and time.”