Overfitting & bias–variance

Your house-price model predicts the 800 homes in its training set with a median error of $4,000. Excellent. Then it sees 200 homes held out from training and misses by $38,000.

Nothing is wrong with the calculator. The model learned the training set too closely, picking up quirks that belonged to those 800 houses: perhaps a neighbourhood code, a photographer’s camera, or unusual renovations. Those details helped on yesterday’s examples and hurt on today’s.

The useful question is not “is the model accurate?” It is:

Will the model keep finding the underlying pattern when the examples change?

That property is generalization, meaning performance on data the model did not train on. Overfitting is a failure of generalization. Bias and variance help diagnose why it happened.

Two ways to be wrong

Imagine predicting house prices from size, bedrooms, and location.

A model that can only draw a straight-line relationship might say every extra square metre adds the same amount everywhere. Real prices are less tidy: a 90-square-metre flat in the city centre does not behave like one beside an airport.

That model makes a bias error: assumptions are too simple to represent the real pattern. This is underfitting, where the model has not learned enough structure.

A flexible model with millions of parameters and only 800 homes can fit the broad relationship, then carve out rules for individual homes. It might learn that listing photo identifier 418 predicts an expensive house, even though the identifier has no causal meaning.

That model makes a variance error: its predictions change too much when the training sample changes. This is overfitting, where training-set noise is learned as a real rule.

The word variance is easy to misread here.

A small model tends to have high bias and low variance; a large model tends to have lower bias and higher variance. This is a tendency, not a law. A badly optimized large model can underfit, and a well-designed small model can win. Optimization, data quality, and features matter too.

The arithmetic behind the trade-off

Freeze one house and imagine retraining on four different training samples. Suppose its predictable, noise-free price is 11 units, where one unit means $10,000. The underlying target is therefore $110,000.

The simple model predicts:

8, 9, 10, 9

Its average prediction is 9, so its squared bias is:

(9 - 11)^2 = 4

Its variance is the average squared distance from its own average:

((8 - 9)^2 + (9 - 9)^2 + (10 - 9)^2 + (9 - 9)^2) / 4 = 0.5

Ignoring measurement noise, its expected squared error is:

bias squared + variance = 4 + 0.5 = 4.5

The flexible model predicts:

7, 11, 13, 9

Its average is 10, giving squared bias:

(10 - 11)^2 = 1

But its variance is:

((7 - 10)^2 + (11 - 10)^2 + (13 - 10)^2 + (9 - 10)^2) / 4 = 5

Its total is:

1 + 5 = 6

The flexible model is less systematically wrong, but its predictions vary so much that its total error is worse in this example.

For squared-error regression, the full decomposition adds irreducible noise, randomness in the target that available information cannot predict:

expected test error = bias squared + variance + irreducible noise

A final sale price may depend on a buyer’s mood or an unrecorded fact. More layers cannot recover a missing feature or predict a genuinely random event.

This is a mental model, not a quantity you can calculate from one run. In practice, compare training and held-out performance, change capacity, and check whether results are stable across splits.

Why capacity makes a U-shaped curve

Capacity is how many patterns a model can represent. Parameters affect it, but so do depth, width, architectural constraints, regularization, and data.

As capacity rises, training error usually falls: a larger network can represent what a smaller one can, then use extra flexibility to fit more closely. Test error first falls as real structure becomes learnable, then rises when extra flexibility mostly fits training-sample quirks. The minimum is the useful capacity range, or “sweet spot”; it depends on the data, loss, noise, and future test distribution.

You cannot watch the true test curve while choosing a model. Repeatedly inspecting the test set makes it part of training through indirect exposure. Use a validation set as a working proxy.

Reading a train-validation curve

Split available data into:

• a training set, used to update parameters;
• a validation set, held aside while training but used to choose settings;
• optionally, a test set, untouched until final evaluation.

For the 1,000-home example, 800 training homes and 200 validation homes is one reasonable split. The exact percentages are not sacred. Small datasets may need cross-validation; time-dependent problems may need chronological splits.

Record the same metric on both sets. For regression, this might be mean absolute error (MAE), the average absolute difference between predicted and actual price.

Read the pattern:

• Both errors high and similar: usually underfitting, though a bad learning rate or input pipeline can look the same.
• Training error high and still improving: train longer, check the learning rate, increase capacity, or improve inputs before adding regularization.
• Training error low while validation error falls then rises: classic overfitting. Useful structure was learned first; training-specific detail came later.
• Both errors low with a modest gap: usually the desired outcome. Whether a gap is acceptable depends on the business.

Here is the curve over epochs:

The validation minimum is evidence, not an oracle: finite validation sets wiggle. Use a patience window instead of stopping at the first non-improvement.

Early stopping: save the best model, not the last one

Early stopping ends training when a held-out metric stops improving. Neural networks often learn broad, repeatable patterns before fitting smaller, training-specific details, so stopping limits time in that second phase.

The production pattern is:

1. Choose a validation metric and whether lower or higher is better.
2. Save a checkpoint whenever it reaches a new best.
3. Stop after a chosen number of unimproved evaluations, then restore the best checkpoint.

“Restore” matters. If validation loss was best at epoch 7 and patience expires at epoch 10, epoch-10 weights are not the ones you want.

This PyTorch example uses a validation loss where lower is better. The val_loss list stands in for values returned by a real evaluation pass:

import copy
import torch

model = torch.nn.Linear(1, 1)

val_loss = [
    0.92, 0.71, 0.55, 0.44, 0.38,
    0.34, 0.31, 0.30, 0.305, 0.31,
    0.32, 0.34, 0.37, 0.40, 0.43,
    0.45, 0.48, 0.50, 0.52, 0.55,
]

best = float("inf")
best_epoch = -1
best_state = None
patience = 3
wait = 0

for epoch, value in enumerate(val_loss):
    # In a real loop, training batches and optimizer.step() run before this.
    model.eval()

    if value < best:
        best = value
        best_epoch = epoch
        wait = 0
        best_state = copy.deepcopy(model.state_dict())
    else:
        wait += 1
        if wait >= patience:
            print(
                f"early stop at epoch {epoch} "
                f"(no improvement for {patience})"
            )
            break

if best_state is None:
    raise RuntimeError("No validation result was recorded")

model.load_state_dict(best_state)

print(
    f"best val loss {best:.3f} at epoch {best_epoch} "
    "— restored those weights"
)

early stop at epoch 10 (no improvement for 3)
best val loss 0.300 at epoch 7 — restored those weights

copy.deepcopy matters because a state_dict contains references to tensors; a shallow reference can change during later training. load_state_dict restores the saved parameters and buffers.

Python counts from zero, so best_epoch 7 is the eighth list item. If a dashboard starts at epoch 1, that checkpoint is usually called epoch 8.

The example saves only model weights for evaluation or inference. To resume training, also save the optimizer state—and scheduler state, if used—because optimizers such as Adam keep moving averages. Run validation without updates: use evaluation mode and disable gradient calculation.

Choose patience for validation noise and evaluation cost. A small validation set or noisy metric may need ten or more evaluations. A min_delta threshold can require a meaningful improvement instead of treating 0.3000 to 0.29999 as a win.

Early stopping is not a substitute for a final test set. If you use the same validation set to select architecture, learning rate, augmentation, patience, and many random seeds, you can overfit that validation set too.

What to change after the diagnosis

Suppose training MAE reaches $8,000 while validation MAE bottoms at $19,000 and then climbs to $31,000. That is a variance problem until evidence says otherwise.

Useful levers include:

• More varied data: makes accidental features less reliable.
• A smaller model: removes patterns the dataset cannot support.
• Weight decay: prefers lower-magnitude, often smoother solutions.
• Dropout: reduces reliance on particular combinations of units, but can hurt when used too aggressively.
• Data augmentation: encodes valid invariances; changing a house’s location feature would not preserve its price.
• Early stopping: limits updates and keeps the best validation checkpoint.

These are not interchangeable. Data reduces uncertainty in the sample; a smaller model attacks excess capacity; weight decay changes preferred parameters; augmentation encodes assumptions; early stopping limits optimization.

If both training and validation MAE are $42,000, first check labels, scaling, the learning rate, and model capacity. Dropout is not an obvious fix. A model cannot infer a missing cause from optimism.

Regularization has a cost: it can increase training error, slow convergence, cause underfitting, or erase rare but real patterns. The goal is not the smallest train-validation gap; it is the best performance on the future distribution you care about.

When the curve lies

A rising validation loss is a clue, not proof of overfitting.

• A sudden validation jump may be a pipeline bug: separately fitted preprocessing, shifted labels, or random training augmentation applied to validation. Inspect raw and transformed examples, labels, and predictions.
• Spectacular validation performance followed by production failure suggests leakage. Data from the same patient, property, user, or device can appear in both splits. Split by the unit that must generalize, and fit scalers, vocabularies, and feature-selection rules on training data only.
• Repeatedly improving validation results but a poor final test score may mean you overfit the validation set through model selection. Keep the test set locked; use cross-validation or another holdout when necessary.
• Sharp epoch-to-epoch oscillation may indicate a large learning rate, a tiny validation set, or an unstable metric rather than excess capacity. Patience handles noise, not a validation set containing 17 examples.
• Bad metrics with a large gap can reflect noisy labels, imbalance, or missing features. Inspect per-class or per-segment results.

A random split is not automatically honest. Leakage means using target, future, or otherwise withheld information in features, preprocessing, fitting, or model-selection decisions.

Bias and variance are about the whole setup

Bias and variance depend on the task, dataset, loss, and training procedure, not just architecture. A model can perform well on randomly sampled homes and fail on homes built after a zoning change. That is distribution shift; it may require time-based evaluation, new features, monitoring, or retraining rather than more regularization.

High capacity is not automatically bad. Modern networks can have more parameters than examples and still generalize because architecture, optimization, data diversity, and learned structure constrain the solutions they find. The U-shaped picture is a useful diagnostic, not a guarantee.

In one breath

• Bias is systematic error from assumptions or a model that is too simple; variance is sensitivity to the particular training sample.
• Underfitting usually shows high, similar training and validation error. Overfitting shows low training error while validation error rises or remains much worse.
• As capacity increases, training error generally keeps falling, while held-out error often falls and then rises.
• A validation set guides choices. A test set measures the final choice and should remain untouched.
• Early stopping saves every new best validation checkpoint, waits through patience, and restores the best weights.
• Regularization can reduce variance, but cannot repair leakage, broken labels, missing features, or distribution shift.

Quick check

Next

Now that you can diagnose the shape of the problem, see the main treatments: dropout, BatchNorm, and LayerNorm. For a clean account of how data enters a model and gets updated, revisit the training loop. And to make each epoch count, see how batch size and learning rate shape optimization.