Bias–variance & learning curves

It is 3 a.m. Your refund-abuse classifier reports 99.8% accuracy on its training examples and 71% on recent held-out transactions. Someone suggests collecting another million transactions. Someone else suggests a deeper model. A third person reaches for regularization because that is what people reach for when a model looks “overfit.”

Several may help, but this gap alone does not tell us which. Use a learning curve and audit leakage and the evaluation split before choosing among:

• more data;
• regularization; or
• a different model.

The bias–variance framework gives you that explanation. A learning curve is the practical plot: training and validation error measured at several training-set sizes. Its shape helps identify whether the model needs more expressive power, more examples, or more restraint.

Two ways to be wrong

Suppose our model predicts whether a customer refund is abusive. Inputs might include account age, refund amount, purchase history, and time between delivery and refund.

A model can fail because it is too simple to represent the useful pattern. That is bias: consistent error caused by restrictive assumptions. A straight line for a curved relationship has high bias and underfits.

A model can also fail because it is too flexible for the available evidence. That is variance: sensitivity to the particular examples in the training set. A deep tree may memorise one customer’s unusual purchase history as a general rule. With a different sample, it builds different rules. This is overfitting.

Here, bias does not mean unfairness toward a protected group. Predictive bias and fairness are separate concepts.

Capacity is how many relationships a model can express. A deep tree has more capacity than a shallow tree; regularization reduces effective capacity. Too little capacity leaves signal unfitted. Too much lets the model fit noise. The useful middle reduces bias without adding excessive variance.

What the decomposition actually says

For squared-error prediction under explicit assumptions, with D a possible training set and f_D(x) the resulting prediction, let g(x)=E[Y|X=x] and Y=g(x)+ε, where E[ε|x]=0. Averaging over possible training sets and fresh target noise gives:

E[(f_D(x)-Y)^2] = Var(ε|x) + (E_D[f_D(x)]-g(x))^2 + Var_D(f_D(x)).

The terms are:

• Irreducible noise, Var(ε|x): target randomness the available input cannot predict.
• Bias squared, (E_D[f_D(x)]-g(x))^2: how far the average learned rule is from the conditional-mean signal.
• Variance, Var_D(f_D(x)): how much predictions change across possible training sets.

For one refund transaction, imagine four training sets produce abuse probabilities of 0.2, 0.4, 0.6, and 0.8. Their mean is 0.5.

If the conditional abuse probability is g(x)=0.6, then Bernoulli target noise is 0.6(1-0.6)=0.24.

So:

• squared bias is (0.5 - 0.6)^2 = 0.01;
• variance is the average squared deviation from 0.5, or 0.05;
• expected squared error is 0.01 + 0.05 + 0.24 = 0.30.

The model cannot remove 0.24 by becoming more complex. It needs one of three things:

• better features;
• a less noisy target; or
• acceptance of that limit.

The thought experiment explains the practical terms: high variance means an unstable learned rule; high bias means the average rule is consistently wrong.

Why regularization can help

A regularizer penalizes complicated parameters or decisions. L2 penalizes large weights; L1 can push weights to zero. This stops the model chasing every small training fluctuation.

Regularization may increase bias slightly, but reduce variance more, lowering validation error. The same trade-off explains why shallower trees, fewer polynomial terms, or stronger early stopping can improve a model with an excellent training score. The goal is performance on new data, not a perfect training fit.

The learning curve tells you which way to turn

A learning curve varies training-set size while holding the model and procedure fixed. At each size, measure:

• Training error on examples used for fitting.
• Validation error on held-out examples.

Use cross-validation or repeated subsamples to show variation between folds. Do not confuse a learning curve with a capacity curve: the latter changes model flexibility and asks whether the model needs to be simpler or more powerful.

“High” is relative to the business goal. The curve diagnoses the source of error, not the acceptable threshold.

A worked curve

For example:

The tiny training error, improving validation error, and closing gap indicate high variance. More representative data may help.

Training error can rise with more examples because a fixed-capacity model has more cases to fit. Validation trend and gap matter more than that rise alone.

A small experiment with scikit-learn

This code creates learning-curve data for an unrestricted decision tree. It uses five-fold cross-validation and converts accuracy to error with 1 - accuracy.

import numpy as np

from sklearn.datasets import make_classification
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(
    n_samples=1500,
    n_features=20,
    n_informative=6,
    random_state=0,
)

sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=None, random_state=0),
    X,
    y,
    train_sizes=np.linspace(0.1, 1.0, 6),
    cv=5,
    scoring="accuracy",
)

train_error = 1 - train_scores.mean(axis=1)
val_error = 1 - val_scores.mean(axis=1)

print(f"{'n_train':>8} {'train_err':>10} {'val_err':>9} {'gap':>7}")
for n, train, val in zip(sizes, train_error, val_error):
    print(f"{int(n):8d} {train:10.3f} {val:9.3f} {val - train:7.3f}")

For a deep tree, training error may be near zero while validation error remains higher. Falling validation error at the largest size supports collecting more data. A flat curve with a large gap supports regularization or a simpler tree.

For losses such as log loss, use loss directly. Convert higher-is-better scores only when you want an error graph.

The production decision

1. Choose the evaluation target. Raw accuracy may be wrong when false positives and false negatives have different costs. Pick a metric and threshold that reflect the decision. See picking the right metric.

2. Make the split honest. Keep a final test set untouched. Use cross-validation on the training portion. Group related customers or devices, and use chronological splits when predicting the future. See train, test, and cross-validation.

3. Vary training size only. Hold the model, features, preprocessing, metric, and split policy fixed so the curve answers one question.

4. Run the smallest justified experiment.

   • High errors with a small gap: add capacity or informative features.
   • Low training error with a large, closing gap: add trustworthy data or regularize.
   • Low and close errors: test the untouched set and deployment path.
   • Unstable folds: investigate groups, rare classes, sample size, and the split.

When the diagnosis lies

Bias and variance describe a modelling setup, not a permanent property of an algorithm. The curve can be distorted by:

Leakage

A post-outcome field such as refund_approved_at can make both errors look low because it records a later human decision.

• Remove unavailable features.
• Split before fitting transformations.
• Audit feature timestamps.

See data leakage — the silent model killer.

The wrong validation split

Randomly splitting rows from the same customer can leak customer information across the split. Use entity-based splits for unseen customers and chronological splits for future predictions. Choose the split that matches the prediction-time population.

Noisy labels and missing signal

If investigators disagree on 15% of refund labels, a flexible model may reduce training error while validation error plateaus.

• Inspect disagreements.
• Improve the label policy.
• Add missing evidence.
• Accept the ceiling.

A model cannot infer a fact absent from the features.

Distribution shift

A policy change can alter customer behaviour while historical curves remain healthy. Monitor live error and feature distributions by time and important subgroups.

• Reweight.
• Retrain on recent data.
• Redesign the target.

The honest limits

“Add more data” is promising when training error is low, validation error is higher, and the gap is shrinking. It still has diminishing returns.

New data may:

• repeat easy cases;
• preserve label noise;
• come from the wrong population; or
• omit the needed feature.

Collection also has costs:

• money;
• privacy review; and
• time.

More capacity cannot recover information absent from the feature set. Better feature engineering or labels may matter more. The curve narrows the search; it does not replace domain knowledge or an honest evaluation.

In one breath

• Bias is systematic error from restrictive assumptions; variance is instability across training samples.
• High, close training and validation error suggests underfitting. Low training error with higher validation error suggests overfitting.
• A learning curve varies data size; a capacity curve varies model flexibility.
• High bias calls for capacity or useful features. High variance with a closing gap may reward more data; regularization can help immediately.
• Leakage, unrealistic splits, noisy labels, and distribution shift can fool both curves.
• Regularization accepts some bias to reduce variance and improve future error.

Quick check

Next

Once you can diagnose the fit, the treatments become less mysterious:

• regularization for high variance;
• feature engineering for high bias; and
• honest cross-validation to measure both.