Bias–variance & learning curves

If you learn one diagnostic in classical ML, make it this one. When a model underperforms, you have a fork in the road: get more data, add capacity, or add regularization — and choosing wrong wastes weeks. The bias–variance framework, read off a learning curve, tells you which way to go.

Two ways to be wrong

Total error decomposes into two sources you trade off against each other:

• Bias — error from a model too simple to capture the real pattern. A line fitting a curve is high-bias. It’s wrong in a consistent, systematic way. This is underfitting.
• Variance — error from a model so flexible it fits the noise in the training set. Resample the data and it learns something wildly different. This is overfitting.

Model capacity trades one for the other: too little → high bias; too much → high variance. (This is why L1/L2 regularization exists — it deliberately adds a little bias to cut a lot of variance.)

The learning curve tells you which one you have

Plotting train and validation error against capacity gives the classic U-curve. But the more actionable view plots them against training-set size — the learning curve. Its shape is a direct diagnosis:

Read it like this:

That middle column is the payoff: a high-bias model and a high-variance model both have “bad validation error,” but the fix is the opposite. The learning curve is how you tell them apart instead of guessing.

import numpy as np
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1500, n_features=20, n_informative=6, random_state=0)

# A deep tree = high variance. Watch the gap between train and CV error.
sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=None, random_state=0),
    X, y, train_sizes=np.linspace(0.1, 1.0, 6), cv=5, scoring="accuracy",
)
train_err = 1 - train_scores.mean(axis=1)
val_err   = 1 - val_scores.mean(axis=1)

print(f"{'n_train':>8} {'train_err':>10} {'val_err':>9} {'gap':>7}")
for n, te, ve in zip(sizes, train_err, val_err):
    print(f"{int(n):8d} {te:10.3f} {ve:9.3f} {ve-te:7.3f}")
print("\nTrain error ~0 but a large, slowly-shrinking gap = high variance.")
print("Fix: limit max_depth, or add more data (the gap is still closing).")

In one breath

• Total error splits into bias (too simple → systematic error → underfit) and variance (too flexible → fits noise → overfit); model capacity trades one against the other.
• The learning curve (train + validation error vs training-set size) is the actionable diagnostic — its shape is the diagnosis.
• Both errors high, small gap = high bias → add capacity/features (more data won’t help). Both low, small gap = good fit → ship. Big gap (low train, high val) = high variance → more data if the gap is still closing, or regularize/simplify.
• The most expensive mistake is throwing more data at a high-bias model — it lands in the same place; read the curve first.
• This is why regularization exists: deliberately add a little bias to cut a lot of variance.

Quick check

Next

Once you can diagnose the fit, the treatments follow: regularization for high variance, richer feature engineering for high bias, and honest cross-validation to measure it all.