The Bias-Variance Trade-off

Last lesson left ridge tugging on one dial — bias up, variance down — and promised those two words run deeper than any single technique. Here is how deep. Picture two archers at the same target. The first has a crooked sight: every arrow lands tight together, but a foot to the left of the bullseye, every time. The second has a true sight but a shaky hand: the arrows scatter all around the centre, some close, some wild. Both miss — but they miss for opposite reasons, and no single fix helps both.

The crooked-sight archer is biased: a systematic error that points the same wrong way no matter how many arrows fly. The shaky-hand archer has high variance: no systematic offset, just sensitivity to every twitch. A model errs in exactly these two ways, plus a third the archer also faces — a gust of wind no skill can predict, the irreducible noise in the data itself. Separating a model’s error into these three is the single most useful lens in machine learning, and it explains why you can never drive the miss to zero: steady the hand and you often skew the sight.

The decomposition and the U-curve

Expected prediction error splits into three additive pieces:

Expected error  ≈  bias²  +  variance  +  irreducible noise

As a model grows more complex, its bias falls — it can now express more of the truth — while its variance rises, because it reacts ever more to the particular training sample it saw. Their sum traces a U: too simple on the left, too flexible on the right, and a lowest point in between that generalises best.

The handles you can pull, and which term each moves:

• Lowers variance: more training data, regularization (such as ridge’s λ), simpler models, bagging or averaging ensembles.
• Lowers bias: more expressive models (higher-degree polynomials, deeper trees), adding informative features.

Almost every lever that cuts one term raises the other. That tension is the trade-off.

How GATE asks this

Reliably an MCQ or MSQ on the direction of an effect: “increasing model complexity does what to bias and variance?”, or “which of the following reduce variance?” The answer pattern is fixed — more complexity means lower bias and higher variance; regularization and more data mean lower variance and higher (or unchanged) bias. It also rides quietly inside ridge questions (more λ → more bias, less variance) and inside overfitting/underfitting questions across nearly every paper.

Worked example

Compare two extreme models on the same data, then put numbers on the error.

• A high-degree polynomial can bend through almost every training point. It has low bias (flexible enough to trace the true shape) but high variance — shift the training set slightly and the wiggly curve lurches. This is the overfitting corner, the right side of the U-curve: our shaky-hand archer.
• A constant predictor that always outputs the mean ignores the inputs entirely. It has high bias (it cannot represent any real structure) but low variance — it barely flinches when the training set changes. This is the underfitting corner, the left side: our crooked-sight archer.

Neither extreme generalises. The best model sits between them, where the rising variance and falling bias sum to the smallest total.

Now put numbers on the decomposition. Suppose a tuned model has bias = 0.2, variance = 0.05, and irreducible noise = 0.01:

total error = bias² + variance + noise
            = 0.2² + 0.05 + 0.01
            = 0.04 + 0.05 + 0.01
            = 0.10

The same sum in Python, including a more flexible model where bias drops but variance climbs past it:

bias, variance, noise = 0.2, 0.05, 0.01
total = bias**2 + variance + noise
print(f"tuned total    = {total:.4f}")

bias2, var2 = 0.05, 0.18          # a more flexible model
flexible = bias2**2 + var2 + noise
print(f"flexible total = {flexible:.4f}")

tuned total    = 0.1000
flexible total = 0.1925

So the tuned model’s expected error is 0.10, and the more flexible one is actually worse (0.19) because its variance has overshot. Note it is bias², not bias, that enters the sum.

In one breath

Every model’s expected error splits into three additive parts — bias² (systematic error from a model too simple, the crooked sight), variance (sensitivity to the particular training sample, the shaky hand), and irreducible noise (randomness in the data no model can remove); raising complexity lowers bias but raises variance, so their sum is a U-curve whose minimum is the sweet spot, and the levers that cut one term — more data, regularization, simpler models for variance; richer models, more features for bias — almost always raise the other.

Practice

A question to carry forward

The U-curve promises a sweet spot — one model complexity where total error bottoms out. Lovely in a diagram. But out in the real world you cannot see bias and variance; you cannot read the y-axis of that curve directly. All you hold is a finite pile of data and a model’s score on it.

So the practical question bites: how do you actually locate the bottom of the U when the only instrument you have is “test the model on data it hasn’t seen”? And a single held-out test gives a jittery reading — score 0.84 today, 0.79 on a reshuffle. Here is the thread onward: how do you turn one noisy, luck-of-the-split test into a stable, trustworthy estimate of how a model will generalise — so you can compare complexities and find that sweet spot with confidence?