Explain the bias-variance tradeoff and how it relates to overfitting.
Bias comes from a model being too simple to capture the real pattern, while variance comes from a model changing too much when the training data changes. Overfitting is typically the high-variance case: training error is low, but validation error is high because the model has learned noise instead of reusable signal.
How to think about it
The bias-variance tradeoff says that a model can miss the real pattern because it is too rigid, which is high bias, or chase quirks of its training sample, which is high variance. Overfitting is usually the high-variance side: training error keeps falling while validation error rises because the model has learned noise that does not repeat on new data.
Why this happens
Imagine training the same model many times on different samples from the same population. The samples will not be identical. One sample may contain unusually expensive houses; another may contain more small apartments. A good model should make roughly the same prediction for the same kind of new house despite those changes.
Bias is the systematic error caused by a model being too simple or making restrictive assumptions. If the true relationship between floor area and price is curved, a straight-line model will consistently miss that curve. Its predictions may be stable from one training sample to the next, but stably wrong.
Variance is the error caused by a model being too sensitive to the particular training sample. A highly flexible model may treat one unusually expensive 180-square-metre house as evidence of a dramatic price jump. Remove that house, retrain, and the jump disappears. The model has not learned a durable rule; it has learned the sample’s personality.
For regression with squared error, the expected prediction error at one input can be separated into:
expected test error = squared bias + variance + irreducible noise
The last term is randomness no model can remove, such as a buyer’s urgency, a hidden renovation, or a measurement error in the sale price. The decomposition is useful because it explains why lowering training error is not the same as improving predictions in production.
A model with high bias underfits. It cannot represent enough of the real relationship, so both its training and validation errors tend to be high. A model with high variance overfits. It can make training error tiny, but its validation error is much worse because it responds to accidental details in the training data.
The tradeoff appears when changing model complexity. Adding flexibility often lowers bias because the model can represent more patterns. But with limited data, that flexibility also gives each observation more influence, increasing variance. Reducing complexity or adding regularization usually does the reverse: it accepts some bias to make predictions less volatile.
That is a tendency, not a law of nature. More data can reduce variance without increasing bias, and a better feature can reduce bias without making the model unstable.
A concrete example
Suppose we have 400 house listings from one city. We use 300 for training and hold out 100 for validation. The only feature is floor area, but the real market also contains neighbourhood effects and random pricing noise.
We fit three polynomial regression models. A degree-one model is a straight line. A degree-four model can bend several times. A degree-fifteen model is flexible enough to wiggle through individual training observations.
The following is a plausible toy result, with RMSE meaning root mean squared error:
| Model | Training RMSE | Validation RMSE | Reading |
|---|---|---|---|
| Straight line | 31,000 dollars | 35,000 dollars | Too rigid; underfits |
| Degree-four curve | 17,000 dollars | 21,000 dollars | Captures useful curvature |
| Degree-fifteen curve | 2,000 dollars | 74,000 dollars | Memorises the training sample |
The degree-fifteen model looks brilliant if you inspect only the training set. It is not brilliant. It has found a way to pass close to nearly every observed listing, including listings whose prices contain noise. On unseen houses, those wiggles become expensive.
Here is the same idea at a single input. Suppose the true average price for a 180-square-metre house is 280,000 dollars. Train each model on five different samples and record its prediction:
| Model | Predictions across five training samples |
|---|---|
| Straight line | 248,000; 249,000; 250,000; 248,000; 249,000 |
| Degree-fifteen curve | 210,000; 340,000; 275,000; 330,000; 245,000 |
The straight-line model has low variance: its predictions barely move. But its average is about 249,000 dollars, well below the true average. That is high bias.
The flexible model has much higher variance: its predictions swing by more than 100,000 dollars depending on which houses it saw. Its average happens to be close to 280,000 dollars, so its bias at this point is small, but its variance makes individual predictions unreliable. Across many new houses, that instability produces the poor validation RMSE.
One train-validation split estimates generalization error; it does not directly reveal bias and variance separately. To study variance, retrain on different resamples or cross-validation folds and observe how much predictions or metrics change.
How I would use the idea in practice
I would first compare training and validation performance.
- If both errors are high and close together, I would suspect high bias. A richer model, better features, or less regularization may help.
- If training error is low but validation error is much higher, I would suspect high variance. More data, stronger regularization, a simpler model, or an ensemble may help.
- If both offline errors are good but production performance collapses, I would investigate leakage, distribution shift, or a validation split that does not resemble deployment. Calling that “overfitting” and immediately adding a penalty is often just debugging by superstition.
Regularization makes the tradeoff explicit. For example, L2 regularization adds a penalty proportional to the squared size of the model’s weights. The parameter lambda controls the penalty strength. A larger lambda discourages extreme coefficients, which usually smooths the fitted relationship and reduces variance, at the cost of some bias.
More training data usually reduces variance because any one noisy observation has less influence on the fitted model. It does not fix bias caused by a fundamentally inadequate model. Giving a straight line ten times as many examples will estimate the wrong straight line more precisely if the real relationship is strongly curved.
Ensembling can also reduce variance. If several models make partly independent errors, averaging them cancels some of their sample-specific mistakes. This is why bagging methods such as random forests can be more stable than a single deep decision tree. Averaging several copies of the same flawed model does not remove systematic bias, however.
In production, I would select complexity using validation or cross-validation, not training error. The split must match the prediction task: a time-based split for future forecasting, or a group-based split when the same customer, patient, or property could otherwise appear on both sides. After choosing the model and hyperparameters, I would evaluate once on an untouched test set.
The senior-level nuance
The clean bias-variance equation is exact for expected squared-error regression. Classification with zero-one loss does not have an equally simple decomposition, so interview answers should present the same intuition without pretending the formula applies unchanged to every metric.
Also, “more complex means worse generalization” is an oversimplification for modern overparameterized neural networks. Some models can fit the training set perfectly and still generalize well because of the data structure, optimizer, architecture, and implicit regularization. In some settings, test error follows a double-descent pattern: it rises near the interpolation threshold and falls again after the model becomes even more overparameterized.
That caveat does not make validation optional. It means complexity must be measured empirically. The useful question is not “Is this model large?” but “How does its performance change on data that imitates what it will see after deployment?”
What they will ask next
How do you tell underfitting from overfitting?
Compare training and validation error. High and similar errors suggest underfitting or high bias. Very low training error paired with much worse validation error suggests overfitting or high variance. Learning curves and cross-validation make the diagnosis less dependent on one lucky split.
Does regularization always improve a model?
No. It helps when variance is the problem by shrinking unstable parameters, but it can worsen an already underfit model by adding more bias. The strength must be selected using validation data.
Does adding more data always solve overfitting?
No. More representative data usually reduces variance, but it cannot repair missing features, a bad model class, label noise, or leakage. If the model is biased because it cannot express the real pattern, more examples may only make that wrong assumption more precise.
Say this in the interview
“Bias is error from a model being too simple, variance is sensitivity to the particular training sample, and overfitting is usually low-bias, high-variance behaviour where training error is low but validation error rises; I choose complexity by trading those errors on data that matches deployment.”