How do L1 and L2 regularization affect bias and variance, and when would you pick one over the other?

How to think about it

The direct answer

L1 and L2 regularization both penalize large model coefficients, so they usually increase bias slightly while reducing variance and overfitting. I would choose L1, or lasso, when I need a sparse model with automatic feature selection; L2, or ridge, when I care more about stable predictions and have correlated features. If I need both properties, I would try elastic net.

Why regularization changes bias and variance

Regularization is a deliberate restriction on a model: instead of allowing the model to fit the training data with any coefficient values, we charge it for using large ones.

Bias is the error caused by a model being systematically too simple or too constrained. Variance is how much the fitted model changes when the training sample changes. An unregularized linear model can have low training error but high variance: remove a few observations, retrain it, and the coefficients move dramatically because the model was using noise as if it were signal.

The usual objective is:

training loss + lambda × penalty

Here, lambda controls the strength of regularization. For a coefficient vector beta, L1 uses the penalty sum(abs(beta_j)), while L2 uses sum(beta_j^2).

The square in L2 matters. A coefficient of 4 contributes 16 to the penalty, while a coefficient of 2 contributes 4. Very large coefficients become expensive quickly. L1 grows linearly: a coefficient of 4 contributes 4. L1 therefore has a stronger tendency to remove weak coefficients entirely, while L2 tends to keep every feature but make its effect smaller.

At lambda = 0, we recover the unregularized model. As lambda increases, the model becomes less flexible. Training error generally rises because the model is no longer allowed to chase every training observation. Validation error may fall at first because the model has stopped fitting noise, then rise again if the penalty becomes so strong that the model underfits. That is the bias–variance trade-off in operational form.

The intercept is normally excluded from the penalty. Otherwise, the model would be pushed toward predicting around zero even when the target’s baseline is not zero.

A concrete example

Suppose we predict monthly apartment rent from 10,000 listings using 200 standardized features: floor area, bedrooms, bathrooms, building age, neighborhood indicators, and interaction features. Several variables carry overlapping information. A large apartment is often more likely to have more bedrooms and bathrooms, so the model has multiple ways to explain the same part of the rent.

Imagine one train–validation experiment produces these illustrative results:

Model	Non-zero coefficients	Training RMSE	Validation RMSE
Ordinary least squares	200	$86	$210
Ridge	200, mostly small	$98	$158
Lasso	34	$104	$164

The unregularized model wins on training error but loses badly on unseen listings. It has enough freedom to memorize quirks of the training sample. Ridge gives up a little training performance and improves validation error by making the coefficients less sensitive to those quirks. Lasso removes 166 features and performs almost as well, which may be valuable if a property manager needs a short list of variables to inspect.

The exact numbers are not universal benchmarks. The reasoning is what matters: a model can become better on new data while becoming worse on its training data.

A small calculation shows why L1 creates zeros. Assume the features have been standardized and are mutually uncorrelated. Under the objectives 1/2 × squared error + lambda × L1 penalty and 1/2 × squared error + lambda/2 × L2 penalty, an unregularized coefficient z becomes:

• Lasso: sign(z) × max(abs(z) - lambda, 0)
• Ridge: z / (1 + lambda)

With lambda = 0.5, a weak unregularized coefficient of 0.4 becomes 0 under lasso, but about 0.267 under ridge. A stronger coefficient of 3 becomes 2.5 under lasso and 2 under ridge.

This calculation assumes a particular scaling of the loss and nicely behaved features. Libraries may define the objective with an average loss or expose inverse regularization strength, so the numeric value of lambda is not portable between implementations. The behavior is portable: L1 can threshold a coefficient to zero; L2 smoothly shrinks it.

The geometry explains the difference

There is also a useful geometric explanation. L1’s constraint region has corners on the coordinate axes. When the model searches for the best coefficients inside that region, the optimum often lands on a corner. A coefficient on an axis is exactly zero, which gives feature selection.

L2’s constraint region is smooth and round. Its boundary has no corners that favor an axis, so the optimum usually has many small, non-zero coefficients. L2 does not ask, “Which one feature should represent this signal?” It spreads the signal across features when that improves stability.

That is especially important with correlated variables. Suppose square footage, number of rooms, and number of bathrooms all predict rent. Lasso may keep square footage and discard the other two, or keep bathrooms and discard square footage. A small change in the sample can reverse that choice because several coefficient combinations produce nearly the same predictions. Ridge usually shares the weight across the correlated variables, making the prediction and coefficients less volatile.

How I would choose in practice

I would choose ridge when the main goal is prediction, the feature set contains correlated variables, and there is no strong reason to produce a sparse model. It is often a good default for dense numeric features, one-hot encodings, and situations where dropping a feature would be risky.

I would choose lasso when there are many plausibly irrelevant features and a compact model has practical value. For example, if a team needs to collect only 30 measurements at inference time instead of 2,000, zero coefficients are not merely attractive; they reduce operational cost.

I would not treat lasso’s selected features as automatically “the true causes.” With correlated predictors, selection can be unstable. With categorical variables represented by several indicator columns, lasso may keep one level and drop another in a way that is mathematically valid but awkward to explain. If the whole group should enter or leave together, a group-aware penalty may be more appropriate.

Elastic net combines both penalties. A common form is lambda × [alpha × L1 penalty + (1 - alpha) × L2 penalty]. With alpha = 1, it behaves like lasso; with alpha = 0, it behaves like ridge. Intermediate values encourage sparsity while also discouraging lasso from choosing arbitrarily among correlated features.

The most common implementation mistake is forgetting feature scaling. A penalty sees coefficient magnitudes, not the real-world units behind them. If income is measured in dollars and age in years, the same predictive effect may require a coefficient of 0.00002 for income and 0.8 for age. Without standardization, the penalty treats those coefficient values differently even if the underlying features matter equally. Fit the scaler inside each cross-validation training fold, not once on the full dataset, or the validation information leaks into the preprocessing.

I would tune lambda with cross-validation using the metric that matches the decision. Then I would evaluate the selected pipeline once on a held-out test set. Regularization does not repair data leakage, bad labels, or a target that is unavailable at prediction time. It only controls coefficient complexity.

There is also a case where regularization may be the wrong emphasis: a small, well-specified model where the goal is estimating coefficients and their uncertainty rather than maximizing prediction accuracy. Regularization deliberately biases coefficient estimates. That can be a good trade for prediction but a poor one if someone will interpret each coefficient as an unbiased scientific effect.

A failure mode to recognise

The first symptom of unstable lasso selection is often not a bad validation score. It is that different random seeds select different features while producing almost identical predictions.

That usually means the predictors contain redundant information. Lasso is making a choice among near-equivalent explanations, and the sample happens to decide which one survives. Check selection stability across resamples, inspect correlations, and consider ridge or elastic net if the features form meaningful groups.

What they’ll ask next

Does increasing lambda increase bias or variance?
Generally, it increases bias and reduces variance. Test error can improve while the penalty removes noise, but excessive regularization produces underfitting and raises test error again.

Why must features be standardized before lasso or ridge?
Because the penalty acts on coefficient size. Changing a feature from metres to centimetres changes its coefficient by a factor of 100 without changing its predictive information. Standardization makes the penalty comparable across features.

Does L1 always select the best features?
No. It selects a useful sparse solution, not a guaranteed list of causal or uniquely important variables. With correlated predictors, small data changes can change which feature receives the non-zero coefficient. Ridge or elastic net is safer when keeping correlated information matters.

Say this in the interview

“Both L1 and L2 trade a little bias for lower variance; I use lasso when I need sparse feature selection, ridge when correlated features make stability more important, and elastic net when I need both.”