Ridge Regression & Regularization

Last lesson closed on an uncomfortable truth: our optimizers are too good. Both the normal equation and gradient descent drive the training loss to its floor — and at that floor, with many features or noisy data, sit huge, jittery weights that fit the noise and swing wildly on anything new. That is overfitting, and you cannot optimize your way out of it, because the trap is the very bottom of the bowl you are descending.

So we change the bowl. Instead of asking the model only to fit the data, we add a second clause to its objective: and keep your weights small. The model now has to balance two pulls — match the data, yet stay modest — and the modesty clause tugs every weight toward zero, smoothing the fit. That added clause is regularization, and its most common form is ridge regression.

The L2 penalty

Ridge minimises the usual squared error plus a penalty on the size of the weight vector, scaled by a knob λ (lambda):

• The first term, Σ(yᵢ − ŷᵢ)², is ordinary least squares — make the predictions match.
• ‖w‖² = Σ wⱼ² is the L2 penalty: the sum of the squares of the weights.
• λ ≥ 0 controls the trade-off. At λ = 0 you recover plain least squares; as λ grows, the penalty dominates and every weight is pulled toward zero (though, for ridge, never exactly to zero).

One naming point GATE leans on hard: ridge uses the L2 norm; lasso uses the L1 norm (Σ|wⱼ|, the sum of absolute values). Lasso can drive weights to exactly zero, performing feature selection; ridge only shrinks them. Do not swap the two.

Slide λ up and watch every coefficient shrink. Toggle between L2 (ridge — a smooth shrink, nothing reaching zero) and L1 (lasso — small weights snapping to zero):

How GATE asks this

Two recurring shapes. As an MCQ, it asks the direction of the effect: what happens to bias and variance as you raise λ. GATE DA 2026 posed exactly this (Q37), and the correct statement was that the regularizer increases bias and decreases variance. As a NAT, it gives you weights, a data-fit error, and λ, and asks for the total regularized objective — pure substitution (GATE DA 2026 Q55 did this with an MAE data term).

Worked example

Part A — the direction (GATE DA 2026). As the regularization coefficient λ rises, the weights are squeezed toward zero, so the model becomes simpler and stiffer. A stiffer model varies less from one training set to the next (variance decreases), but systematically misses more structure (bias increases). The 2026 answer was exactly that pair: bias ↑, variance ↓.

Part B — a regularized loss (clean numbers). Take the ridge objective J = (squared error on the data) + λ · ‖w‖² with weights w = (3, −1), a data-fit squared error of 4, and λ = 0.5:

‖w‖²    = 3² + (−1)²        = 9 + 1   = 10
penalty = λ · ‖w‖² = 0.5 · 10         = 5
J       = data error + penalty = 4 + 5 = 9

The same steps in Python:

w          = [3, -1]
data_error = 4
lam        = 0.5

l2_norm_sq = sum(wi**2 for wi in w)   # 9 + 1 = 10
penalty    = lam * l2_norm_sq          # 0.5 * 10 = 5
J          = data_error + penalty      # 4 + 5 = 9

print(f"||w||^2 = {l2_norm_sq}")
print(f"penalty = {penalty}")
print(f"J       = {J}")

||w||^2 = 10
penalty = 5.0
J       = 9.0

So the total regularized objective is J = 9. Notice the penalty (5) is larger than the data error (4): with λ = 0.5 and weights this size, the model really is under pressure to shrink them.

In one breath

Ordinary least squares can overfit by driving the weights huge to chase noise, so ridge regression adds an L2 penalty to the objective — minimise Σ(yᵢ − ŷᵢ)² + λ·‖w‖², where ‖w‖² = Σwⱼ² and λ ≥ 0 sets how hard the weights are pulled toward zero (λ = 0 is plain OLS); raising λ shrinks the weights, lowering variance at the cost of higher bias, and the one naming trap is that ridge is L2 while lasso is L1 (Σ|wⱼ|), the version that can zero weights out entirely.

Practice

A question to carry forward

Ridge gave us one precise dial: turn λ up, and variance falls while bias rises. We have been saying those two words as if their meaning were settled — but they are the deepest pair in all of modelling, and ridge only nudges one instance of a tug that every model feels.

Why can you never just drive both to zero? A model simple enough to be steady across datasets (low variance) is too rigid to capture the truth (high bias); a model flexible enough to capture every wrinkle (low bias) reshapes itself wildly from one sample to the next (high variance). Here is the thread onward: what exactly are bias and variance, how do they split the total error of any model, and where does their sum bottom out — the sweet spot every technique in this chapter is secretly chasing?