Regularization — Ridge, Lasso, Elastic Net

Linear regression has one job: drive the squared error on the training data as low as possible. Give it 200 features and 80 rows, and it will happily find weights that make the training error nearly zero — by fitting the noise. The test error will be a disaster.

The fix isn’t fewer features (you usually want all of them). The fix is to add a penalty to the loss that punishes large weights. That penalty is regularization, and it’s the difference between a linear model that ships to production and one that doesn’t.

The intuition

Ordinary least squares minimises:

MSE(w) = (1/n) Σ (y_i − Xw_i)²

Regularization adds a term:

Ridge:  MSE(w) + α · Σ w_j²        (L2 penalty)
Lasso:  MSE(w) + α · Σ |w_j|       (L1 penalty)
ElasticNet: combination of both

Big weights cost you more — so the optimizer prefers smaller, more “conservative” weights. That conservatism is what protects you against fitting noise.

See it on a noisy, many-feature dataset

Predicting house prices again, but with 30 engineered features (one real signal, 29 noise columns). Plain linear regression will memorize the noise; Ridge won’t.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
n, n_features = 80, 30

# Only the first feature is real signal — the rest are pure noise
X = rng.normal(size=(n, n_features))
y = 3 * X[:, 0] + rng.normal(scale=0.5, size=n)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

ols = Pipeline([("s", StandardScaler()), ("m", LinearRegression())]).fit(X_train, y_train)
rdg = Pipeline([("s", StandardScaler()), ("m", Ridge(alpha=10))]).fit(X_train, y_train)

print(f"plain  LinReg — train R²: {ols.score(X_train, y_train):.3f},"
      f"  test R²: {ols.score(X_test, y_test):.3f}")
print(f"Ridge α=10   — train R²: {rdg.score(X_train, y_train):.3f},"
      f"  test R²: {rdg.score(X_test, y_test):.3f}")

Plain OLS gets near-perfect training R² and pitiful test R². Ridge gives up some training fit and gets much higher test R². That gap is overfitting, and the penalty is what closed it.

Always scale before regularizing

The penalty Σ w_j² treats all weights equally. If one feature is in the range [0, 10⁶] and another is in [0, 1], the first will get a tiny weight and the second a huge one — purely because of units, not importance. The penalty then punishes the second feature unfairly.

The fix is StandardScaler before Ridge / Lasso. Roll it into a Pipeline (recommended) — RidgeCV does NOT scale automatically, it only adds alpha cross-validation. Forgetting to scale is the single most common regularization bug.

Lasso — same idea, sharper consequence

L2 (Ridge) shrinks weights smoothly toward zero. L1 (Lasso) shrinks them to exactly zero. The geometry of the |w| penalty has sharp corners, and the optimizer settles on those corners. Effect: Lasso both regularizes and selects features, in one pass.

import numpy as np
import pandas as pd
from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
n, n_features = 80, 30
X = rng.normal(size=(n, n_features))
y = 3 * X[:, 0] - 2 * X[:, 1] + rng.normal(scale=0.5, size=n)   # only 2 real features

pipe = Pipeline([("s", StandardScaler()), ("m", Lasso(alpha=0.1))]).fit(X, y)

coefs = pipe.named_steps["m"].coef_
print("non-zero coefficients (Lasso α=0.1):")
for i, c in enumerate(coefs):
    if abs(c) > 1e-6:
        print(f"  feature {i}: {c:+.3f}")
print(f"\ntotal features kept: {(np.abs(coefs) > 1e-6).sum()} / {n_features}")

Two real signals are recovered; most of the noise features are driven to exactly zero. That gives you a sparse, interpretable model — “here are the 4 features that mattered” — without running a separate feature selection pass.

Elastic Net — the practical default

Lasso has a quirk: when two features are highly correlated, it tends to arbitrarily pick one and zero out the other. Ridge keeps both but shrinks together. ElasticNet combines both penalties:

ElasticNet: MSE(w) + α · [ l1_ratio · Σ|w| + (1 − l1_ratio) · Σw² ]

l1_ratio=0 is Ridge, l1_ratio=1 is Lasso, 0.5 is the standard middle ground. In production tabular models, Elastic Net is often the linear baseline of choice because it inherits Lasso’s feature selection without its instability under correlated features.

Tuning alpha — never by hand

The right alpha depends on your data. Use the built-in CV variants — they sweep alpha over a grid and pick the best on cross-validated score:

import numpy as np
from sklearn.linear_model import RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 20))
y = X @ rng.normal(size=20) + rng.normal(scale=0.5, size=200)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

# Sweep a log-spaced grid of alphas, pick best by 5-fold CV
alphas = np.logspace(-3, 3, 13)
pipe = Pipeline([("s", StandardScaler()), ("m", RidgeCV(alphas=alphas, cv=5))]).fit(X_tr, y_tr)

print("chosen alpha:", pipe.named_steps["m"].alpha_)
print("test R²:", pipe.score(X_te, y_te).round(3))

RidgeCV and LassoCV are drop-in replacements for Ridge / Lasso that handle the tuning loop for you. Use them.

Quick reference

In one breath

• Linear models overfit when features outnumber signal — they fit noise (near-zero train error, terrible test); the fix is a penalty on large weights, not fewer features.
• Ridge (L2) adds α·Σw² and shrinks weights smoothly toward 0; Lasso (L1) adds α·Σ|w| and drives some weights exactly to 0 (its constraint has corners), doubling as feature selection.
• ElasticNet mixes both via l1_ratio — the practical default for correlated features (Lasso’s sparsity without its instability).
• alpha is the penalty strength (0 = OLS, large = everything → 0); never hand-tune — use RidgeCV / LassoCV.
• Always scale before regularizing — the penalty treats all weights equally, so raw units distort it; forgetting is the #1 regularization bug.

Quick check