Linear regression

If you remember one model from this whole section, make it linear regression. It is fast, interpretable, and — when the relationship really is roughly linear — extremely hard to beat with anything fancier. Most serious ML teams keep a linear baseline next to every model they ship, specifically so they can answer the question “is the gradient boosting machine actually earning its complexity?”

The model in one line

You have features X (one row per sample, one column per feature) and a target y. The model is:

y_hat = X · w + b

w is a vector of weights (also called coefficients — one number per feature, telling the model how much that feature matters). b is the intercept (or “bias”). The model learns w and b by minimising the mean squared error:

MSE = (1/n) Σ (y_i − y_hat_i)²

Squaring the residuals means large errors are punished disproportionately — a prediction that’s off by 10 costs 100× more than one that’s off by 1. That’s usually what you want: big misses matter far more than small ones.

That’s it. The whole machine. There’s even a closed-form solution (w = (XᵀX)⁻¹Xᵀy) — no gradient descent required for the plain version.

The picture below is that whole idea in one variable. Each square is a single squared residual — its side is how far a point sits from the line — and the line that makes their total area as small as possible is the OLS fit. No other line beats it.

House prices — the canonical example

You’re on a real-estate pricing team. Given square footage and number of bedrooms, predict listing price.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Mini dataset — sqft, bedrooms, age, price (in $1000s)
df = pd.DataFrame({
    "sqft":     [800, 1200, 1500, 2000, 2400, 1100, 1800, 950, 2700, 1600],
    "bedrooms": [2,   3,    3,    4,    4,    2,    3,    2,   5,    3],
    "age":      [30,  10,   5,    2,    1,    25,   8,    40,  3,    12],
    "price":    [180, 285,  365,  470,  540,  240,  430,  205, 620,  380],
})

X = df[["sqft", "bedrooms", "age"]]
y = df["price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

model = LinearRegression().fit(X_train, y_train)

print("R² on test:", round(model.score(X_test, y_test), 3))
print()
print("intercept (b):", round(model.intercept_, 1))
for name, w in zip(X.columns, model.coef_):
    print(f"  weight for {name:>8}: {w:+.3f}")

R² on test: 0.966

intercept (b): 15.7
  weight for     sqft: +0.232
  weight for bedrooms: -0.670
  weight for      age: -0.776

model.coef_ is the vector w, model.intercept_ is b. Each coefficient is read as “holding everything else fixed, a one-unit increase in this feature shifts the predicted price by this many thousand dollars.”

That last sentence — “holding everything else fixed” — is the entire reason linear regression dominates in regulated industries. You can hand a credit risk model to a regulator and explain exactly how each input affects the output. Try doing that with a neural network.

Coefficients only mean something if the units make sense

Coefficients are scaled by the units of each feature. A sqft coefficient of 0.18 ($180 per square foot) is large because sqft itself is large; an age coefficient of -2.5 doesn’t mean age matters more — it means each year of age subtracts $2.5k from the predicted price.

If you want to compare which feature matters most, standardize first:

import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    "sqft":     [800, 1200, 1500, 2000, 2400, 1100, 1800, 950, 2700, 1600],
    "bedrooms": [2,   3,    3,    4,    4,    2,    3,    2,   5,    3],
    "age":      [30,  10,   5,    2,    1,    25,   8,    40,  3,    12],
    "price":    [180, 285,  365,  470,  540,  240,  430,  205, 620,  380],
})
X, y = df[["sqft", "bedrooms", "age"]], df["price"]

pipe = Pipeline([("scale", StandardScaler()), ("lr", LinearRegression())])
pipe.fit(X, y)

print("standardised coefficients (comparable across features):")
for name, w in zip(X.columns, pipe.named_steps["lr"].coef_):
    print(f"  {name:>8}: {w:+.2f}")

standardised coefficients (comparable across features):
      sqft: +124.07
  bedrooms: +2.20
       age: -14.96

Now the largest |coef| is the most influential feature. Without scaling you cannot make that comparison.

Does scaling change predictions?

For plain linear regression — no. The model can absorb any rescaling into the weights, and the predictions and R² are identical with or without StandardScaler. Scaling only changes the interpretability of the coefficients (and the speed of iterative solvers, which OLS doesn’t need).

That changes the moment you add regularization (Ridge, Lasso). There, scaling is mandatory — coming next lesson.

The assumptions, briefly

Statistics textbooks list four assumptions: linearity, independence, homoscedasticity (constant variance of errors), and normality of residuals. In an ML context where you mostly care about predictive accuracy on held-out data, you usually don’t care:

• Linearity — if the relationship really is curvy, your test R² will expose it. Add polynomial features or move to a tree.
• Independence — matters if you’re computing p-values or confidence intervals on the coefficients. For pure prediction, it mostly affects whether your CV scheme is honest.
• Homoscedasticity & normality — only matter if you want statistical inference. Predictions don’t care.

So: if you’re shipping a model, look at the test set. If you’re writing a paper with p-values, audit the assumptions properly.

In one breath

• The model is y_hat = X·w + b, fit by minimizing MSE (squared residuals punish big misses disproportionately); plain OLS has a closed-form solution — no gradient descent.
• model.coef_ (weights) and model.intercept_ read as “holding everything else fixed, a one-unit increase shifts the prediction by this much” — the interpretability that makes linear models dominate regulated industries.
• Coefficients are in each feature’s units, so standardize before comparing which feature matters most (largest |coef| wins).
• For plain linear regression, scaling does not change predictions or R² (the weights absorb it) — but it becomes mandatory once you regularize (Ridge/Lasso, next).
• Always fit a linear baseline beside any fancy model: if the baseline is nearly as good, the complexity isn’t earning its keep.

Quick check