Simple Linear Regression

Last lesson ended on a sharp question: out of the infinitely many lines you could draw through a cloud of points, what makes one of them the best fit — and is there a formula that hands it to you? Here is the answer, and it starts with an instinct you already have.

Give a child a scatter of dots and a ruler, and they lay the ruler where it “looks right” — close to as many dots as it can manage. That instinct is the whole idea; we only have to make “looks right” precise. Once it is precise, the vague “close to most dots” becomes a single number to minimise, and minimising a single number is something a formula can do for us. It is also the model every data team reaches for first — the honest baseline a fancier model has to beat before anyone will trust it.

The model and the cost

A line carries the model y = mx + c, where m is the slope — how much y moves per unit of x. The c is the intercept, the value y takes when x = 0. Sometimes we force the line through the origin, y = wx, with a single weight w and no intercept at all.

For each data point the line predicts ŷᵢ = m·xᵢ + c, and the gap between truth and prediction, yᵢ − ŷᵢ, is the residual.

Least squares pins down “looks right” by choosing the line that minimises the sum of squared residuals:

Squaring does two jobs at once. It makes every error positive, so gaps above and below the line cannot cancel out, and it punishes a big miss far more than a small one. In the picture below each shaded square is one residual squared — literally a square whose side is the gap. The least-squares line is the one that makes the total shaded area smallest. Drag the endpoints to try to beat it, then press Solve to snap to the optimum.

The normal equations

Setting the derivative of the cost with respect to each parameter to zero gives the normal equations — the conditions the best line must satisfy. For the through-origin model y = wx there is just one parameter, and the solution is beautifully compact:

For the full line y = mx + c the normal equations give two formulas, but GATE’s 2025 question used the through-origin form above — so that is the one to keep at your fingertips.

How GATE asks this

The bread-and-butter form is a NAT: you are handed 3 to 5 points and asked for the slope (or a prediction) of the least-squares fit. GATE DA 2025 asked exactly this — fit y = wx to three points and report w.

The recipe never changes: form Σ xᵢ yᵢ and Σ xᵢ², then divide. The occasional MCQ probes the concept instead — what least squares minimises, or what an outlier does to the line.

Worked example — a real GATE DA 2025 NAT

Fit the model y = wx (through the origin) to the points (−1, 1), (2, −5), and (3, 5) by least squares. Find w. (Real GATE DA 2025 question.)

Build the two sums column by column, then divide:

point        x·y                 x²
(−1,  1)    (−1)(1)  = −1        (−1)² = 1
( 2, −5)    (2)(−5)  = −10        (2)² = 4
( 3,  5)    (3)(5)   =  15        (3)² = 9
            ─────────────        ──────────
   Σ x·y  = −1 − 10 + 15 = 4     Σ x² = 1 + 4 + 9 = 14

        Σ x·y       4
  w  =  ───────  =  ──  =  0.2857…  ≈  0.286
         Σ x²       14

The same arithmetic, written as a few lines of Python, lands on the same number:

xs = [-1, 2, 3]
ys = [ 1, -5, 5]

sxy = sum(x*y for x, y in zip(xs, ys))   # Σ x·y
sxx = sum(x*x for x in xs)               # Σ x²
w   = sxy / sxx

print(f"Σ x·y = {sxy}")
print(f"Σ x²  = {sxx}")
print(f"w     = {w:.4f}")

Σ x·y = 4
Σ x²  = 14
w     = 0.2857

So w ≈ 0.286. Notice the fit passes through none of the three points — it balances all of them so that the squared vertical gaps are collectively smallest, exactly as the drag-the-line widget hinted.

There is a tempting shortcut here that is flatly wrong, and it is worth watching it fail on these very numbers. Each point suggests a slope of its own, yᵢ/xᵢ: 1/(−1) = −1, −5/2 = −2.5, and 5/3 ≈ 1.667. Average those three and you get ≈ −0.611 — a negative slope, nothing like the correct 0.286.

Least squares is not the plain average of the per-point slopes. It is the x²-weighted average of them, which is a different thing entirely: rewrite the formula as w = Σ xᵢ²·(yᵢ/xᵢ) / Σ xᵢ² and the weighting is explicit.

Here (3, 5) carries weight x² = 9 while (−1, 1) carries weight 1 — nine times the say. Points far from the origin dominate, because a wrong slope costs far more error out there. Averaging slopes gives every point an equal vote; least squares deliberately does not.

In one breath

Linear regression fits the line y = mx + c (or y = wx through the origin) by least squares — choosing the line that minimises the sum of squared vertical residuals Σ(yᵢ − ŷᵢ)².

Squaring does two things: it kills sign-cancellation and punishes big misses hardest. Setting the cost’s derivative to zero gives the normal equations.

For the through-origin model, they collapse to the one-line formula w = Σ xᵢyᵢ / Σ xᵢ². That same squaring is also exactly what makes the fit sensitive to outliers.

Practice

A question to carry forward

One feature, one slope, one tidy formula. But the house whose price we set out to predict never depended on size alone. It depends on three things at once:

• location
• age
• the number of rooms

A single slope cannot carry all of that.

So the line has to grow: one weight per feature, the prediction becoming a weighted sum of many inputs. The hand-formula Σxy / Σx² will not stretch to cover it.

Here is the thread onward: when a model has many features at once, is there still a single closed-form solution for the best weights? And what does the least-squares formula look like once the bookkeeping is done in matrices instead of scalars?