Simple Linear Regression

Last lesson ended on a sharp question: out of the infinitely many lines you could draw through a cloud of points, what makes one of them the best fit — and is there a formula that hands it to you? Here is the answer, and it starts with an instinct you already have.

Give a child a scatter of dots and a ruler, and they lay the ruler where it “looks right” — close to as many dots as it can manage. That instinct is the whole idea; we only have to make “looks right” precise. Once it is precise, the vague “close to most dots” becomes a single number to minimise, and minimising a single number is something a formula can do for us. It is also the model every data team reaches for first — the honest baseline a fancier model has to beat before anyone will trust it.

The model and the cost

A line carries the model y = mx + c, where m is the slope and c the intercept. Sometimes we force the line through the origin, y = wx, with a single weight w and no intercept at all. For each data point the line predicts ŷᵢ = m·xᵢ + c, and the gap between truth and prediction, yᵢ − ŷᵢ, is the residual.

Least squares pins down “looks right” by choosing the line that minimises the sum of squared residuals:

Squaring does two jobs at once. It makes every error positive, so gaps above and below the line cannot cancel out, and it punishes a big miss far more than a small one. In the picture below each shaded square is one residual squared — literally a square whose side is the gap. The least-squares line is the one that makes the total shaded area smallest. Drag the endpoints to try to beat it, then press Solve to snap to the optimum.

The normal equations

Setting the derivative of the cost with respect to each parameter to zero gives the normal equations — the conditions the best line must satisfy. For the through-origin model y = wx there is just one parameter, and the solution is beautifully compact:

For the full line y = mx + c the normal equations give two formulas, but GATE’s 2025 question used the through-origin form above — so that is the one to keep at your fingertips.

How GATE asks this

The bread-and-butter form is a NAT: you are handed 3 to 5 points and asked for the slope (or a prediction) of the least-squares fit. GATE DA 2025 asked exactly this — fit y = wx to three points and report w. The recipe never changes: form Σ xᵢ yᵢ and Σ xᵢ², then divide. The occasional MCQ probes the concept instead — what least squares minimises, or what an outlier does to the line.

Worked example — a real GATE DA 2025 NAT

Fit the model y = wx (through the origin) to the points (−1, 1), (2, −5), and (3, 5) by least squares. Find w. (Real GATE DA 2025 question.)

Build the two sums column by column, then divide:

point        x·y                 x²
(−1,  1)    (−1)(1)  = −1        (−1)² = 1
( 2, −5)    (2)(−5)  = −10        (2)² = 4
( 3,  5)    (3)(5)   =  15        (3)² = 9
            ─────────────        ──────────
   Σ x·y  = −1 − 10 + 15 = 4     Σ x² = 1 + 4 + 9 = 14

        Σ x·y       4
  w  =  ───────  =  ──  =  0.2857…  ≈  0.286
         Σ x²       14

The same arithmetic, written as a few lines of Python, lands on the same number:

xs = [-1, 2, 3]
ys = [ 1, -5, 5]

sxy = sum(x*y for x, y in zip(xs, ys))   # Σ x·y
sxx = sum(x*x for x in xs)               # Σ x²
w   = sxy / sxx

print(f"Σ x·y = {sxy}")
print(f"Σ x²  = {sxx}")
print(f"w     = {w:.4f}")

Σ x·y = 4
Σ x²  = 14
w     = 0.2857

So w ≈ 0.286. Notice the fit passes through none of the three points — it balances all of them so that the squared vertical gaps are collectively smallest, exactly as the drag-the-line widget hinted.

In one breath

Linear regression fits the line y = mx + c (or y = wx through the origin) by least squares — choosing the line that minimises the sum of squared vertical residuals Σ(yᵢ − ŷᵢ)², since squaring both kills sign-cancellation and punishes big misses hardest; setting the cost’s derivative to zero gives the normal equations, which for the through-origin model collapse to the one-line formula w = Σ xᵢyᵢ / Σ xᵢ², and the squaring is also exactly what makes the fit sensitive to outliers.

Practice

A question to carry forward

One feature, one slope, one tidy formula. But the house whose price we set out to predict never depended on size alone — it depends on location, on age, on the number of rooms, all at once. A single slope cannot carry all of that.

So the line has to grow: one weight per feature, the prediction becoming a weighted sum of many inputs. The hand-formula Σxy / Σx² will not stretch to cover it. Here is the thread onward: when a model has many features at once, is there still a single closed-form solution for the best weights — and what does the least-squares formula look like once the bookkeeping is done in matrices instead of scalars?