Logistic Regression

Last lesson left every metric waiting for a model to grade — a model that emits a class, or better, a score you can threshold, the very thing the ROC curve sweeps. We have built six regressors, and not one of them does that; each predicts a bare number on the open line. So we need to take that number and bend it into a probability of being positive, a value safely between 0 and 1. The model that does exactly this is, despite its name, the first true classifier — and the name is the trap.

Logistic regression is a classifier. It earns “regression” only because, under the hood, it first computes a plain linear score z = wᵀx + b — the same weighted sum of features you have used since the very first regression lesson — and then bends that score into a probability. Two stages, then: a familiar linear score, followed by a squashing function that turns any real z into a probability in (0, 1). That squasher is the sigmoid, and it is the whole reason the linear machinery now works for classification. It remains the default first classifier in industry — fast, interpretable, and the baseline a neural network has to justify beating.

The sigmoid turns a score into a probability

The score z = wᵀx + b can be any real number, large positive or large negative. We need a probability in (0, 1). The sigmoid delivers exactly that:

Three facts GATE leans on, read straight off the curve:

• As z grows large and positive, e⁻ᶻ → 0, so σ(z) → 1.
• As z grows large and negative, e⁻ᶻ → ∞, so σ(z) → 0.
• At z = 0, e⁻ᶻ = 1, so σ(0) = 1 / (1 + 1) = 0.5 exactly.

The output σ(z) is read as P(y = 1 | x) — the model’s estimated probability that the point belongs to the positive class. That is precisely the score the ROC curve from last lesson was sweeping.

The decision boundary is z = 0

To turn a probability into a class, threshold at 0.5: predict positive when σ(z) ≥ 0.5, negative otherwise. But σ(z) = 0.5 happens exactly when z = 0, so the decision boundary is the set of points where wᵀx + b = 0. That is a straight line — a hyperplane in higher dimensions — so the boundary is linear in x, even though the sigmoid mapping itself is curved.

Drag the boundary below to separate the two classes by hand, then press Fit to watch the model find the separator for you:

The model is trained with log-loss (cross-entropy), −[y log p + (1 − y) log(1 − p)], not squared error. Log-loss punishes a confident wrong prediction — say p = 0.99 when the true label is 0 — far more harshly, which is exactly what you want from a probability model, and it keeps the optimisation convex and well-behaved.

How GATE asks this

Usually an MCQ probing one of three things: evaluate the sigmoid at a given score (often a NAT, with the relevant e value supplied), identify the decision boundary (the answer is the linear equation wᵀx + b = 0, never a curve), or name the loss (cross-entropy / log-loss, never mean squared error). A favourite distractor claims logistic regression outputs a continuous quantity like linear regression — it does not; it outputs a class probability.

Worked example — evaluate the sigmoid

A logistic model produces score z for a point. Find σ(z) for z = 0, z = 2, and z = −2. Use e⁻² ≈ 0.135. Which class is z = 2?

Apply σ(z) = 1 / (1 + e⁻ᶻ) term by term:

σ(0)  = 1 / (1 + e⁰)    = 1 / (1 + 1)     = 0.5      ← on the boundary
σ(2)  = 1 / (1 + e⁻²)   = 1 / (1 + 0.135) = 1/1.135  ≈ 0.881
σ(−2) = 1 / (1 + e²)    = 1 / (1 + 7.389) = 1/8.389  ≈ 0.119

A quick shortcut to check σ(−2): the sigmoid is symmetric, σ(−z) = 1 − σ(z), so σ(−2) = 1 − 0.881 = 0.119. ✓ — and indeed σ(2) + σ(−2) ≈ 1, just as the prediction prompt hinted.

Since σ(2) ≈ 0.881 > 0.5, the point with z = 2 is classified positive, with about 88% confidence. The point with z = −2 would be classified negative (only about a 12% chance of being positive).

In one breath

Logistic regression is a classifier that computes the familiar linear score z = wᵀx + b, then squashes it through the sigmoid σ(z) = 1/(1 + e⁻ᶻ) into a probability P(y=1|x) in (0, 1) — near 1 for big positive z, near 0 for big negative z, exactly 0.5 at z = 0; thresholding at 0.5 puts the decision boundary at wᵀx + b = 0, a straight line linear in x, and the model is trained by minimising log-loss (cross-entropy), not squared error, because log-loss harshly penalises confident wrong probabilities.

Practice

A question to carry forward

Logistic regression hands us a real classifier at last — but look at the shape of what it learned: a single straight line, fixed once, drawn from all the data at once. That global commitment is its strength (interpretable, fast) and its cage. Hand it two classes coiled around each other like a spiral, and no single straight cut can separate them.

So swing to the opposite extreme. What if a classifier learned nothing in advance — no weights, no boundary, no training at all — and instead, to label a brand-new point, simply looked at the handful of training points sitting closest to it and took a vote? Here is the thread onward: can “you are like your neighbours” be a whole classification algorithm, what exactly does closest mean, and what new dial — the number of neighbours you poll — quietly slides you right back along the bias-variance curve?