Logistic Regression

Last lesson left every metric waiting for a model to grade. That model emits a class, or better, a score you can threshold — the very thing the ROC curve sweeps. We have built six regressors, and not one of them does that; each predicts a bare number on the open line.

So we need to take that number and bend it into a probability of being positive, a value safely between 0 and 1. The model that does exactly this is, despite its name, the first true classifier — and the name is the trap.

Logistic regression is a classifier. It earns “regression” only because, under the hood, it first computes a plain linear score z = wᵀx + b. That is the same weighted sum of features you have used since the very first regression lesson. It then bends that score into a probability.

Two stages, then: a familiar linear score, followed by a squashing function that turns any real z into a probability in (0, 1). That squasher is the sigmoid, and it is the whole reason the linear machinery now works for classification. It remains the default first classifier in industry — fast, interpretable, and the baseline a neural network has to justify beating.

The sigmoid turns a score into a probability

The score z = wᵀx + b can be any real number, large positive or large negative. We need a probability in (0, 1). The sigmoid delivers exactly that:

Three facts GATE leans on, read straight off the curve:

• As z grows large and positive, e⁻ᶻ → 0, so σ(z) → 1.
• As z grows large and negative, e⁻ᶻ → ∞, so σ(z) → 0.
• At z = 0, e⁻ᶻ = 1, so σ(0) = 1 / (1 + 1) = 0.5 exactly.

The output σ(z) is read as P(y = 1 | x) — the model’s estimated probability that the point belongs to the positive class. That is precisely the score the ROC curve from last lesson was sweeping.

The decision boundary is z = 0

To turn a probability into a class, threshold at 0.5. Predict positive when σ(z) ≥ 0.5, negative otherwise. But σ(z) = 0.5 happens exactly when z = 0, so the decision boundary is the set of points where wᵀx + b = 0.

That is a straight line — a hyperplane in higher dimensions — so the boundary is linear in x, even though the sigmoid mapping itself is curved.

This is the point most people have to read twice, so here it is plainly: the curve you are looking at is not the boundary. The sigmoid’s S-shape lives in a picture whose axes are the score z and the probability — it describes the squashing, nothing else.

The decision boundary lives in feature space, out among the actual data points, and there it is dead straight. Bending z through a curved function never bends the line wᵀx + b = 0; it only decides how fast confidence climbs as you walk away from that line.

Drag the boundary below to separate the two classes by hand, then press Fit to watch the model find the separator for you:

The model is trained with log-loss (also called cross-entropy). In words, log-loss asks one question of every training point: what probability did you assign to the thing that actually happened? It charges you −log of that number. A confident correct answer costs almost nothing, while a confident wrong one costs enormously.

Written out for a single point, that is −[y log p + (1 − y) log(1 − p)].

Notice it is not squared error, and the difference is deliberate. Log-loss punishes a confident wrong prediction — say p = 0.99 when the true label is 0 — far more harshly than squaring the gap would. That is exactly what you want from a probability model, and it keeps the optimisation convex and well-behaved.

How GATE asks this

Usually it is an MCQ probing one of three things:

• evaluate the sigmoid at a given score (often a NAT, with the relevant e value supplied);
• identify the decision boundary (the answer is the linear equation wᵀx + b = 0, never a curve);
• name the loss (cross-entropy / log-loss, never mean squared error).

A favourite distractor claims logistic regression outputs a continuous quantity like linear regression — it does not; it outputs a class probability.

Worked example — evaluate the sigmoid

A logistic model produces score z for a point. Find σ(z) for z = 0, z = 2, and z = −2. Use e⁻² ≈ 0.135. Which class is z = 2?

Apply σ(z) = 1 / (1 + e⁻ᶻ) term by term:

σ(0)  = 1 / (1 + e⁰)    = 1 / (1 + 1)     = 0.5      ← on the boundary
σ(2)  = 1 / (1 + e⁻²)   = 1 / (1 + 0.135) = 1/1.135  ≈ 0.881
σ(−2) = 1 / (1 + e²)    = 1 / (1 + 7.389) = 1/8.389  ≈ 0.119

A quick shortcut to check σ(−2): the sigmoid is symmetric, σ(−z) = 1 − σ(z), so σ(−2) = 1 − 0.881 = 0.119. ✓ — and indeed σ(2) + σ(−2) ≈ 1, just as the prediction prompt hinted.

Since σ(2) ≈ 0.881 > 0.5, the point with z = 2 is classified positive, with about 88% confidence. The point with z = −2 would be classified negative (only about a 12% chance of being positive).

In one breath

Logistic regression is a classifier that computes the familiar linear score z = wᵀx + b, then squashes it through the sigmoid σ(z) = 1/(1 + e⁻ᶻ) into a probability P(y=1|x) in (0, 1). That probability is near 1 for big positive z, near 0 for big negative z, and exactly 0.5 at z = 0.

Thresholding at 0.5 puts the decision boundary at wᵀx + b = 0, a straight line linear in x. The model is trained by minimising log-loss (cross-entropy), not squared error, because log-loss harshly penalises confident wrong probabilities.

Practice

A question to carry forward

Logistic regression hands us a real classifier at last — but look at the shape of what it learned: a single straight line, fixed once, drawn from all the data at once. That global commitment is its strength (interpretable, fast) and its cage. Hand it two classes coiled around each other like a spiral, and no single straight cut can separate them.

So swing to the opposite extreme. What if a classifier learned nothing in advance — no weights, no boundary, no training at all? Instead, to label a brand-new point, it simply looked at the handful of training points sitting closest to it and took a vote.

Here is the thread onward:

• can “you are like your neighbours” be a whole classification algorithm?
• what exactly does closest mean?
• what new dial — the number of neighbours you poll — quietly slides you right back along the bias-variance curve?