Logistic regression

The name is a trap. Logistic regression is a classifier, not a regressor. It’s the model you reach for first on any binary classification problem in tabular data — churn, conversion, fraud, default risk — for the same reasons you reach for linear regression on continuous problems: fast, interpretable, well-calibrated, and almost always within a few percentage points of fancier methods.

What the model does

Linear regression outputs Xw + b, which can be any real number. We need something between 0 and 1 to interpret as a probability. That’s the sigmoid function:

σ(z) = 1 / (1 + e^(−z))

It squashes any real number into (0, 1). So logistic regression is:

p(y=1 | x) = σ(Xw + b)

Why not just minimise MSE on the 0/1 labels? Because MSE treats probability outputs as if they live on a number line — it would pull predicted probabilities toward 0 and 1 mechanically, ignoring the asymmetry of wrong confidence. Log-loss (a.k.a. binary cross-entropy) is the correct loss when outputs are probabilities: it penalises confident wrong predictions exponentially, which is exactly the right incentive.

loss = −(1/n) Σ [ y log p̂ + (1−y) log(1−p̂) ]

The model heavily penalises confident wrong predictions. A model that predicts p=0.99 for a churner who didn’t churn pays a huge cost; a hedged 0.55 prediction pays much less. This is what makes the probabilities meaningful.

Geometrically, Xw + b = 0 is a straight line (the decision boundary) and the sigmoid turns distance from that line into a probability: deep on one side → confident, right on the line → 50/50. The S-curve below is that mapping — z = Xw + b runs along the x-axis, the predicted probability up the y-axis:

Predicting churn — the canonical setup

You’re on a SaaS retention team. Given a customer’s usage signals, predict whether they’ll churn in the next 30 days.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Simulated churn data — usage features and binary churn label
rng = np.random.default_rng(0)
n = 400
df = pd.DataFrame({
    "logins_per_week": rng.integers(0, 30, n),
    "days_since_last_login": rng.integers(0, 60, n),
    "support_tickets": rng.integers(0, 8, n),
    "monthly_spend":   rng.gamma(2, 50, n).round(2),
})
# True signal: low logins + high days_since + high tickets = churn
score = (-0.1 * df["logins_per_week"] + 0.05 * df["days_since_last_login"]
         + 0.4 * df["support_tickets"] - 0.005 * df["monthly_spend"] + rng.normal(scale=0.5, size=n))
df["churned"] = (score > 0.5).astype(int)

X, y = df.drop(columns=["churned"]), df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

pipe = Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression())]).fit(X_train, y_train)

print("test accuracy:", round(pipe.score(X_test, y_test), 3))
print("\ncoefficients (on standardised features):")
for name, w in zip(X.columns, pipe.named_steps["clf"].coef_[0]):
    print(f"  {name:>22}: {w:+.3f}")

Positive coefficient → that feature pushes probability of churn up. Negative coefficient → pushes it down. Magnitude (on standardised features) tells you relative importance. Read those numbers out loud and you’ve explained the model to a non-technical PM in 30 seconds.

.predict vs .predict_proba

.predict() returns the class label (0 or 1). It uses an internal threshold of 0.5 — predict 1 if p̂ ≥ 0.5, else 0. That threshold is almost never the right one in business contexts. You usually want the probability and then a threshold tuned to your decision.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
n = 200
X = rng.normal(size=(n, 3))
y = (X[:, 0] - X[:, 1] + 0.3 * rng.normal(size=n) > 0).astype(int)

pipe = Pipeline([("s", StandardScaler()), ("m", LogisticRegression())]).fit(X, y)
probs = pipe.predict_proba(X[:5])
preds_default = pipe.predict(X[:5])
preds_high_recall = (probs[:, 1] > 0.3).astype(int)   # lower threshold = catch more positives

print("probabilities (P[class=0], P[class=1]):")
print(probs.round(3))
print("default-threshold predictions:", preds_default)
print("threshold 0.3 (higher recall):", preds_high_recall)

If a missed churner costs you $200 and a wrongly-flagged loyal customer costs you a $10 promo, you should lower the threshold. If a flagged fraud means freezing a customer’s account, you should raise it. Picking the threshold is a business decision; the model gives you the probability.

Handling class imbalance

Real classification problems are imbalanced — 3% fraud, 5% churn, 0.5% click-through. By default, logistic regression’s loss treats every row equally, so it can drift toward “always predict no”. The simplest fix is class_weight="balanced", which re-weights the loss inversely to class frequencies.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
n = 1000
X = rng.normal(size=(n, 3))
y = (X[:, 0] + 0.2 * rng.normal(size=n) > 2.0).astype(int)   # ~2.3% positive
print("positive rate:", y.mean().round(3))

X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)

plain    = LogisticRegression().fit(X_tr, y_tr)
weighted = LogisticRegression(class_weight="balanced").fit(X_tr, y_tr)

from sklearn.metrics import recall_score, precision_score
for name, m in [("plain", plain), ("balanced", weighted)]:
    p = m.predict(X_te)
    print(f"{name:>10} — precision: {precision_score(y_te, p):.2f},"
          f" recall: {recall_score(y_te, p):.2f}")

class_weight="balanced" typically trades precision for recall — exactly the trade-off you want on imbalanced problems where missing a positive is expensive.

Multi-class — OvR and softmax

Logistic regression generalizes to multi-class in two ways:

• One-vs-Rest (OvR) — fit one binary classifier per class (“is this iris versicolor or not?”). Default in older sklearn versions.
• Multinomial (softmax) — fit all classes jointly with a softmax output. Better-calibrated probabilities. Default in modern sklearn.

LogisticRegression()   # multinomial is the default; multi_class= was removed in sklearn 1.5

For most tabular problems with more than two classes, leave it on default and move on.

Probability calibration

A pleasant property of logistic regression: its outputs are usually well-calibrated out of the box. If the model says p̂ = 0.7 for a batch of customers, roughly 70% of them really do churn. That’s not true of trees, random forests, or boosting — those need a post-hoc calibration pass (CalibratedClassifierCV) before their probabilities mean anything.

If your business uses the probability directly (expected value calculations, threshold tuning, ranking), calibration matters more than raw accuracy. This is a major reason logistic regression stays in production even when fancier models exist.

In one breath

• Despite the name, logistic regression is a classifier: it puts z = Xw + b through the sigmoid σ(z) = 1/(1+e^(−z)) to get p(y=1), with z = 0 the decision boundary (p = 0.5).
• It trains with log-loss (cross-entropy), which punishes confident-wrong predictions hard — what makes the probabilities meaningful and well-calibrated out of the box (unlike trees).
• Coefficients on standardized features read directly: sign = direction, magnitude = importance — the interpretability that keeps it deployed in regulated fintech/health/marketing.
• .predict uses a 0.5 threshold that’s rarely the right business threshold; prefer .predict_proba and tune the threshold to your costs.
• On imbalanced data, accuracy lies (predict-all-0 scores high) — use precision/recall and class_weight="balanced"; multi-class defaults to softmax (multinomial).

Quick check