Why is linear regression unsuitable for binary classification, and what specific problems does logistic regression fix?

How to think about it

Linear regression is not incapable of separating two classes, but it is a poor default when the output must be a probability: its line can predict values below 0 or above 1, and ordinary least squares does not model the Bernoulli nature of a binary target. Logistic regression fixes the range and uses the probability model for a Bernoulli outcome, a single zero-or-one trial, by applying a sigmoid to a linear score and fitting it with the outcome likelihood.

The distinction the interviewer is testing

Binary classification means choosing between two labels, such as churn and no churn. Usually, we want both a class decision and an estimate of uncertainty. “This customer has a 0.82 probability of churning” is more useful than “the model gave this customer a score of 2.4.”

Ordinary least squares, or OLS, chooses coefficients by minimizing the sum of squared differences between its predictions and the labels. Its model is:

ŷ = Xβ

Here, X is the feature vector and β is the learned coefficient vector. There is no mathematical restriction on ŷ. It can be negative, 1.7, or 42.

For a binary target, the average value of y at a particular feature value is a probability. If 30 out of 100 similar customers churn, the conditional average is 0.30. OLS can estimate that average, and this is why a linear probability model is not nonsense. Under the right assumptions, it can even give useful coefficients.

The trouble is that OLS estimates the average with an unrestricted line. It does not know that the answer must stay between 0 and 1.

There is a second distinction. A model only needs a decision boundary, the surface separating predicted class zero from predicted class one, to make hard classifications. A linear regression model can create one by predicting class one whenever ŷ is at least 0.5. So it may classify reasonably while producing terrible “probabilities.”

That is the first senior-level nuance: linear regression is not automatically useless for classification. It is unsuitable mainly when you need valid, well-behaved probabilities or a statistically appropriate model for binary outcomes.

A concrete example

Imagine a subscription company predicting whether a customer will churn, using the number of support tickets opened in the last 30 days. Let y = 1 mean churn and y = 0 mean retained.

Suppose an OLS fit produces:

ŷ = 0.35 + 0.12(tickets - 4)

The predictions look like this:

Support tickets	Linear output	Interpretation as probability
0	-0.13	Impossible
4	0.35	Plausible
10	1.07	Impossible

At zero tickets, negative 13 percent churn is not a probability. At ten tickets, 107 percent churn is no more meaningful.

You could clip the outputs to the interval from 0 to 1. That hides the symptom, but it does not repair the model. The coefficients were still learned by allowing impossible values, and every prediction above 1 gets flattened to the same answer. A customer predicted at 1.07 and one predicted at 1.90 both become 1.00, even though the original model thought they were very different.

The model can still draw a boundary. In this example, setting ŷ to 0.5 gives a cutoff around 5.25 tickets. That might be adequate if the only question is “which side of the line is this customer on?” It is not adequate if the retention team uses predicted probabilities to prioritize outreach or estimate expected revenue.

Why the loss function matters

A common interview answer says that MSE is wrong because it penalizes confident correct predictions. That claim is inaccurate.

For a probability forecast on a zero-or-one target, mean squared error is the Brier score, which is a proper scoring rule. A proper scoring rule is one whose expected penalty is minimized by reporting the true probability. If the true churn probability is 0.8, a forecast of 0.8 is preferred in expectation to 0.6 or 0.99.

For one customer who actually churns, MSE gives:

• Prediction 0.99: (1 - 0.99)^2 = 0.0001
• Prediction 0.60: (1 - 0.60)^2 = 0.16

MSE correctly rewards the confident prediction because it was right.

The real difference is that logistic regression uses log loss, also called binary cross-entropy, which comes from the likelihood of Bernoulli outcomes. For one example, the loss is:

-[y log(p) + (1-y) log(1-p)]

For the same customer who churned:

• Prediction 0.99: approximately 0.010
• Prediction 0.60: approximately 0.511

Now consider a customer who did not churn. A confident wrong prediction of 0.99 gives:

• MSE: 0.9801
• Log loss: approximately 4.605

A less confident wrong prediction of 0.60 gives:

• MSE: 0.36
• Log loss: approximately 0.916

Log loss punishes confident mistakes much more aggressively. That is deliberate. A system that says “99 percent certain” and is wrong has supplied more damaging information than one that says “60 percent likely.”

So logistic regression is not fixing an “improper MSE” problem. It is using a likelihood that matches the binary data-generating assumption and gives a particularly strong penalty to overconfident errors.

Why the variance assumption matters

A binary target is a Bernoulli random variable. If its probability of being one is p, its conditional variance is:

p(1-p)

That variance is not constant. It is largest at p = 0.5, where it is 0.25, and smaller at p = 0.1, where it is 0.09.

OLS textbooks usually assume homoscedasticity, meaning the error variance is constant across observations. Binary data violate that assumption by construction. The outcome is more variable for uncertain customers and less variable for customers who are almost certainly going to churn or stay.

This does not mean every OLS coefficient becomes useless. With suitable assumptions, OLS can estimate the best linear approximation to the conditional probability. But ordinary textbook standard errors are not valid under heteroscedastic errors, and OLS does not use the available Bernoulli variance structure efficiently. Robust standard errors help with inference. They do not make the predictions valid probabilities.

Logistic regression models the binary outcome directly, so its likelihood incorporates the changing variance rather than pretending every observation has the same noise level.

What logistic regression changes

Logistic regression first calculates an unrestricted linear score:

z = Xβ

It then converts that score into a probability with the sigmoid function:

p = σ(z) = 1 / (1 + e^(-z))

The sigmoid maps every real-valued score into the open interval from 0 to 1. A score of zero becomes a probability of 0.5. Large positive scores approach 1, and large negative scores approach 0.

The underlying relationship is linear in log odds, where odds means the probability of the event divided by the probability of no event:

log(p / (1-p)) = Xβ

That is the key mechanism. Logistic regression does not force the probability itself to be a straight line. It forces the log odds to be a straight line, and the sigmoid converts that line into a curved probability scale.

The model then chooses coefficients by maximizing the Bernoulli likelihood, or equivalently minimizing log loss. This gives it:

1. Valid probability-shaped outputs.
2. A likelihood designed for zero-or-one outcomes.
3. A decision boundary that remains easy to interpret.

Because the usual logistic objective is convex, meaning it has no spurious local valleys, optimization is generally well behaved. But “convex” does not guarantee a unique finite solution. If one feature perfectly separates churners from non-churners, the likelihood can keep improving as a coefficient grows without bound. The practical symptom is often a convergence warning, enormous coefficients, and predicted probabilities pinned extremely close to zero or one. Regularization usually gives a finite solution, but the underlying separation should still be investigated.

How I would use it in practice

A basic implementation looks like this:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

p_churn = model.predict_proba(X_test)[:, 1]
predicted_churn = (p_churn >= 0.50).astype(int)

With labels zero and one, the second probability column is the model’s estimated probability of class one. Those estimates are not automatically calibrated. Calibration means that among customers receiving a predicted probability of 0.70, roughly 70 percent should actually churn over time. Check that on held-out data rather than assuming it from the method name.

The 0.50 cutoff is a decision policy, not a law of nature. It is sensible when the probabilities reflect the deployment population and false positives and false negatives have roughly equal cost. If missing a likely churner costs $200 while contacting a safe customer costs $5, the economically sensible threshold will usually be lower. Select it using business costs and validation data.

Logistic regression also does not solve nonlinear relationships. If churn rises for both very low and very high usage, a single linear term cannot represent that shape. Add transformations, interactions, or splines, or use a nonlinear model such as a tree ensemble. Logistic regression remains attractive when the relationship is reasonably linear in log odds, the dataset is small or medium-sized, and interpretability and probability estimates matter.

What they’ll ask next

Can linear regression ever be acceptable for classification?

Yes, as a quick baseline or when only a rough boundary is needed. It can work surprisingly well when classes are balanced, features are well behaved, and predictions mostly remain within the valid probability range. I would not use it as the default for calibrated probabilities, especially when downstream decisions depend on the difference between 0.60 and 0.90.

Why not just clip the linear predictions to zero and one?

Clipping is post-processing, not a fix to the training objective. OLS still learns using impossible values, and many distinct high scores collapse to the same clipped output. It also does not address Bernoulli variance or the stronger penalty that log loss gives to confident mistakes.

Does logistic regression guarantee calibrated probabilities, and should the threshold always be 0.5?

No to both. A misspecified feature relationship, regularization, class weighting, sampling changes, or deployment drift can make the probabilities overconfident. Validate calibration on data that resembles production, and choose the classification threshold from the relative costs of the errors.

Say this in the interview

“Linear regression can draw a classification boundary, but its unbounded outputs and constant-variance error model make it a poor probability model for binary outcomes; logistic regression fixes that with a sigmoid link and Bernoulli log-likelihood, while threshold and calibration still need to be handled separately.”