Why use cross-entropy loss instead of MSE for classification?

How to think about it

Use cross-entropy as the default classification loss because it matches the probabilistic output of a classifier and gives useful gradients when the model is confidently wrong. MSE is not mathematically invalid for classification, but its gradients often become tiny exactly when the model most needs a strong correction.

The failure it prevents

Imagine a fraud classifier looking at a fraudulent transaction. The true label is y = 1, but the model assigns only p = 0.01 probability to fraud. It is not merely wrong. It is confidently wrong.

A useful loss should make that example expensive and push the model hard in the right direction. MSE does the first part weakly and the second part poorly when its output uses a sigmoid.

For binary classification, the model usually produces a raw score called a logit, which is an unrestricted number such as -4.595. A sigmoid converts that score into a probability:

p = sigmoid(z)

Here, z is the logit and p is between zero and one. The value z = -4.595 produces approximately p = 0.01.

If we use half the squared error,

L_MSE = 1/2 (p - y)^2

the derivative with respect to the logit is:

dL/dz = (p - y) p(1 - p)

The factor p(1 - p) comes from the sigmoid’s derivative. It is largest around p = 0.5 and approaches zero near p = 0 or p = 1.

For the fraud example:

• p = 0.01
• y = 1
• MSE is 1/2 × (0.01 - 1)^2 = 0.49005
• The MSE gradient is approximately -0.009801

The probability error looks large, at -0.99, but the sigmoid factor shrinks it by roughly one hundred times. Gradient descent sees only a small instruction to increase the logit.

Binary cross-entropy is:

L_CE = -[y log(p) + (1 - y) log(1 - p)]

For the same example, the loss is -log(0.01), or approximately 4.605. More importantly, after applying the chain rule, its derivative with respect to the logit simplifies to:

dL/dz = p - y

That gives a gradient of -0.99, with no extra p(1 - p) factor. The model receives a strong correction because its prediction is far from the label.

This is the central reason. Cross-entropy does not somehow make the sigmoid itself non-saturating. The logarithmic loss cancels the sigmoid derivative that would otherwise weaken the error signal.

The loss value and the gradient answer different questions. MSE’s value of 0.49005 may look substantial, but the optimizer updates parameters using the gradient, not the loss value alone.

What cross-entropy is measuring

Cross-entropy is the negative log-likelihood, meaning it measures how unlikely the observed label was under the model’s predicted distribution.

For a positive example:

• Predicting p = 0.9 gives a loss of about 0.105
• Predicting p = 0.5 gives a loss of about 0.693
• Predicting p = 0.01 gives a loss of about 4.605
• Predicting p = 0.000001 gives a loss of about 13.816

The punishment grows sharply when the model assigns tiny probability to what actually happened. That is useful during training: a classifier should not be rewarded for being confidently wrong just because its probability error is bounded.

MSE on probabilities is bounded. With the half-squared convention, a binary error cannot exceed 0.5. Cross-entropy has no such upper bound. As the predicted probability of the true class approaches zero, its loss approaches infinity.

There is also a statistical reason for the choice. Cross-entropy is the natural maximum-likelihood objective for a Bernoulli model in binary classification and a categorical model in multiclass classification. If the data-generating process says that examples with a particular feature pattern are positive 70 percent of the time, the expected cross-entropy is minimized by predicting p = 0.7.

MSE on probabilities also has that ideal minimum. Calling MSE “wrong” would therefore be too strong. For one-hot classification probabilities, MSE is often called the Brier score, a valid probabilistic scoring rule. The practical difference is usually the shape of the optimization problem and how harshly the loss treats confident mistakes.

The multiclass case

For a problem with three mutually exclusive classes, the model produces three logits, such as:

z = [2.5, -1.0, 0.3]

A softmax converts them into probabilities that add up to one. If the correct class is c, multiclass cross-entropy is simply:

L = -log(p_c)

Its derivative for logit z_k is:

dL/dz_k = p_k - 1[k = c]

Here, 1[k = c] is one for the correct class and zero for every other class. In vector form, the gradient is predicted probabilities minus the one-hot target.

The same pattern appears: the correct class gets a negative correction when its probability is too low, while incorrect classes get positive corrections in proportion to the probability they wrongly received.

MSE can also be applied to one-hot vectors, but the softmax Jacobian introduces the same kind of saturation and coupling. Cross-entropy is the loss designed to work with categorical probabilities and softmax logits.

A checkable PyTorch example

This compares both losses for the confidently wrong fraud prediction:

import torch
import torch.nn.functional as F

z = torch.tensor([[-4.59512]], requires_grad=True)
target = torch.ones_like(z)

p = torch.sigmoid(z)
mse = 0.5 * (p - target).square().mean()
mse.backward()
mse_grad = z.grad.item()

z.grad.zero_()

ce = F.binary_cross_entropy_with_logits(z, target)
ce.backward()
ce_grad = z.grad.item()

print(
    f"p={p.item():.3f}, "
    f"MSE={mse.item():.5f}, "
    f"|grad|={abs(mse_grad):.6f}"
)
print(f"CE={ce.item():.3f}, |grad|={abs(ce_grad):.3f}")

To the displayed precision, it prints:

p=0.010, MSE=0.49005, |grad|=0.009801
CE=4.605, |grad|=0.990

In production, use the fused logits-aware losses:

• Binary classification: one raw logit with BCEWithLogitsLoss
• Multiclass classification: one raw logit per class with CrossEntropyLoss
• Multi-label classification: one independent raw logit per label with BCEWithLogitsLoss

“Multiclass” means exactly one class is correct, such as cat, dog, or horse. “Multi-label” means several labels may be correct at once, such as an image tagged both beach and sunset.

Warning. Do not apply sigmoid before BCEWithLogitsLoss, or softmax before CrossEntropyLoss. These losses apply the needed transformation internally, using numerically stable operations such as log-sum-exp. Applying the activation first can distort the gradients and create underflow or overflow problems. Apply sigmoid or softmax when you need probabilities for reporting or inference.

A common symptom is a loss that decreases very slowly while the model remains oddly under-confident or stops improving on hard examples. Another common bug is passing floating-point class indices to the usual multiclass form of CrossEntropyLoss; PyTorch may report an error such as expected scalar type Long but found Float. Use integer class indices for that form, not a manually softmaxed output.

When MSE is still reasonable

Use MSE for regression, where the target is genuinely continuous, such as predicting house price or temperature. The squared distance directly represents the error you care about there.

MSE can also be a defensible classification choice when you deliberately want the Brier score’s bounded penalty. It may be adequate when predictions stay away from sigmoid extremes, the dataset is small, or probability error matters more symmetrically than likelihood. It is not automatically a disaster.

There is a trade-off in the other direction. Cross-entropy strongly punishes mislabeled examples. If a bad data label says a transaction is fraudulent while the model is almost certain it is legitimate, cross-entropy can produce a very large gradient and make the model chase the bad label. MSE limits that individual example’s influence. Whether that is desirable depends on label quality and the cost of fitting noise.

Class imbalance is a separate issue. Weighted cross-entropy, positive-class weights, resampling, and a decision threshold can address it, but they do different things. In particular, class weighting changes the training objective and can make the resulting probabilities poorly calibrated. A model can rank examples well while its reported probabilities are wrong.

What they will ask next

Does cross-entropy always produce a large gradient?

No. For a correct, confident positive prediction such as p = 0.999 and y = 1, the gradient p - y is about -0.001. That small gradient is desirable because the example already needs little correction. The advantage is relative: for a confidently wrong prediction, cross-entropy avoids MSE’s additional saturation factor.

Why pass logits instead of probabilities to the PyTorch loss?

The fused loss combines the activation and logarithm in a numerically stable way. Explicitly computing a softmax can round a very small probability to zero; taking its logarithm then creates an infinite or unusable loss. CrossEntropyLoss expects raw multiclass logits, and BCEWithLogitsLoss expects raw binary or multi-label logits.

Can I use MSE if my labels are soft, such as 0.7?

Yes, but cross-entropy is usually still the better default when the target represents a probability distribution. Both can learn the average conditional probability in an ideal setting. Choose based on the desired penalty, robustness to noisy labels, calibration behaviour, and training dynamics rather than claiming that one is mathematically legal and the other is not.

Say this in the interview

“Cross-entropy is preferred because it is the likelihood-based loss for classification and gives a logit gradient of p - y; MSE adds the sigmoid or softmax saturation factor, so a confidently wrong prediction can receive an unnecessarily tiny update.”