What loss function does logistic regression optimize, and why is it convex?

How to think about it

Logistic regression optimizes binary cross-entropy, also called log-loss: the negative log-likelihood of the observed binary labels under Bernoulli probabilities. It is convex in the model coefficients because its Hessian has the form XᵀSX, where S has non-negative diagonal entries, so every local minimum is global. The careful answer is that convex does not always mean there is a unique finite minimum.

Why that loss appears

Suppose I am predicting whether a customer will churn. For each customer, the label y_i is either 0 or 1. Logistic regression first computes a linear score, called the logit, with z_i = x_iᵀβ, then converts that score into a probability with the sigmoid function:

p_i = σ(z_i) = 1 / (1 + exp(-z_i))

Here, x_i is the feature vector and β is the vector of coefficients. The sigmoid maps any real-valued score to a number between 0 and 1, which we interpret as the probability that the label is 1.

A binary label follows a Bernoulli distribution, which is the probability model for one trial with two possible outcomes. The probability assigned to the observed label is:

P(y_i | x_i, β) = p_i^{y_i}(1-p_i)^{1-y_i}

If the observed label is 1, this becomes p_i. If the observed label is 0, it becomes 1-p_i.

Maximum likelihood means choosing β so that the labels we actually observed become as probable as possible. Assuming rows are independent, the likelihood is the product of those per-row probabilities. Taking a logarithm turns the product into a sum, which is easier to optimize. Negating it turns maximization into minimization:

J(β) = -(1/n) Σ [y_i log(p_i) + (1-y_i) log(1-p_i)]

That objective is binary cross-entropy. Some libraries use the sum rather than the mean. The two objectives have the same minimizer when there is no regularization; the mean simply keeps the loss and gradient scale independent of the number of rows.

The logarithm creates the penalty that logistic regression needs. If the true label is 1:

Predicted probability	Loss
0.99	0.010
0.50	0.693
0.01	4.605

A confidently wrong prediction is punished much more heavily than an uncertain one. Predicting 0.01 for a true positive is not merely “a little wrong”; the model has assigned almost no probability to reality.

The convexity argument

A function is convex if the line between any two points on its surface never falls below the surface. For an optimization problem, that means there are no bad local valleys: if a finite point has zero gradient and is a valid minimum, it is globally optimal.

For one training row, the loss as a function of its logit is:

ℓ(z) = -[y log(σ(z)) + (1-y) log(1-σ(z))]

The useful cancellation happens when differentiating. The derivative of the sigmoid is σ(z)(1-σ(z)), and after applying the chain rule, the derivative of the loss with respect to the logit becomes:

dℓ/dz = p-y

That gives the gradient with respect to all coefficients:

∇J(β) = (1/n) Xᵀ(p-y)

In words, logistic regression takes each prediction error, p-y, and sends it backward through the feature matrix. This resembles the ordinary least-squares gradient, but the prediction error here comes from a probability produced by the sigmoid.

Differentiate once more. The second derivative of the loss with respect to one logit is:

d²ℓ/dz² = p(1-p)

For finite coefficients, p lies strictly between 0 and 1, so p(1-p) is positive. Put those values on the diagonal of a matrix S. The Hessian, meaning the matrix of second derivatives with respect to β, is:

H = (1/n) XᵀSX

To see why this is positive semi-definite, take any direction v in coefficient space:

vᵀHv = (1/n) Σ p_i(1-p_i)(x_iᵀv)²

Every term is non-negative. A squared quantity cannot be negative, and p_i(1-p_i) is non-negative. Therefore the curvature is never negative in any direction. That is the mechanical reason the objective is convex.

Warning — the common misconception: convex does not automatically mean “one unique global minimum.” If the columns of X are linearly dependent, several coefficient vectors can produce the same predictions. The Hessian can then be positive semi-definite rather than positive definite, leaving flat directions.

There is another complication: the objective may approach its best value without reaching it at any finite coefficient vector. That is the separation problem, and it matters in real training runs.

A concrete calculation

Consider four customers and one feature. The first column of X is the intercept.

import numpy as np

X = np.array([
    [1.0, -2.0],
    [1.0,  0.0],
    [1.0,  1.0],
    [1.0,  3.0],
])
y = np.array([0.0, 1.0, 0.0, 1.0])
beta = np.array([0.0, 0.0])

z = X @ beta
p = 1.0 / (1.0 + np.exp(-z))

loss = np.mean(np.logaddexp(0.0, z) - y * z)
gradient = X.T @ (p - y) / len(y)

print(round(loss, 6))
print(gradient.round(6))

The output is:

0.693147
[ 0.  -0.5]

At β = [0, 0], every customer receives probability 0.5, so the average loss is -log(0.5), or approximately 0.693147.

The intercept gradient is zero because the model has two positive and two negative labels. The feature gradient is -0.5, so increasing the feature coefficient reduces the loss at this point. With a learning rate of 0.1, one gradient-descent update gives:

β_new = β - 0.1 ∇J = [0, 0.05]

The code uses np.logaddexp for a numerically stable version of the loss. Computing log(sigmoid(z)) directly can underflow when z is very negative. In production code, stable log-loss implementations matter more than the formula’s innocent appearance suggests.

What the convexity does and does not buy you

Unlike ordinary least squares, logistic regression has no general closed-form normal-equation solution. The stationarity condition is:

Xᵀ(p-y) = 0

But p itself depends on β through the sigmoid, so this is a nonlinear equation. The model is convex, but the optimum still has to be found iteratively.

Common approaches include gradient descent, Newton’s method, and quasi-Newton methods such as L-BFGS. Newton’s method uses the curvature directly:

β_new = β - H^{-1}∇J(β)

That can converge rapidly near the solution, although forming or solving with the Hessian can be expensive for very large feature sets. L-BFGS keeps an approximation to the curvature without storing the full Hessian. The important interview point is not the solver name. It is this: convexity makes the optimization landscape well behaved, but it does not make every numerical implementation equally easy.

Feature scaling can improve conditioning, meaning it makes the curvature less uneven across directions. That can reduce the number of iterations. It does not change the underlying fact that the objective is convex.

Regularization changes the objective used in practice. With L2 regularization:

J_λ(β) = J(β) + (λ/2)||β||²

the Hessian becomes:

H_λ = (1/n)XᵀSX + λI

If every coefficient is regularized and λ is positive, the added λI makes the Hessian positive definite. The objective is then strictly convex and has a unique finite minimizer. Many implementations leave the intercept unregularized, so that conclusion must be stated with that caveat.

L1 regularization keeps the problem convex too, but it is not differentiable at zero. That is why sparse logistic regression needs methods designed for a kink in the objective rather than a plain Hessian-based method.

The failure mode that exposes the nuance

Suppose the data are perfectly separable: every row with y = 0 has a smaller feature value than every row with y = 1. The model can keep increasing the slope so that negative examples receive probabilities closer to 0 and positive examples receive probabilities closer to 1.

The log-loss keeps decreasing toward zero, but a finite sigmoid never produces exactly 0 or 1. The coefficients can therefore grow without bound while the objective approaches its infimum. The objective is still convex; it simply has no finite minimizer.

The first symptoms are practical:

• the coefficient norm becomes very large;
• predicted probabilities become numerically indistinguishable from 0 or 1;
• training reaches its iteration limit or emits a convergence warning;
• small data changes produce enormous coefficient changes.

L2 regularization usually fixes this by making large coefficients expensive. Otherwise, I would inspect feature scaling, duplicate or dependent columns, and whether the labels are genuinely separable.

Convexity also says nothing about whether the chosen metric is the business metric. Logistic regression minimizes log-loss, not accuracy, F1, or AUC. A model can improve log-loss by becoming better calibrated while leaving the default classification threshold unchanged. If the decision cost of a false negative differs from that of a false positive, threshold selection or class weighting must be handled separately.

Finally, convexity applies to logistic regression with a fixed feature matrix and a linear coefficient vector. Adding fixed polynomial or interaction features preserves convexity because the model is still linear in its coefficients. Learning those features with a neural network does not preserve this guarantee.

What they’ll ask next

Does convex mean logistic regression always has a unique solution?
No. Full column rank and a finite optimum are needed for uniqueness in the unregularized case. Rank deficiency creates flat directions, and complete separation can send coefficients toward infinity. L2 regularization gives a unique finite solution when all relevant coefficients are penalized.

Why not use mean squared error with the sigmoid?
You can, but it is not the natural Bernoulli likelihood objective, and the resulting function is generally non-convex in the coefficients. Binary cross-entropy produces the clean gradient p-y and the positive-curvature term p(1-p).

What would you do if training does not converge?
I would check for separation, extreme feature scales, dependent columns, and an unsuitable step size. I would use a stable loss implementation, a solver with line search or curvature information, and L2 regularization when large coefficients are not substantively required.

Say this in the interview: Logistic regression minimizes Bernoulli negative log-likelihood, or binary cross-entropy, and its Hessian is XᵀSX, which is positive semi-definite, so every finite local optimum is global, although separation or rank deficiency means a unique finite optimum is not guaranteed.