Walk me through how backpropagation works.

How to think about it

Backpropagation is the algorithm that computes a gradient, meaning the rate at which the loss changes when each parameter changes, for every weight, a learned multiplier, and bias, a learned offset, in a neural network. It first runs the network forward to produce a prediction and loss, then moves backward through the same computation graph with the chain rule; an optimizer, not backpropagation itself, uses those gradients to update the parameters.

What the backward pass is doing

A neural network is a composition of small operations: matrix multiplication, addition, and activation functions. We can represent those operations as a computational graph, a record of how inputs become outputs. Backpropagation walks that graph in reverse.

The reason is the chain rule. If u = g(x) and L = f(u), then:

dL/dx = dL/du × du/dx

The term dL/du is the upstream gradient: how sensitive the final loss already is to u. The term du/dx is the local derivative of the current operation. Multiply them, and we know how sensitive the loss is to x.

At a branching point, backpropagation adds the gradient contributions from every path. That matters when one activation feeds two later layers: changing that activation changes both paths, so both effects belong in its gradient.

Backpropagation is therefore a form of reverse-mode automatic differentiation, which means software applies the chain rule to a graph without asking us to derive every full formula by hand. For a scalar loss and millions of parameters, reverse mode is efficient: one backward traversal produces derivatives for all parameters, with computational cost of roughly the same order as a forward traversal.

Warning — common misconception: backpropagation is not gradient descent. Backpropagation computes the derivatives. Gradient descent is an update rule. Adam, momentum SGD, and other optimizers use those derivatives in different ways.

A numerical walk-through

Consider a tiny network with one input, one hidden unit, and one output. The hidden activation is a rectified linear unit, or ReLU, defined as max(0, z).

Use this model:

• Input: x = 2
• Target: y = 8
• Hidden pre-activation: z = w × x + b
• Hidden activation: a = ReLU(z)
• Prediction: y_hat = v × a + c
• Loss: L = 1/2 × (y_hat - y)^2

Start with:

• w = 1
• b = 1
• v = 2
• c = 0.5

The forward pass gives:

Quantity	Calculation	Value
z	1 × 2 + 1	3
a	ReLU(3)	3
y_hat	2 × 3 + 0.5	6.5
L	0.5 × (6.5 - 8)^2	1.125

The model underpredicts the target. Now we reverse the graph. I will write dL/dw for the derivative of the loss with respect to w.

First, differentiate the loss with respect to the prediction:

dL/dy_hat = y_hat - y = 6.5 - 8 = -1.5

The output layer gives:

dL/dv = dL/dy_hat × a = -1.5 × 3 = -4.5

dL/dc = dL/dy_hat × 1 = -1.5

Now carry the gradient into the hidden activation:

dL/da = dL/dy_hat × v = -1.5 × 2 = -3

Because z is positive, the local derivative of ReLU is 1:

dL/dz = dL/da × 1 = -3

Finally, apply the derivatives of z = w × x + b:

dL/dw = dL/dz × x = -3 × 2 = -6

dL/db = dL/dz × 1 = -3

The negative signs have an intuitive meaning here. Increasing any of these parameters, locally, increases the prediction. Since the prediction is too small, increasing them should reduce the loss, so the loss derivative is negative.

Suppose the learning rate, the step size used by the update, is 0.01. Basic gradient descent updates each parameter as:

parameter_new = parameter_old - learning_rate × gradient

The new values are:

Parameter	Old value	Gradient	New value
w	1	-6	1.06
b	1	-3	1.03
v	2	-4.5	2.045
c	0.5	-1.5	0.515

A new forward pass gives z = 3.15, y_hat = 6.95675, and L = 0.54418528125. The loss fell from 1.125 to about 0.544, so this step moved the model in a useful direction.

Real networks do the same thing with vectors and matrices. For a dense layer written as Y = XW, under the usual row-batch convention, the weight gradient is dL/dW = Xᵀ × dL/dY. The scalar example is not a different algorithm; it is the same local rule with the dimensions removed.

The production training pattern

A training step usually follows this sequence:

1. The forward pass computes activations and the prediction. It stores intermediate values, often called the cache, because the backward pass needs them again.
2. The loss function compares the prediction with the target. For classification, this might be cross-entropy; for regression, it might be squared error.
3. Backpropagation starts at the scalar loss with sensitivity 1 and traverses operations in reverse order.
4. Each operation receives an upstream gradient, multiplies it by its local derivative, and passes the result to earlier operations. If a parameter is used more than once, its gradient contributions are summed.
5. The optimizer updates the parameters. The old gradients are then cleared before the next step, so one batch does not accidentally contaminate the next batch.

With a minibatch, the model normally computes one loss per example and reduces them, often by taking their mean. The resulting gradient is then an average gradient. If the loss is instead summed, the gradient grows with batch size, which changes the effective update scale.

Backpropagation is needed for training, not ordinary inference. Once the parameters are learned, producing a prediction only requires the forward pass. Running the backward pass during serving would add work and memory without improving the prediction.

The senior-level nuance

Backpropagation needs a useful derivative path. ReLU is not differentiable exactly at zero, but implementations choose a subgradient convention, commonly zero. That is usually harmless. A hard argmax, a discrete branch, or a string lookup is different: there is no ordinary continuous derivative that tells the model how to change the choice. Such operations need to be moved outside the gradient path or replaced with a differentiable approximation.

Gradients can also vanish or explode. In a deep chain, local derivatives are multiplied. If each layer contributes a factor of 0.5, then after 20 layers the signal is about 0.00000095. If each contributes a factor of 2, the signal is 1,048,576 times larger. Vanishing gradients leave early layers barely learning; exploding gradients produce unstable updates or numerical overflow. Initialization, normalization, suitable activations, residual connections, clipping, and optimizer settings can help, but none is a universal cure.

There is a memory trade-off. Backpropagation is efficient because it reuses forward values, but storing those activations can consume more memory than inference. Activation checkpointing reduces memory by discarding selected values and recomputing them during the backward pass. It buys memory at the cost of extra computation.

Finally, a gradient is a local direction, not a guarantee of the globally best solution. Neural-network losses are generally non-convex, and minibatch gradients are noisy estimates. A smaller loss after one update is desirable, but the learning rate, batch composition, parameterization, and optimizer all affect whether training remains stable.

A failure mode you would actually see

Suppose a binary classifier sits at a loss of about 0.693 for thousands of steps, predicts about 0.5 for every example, and reports zero gradient norms in its early layers. In a ReLU network, the likely culprit is that those units receive negative pre-activations for every example. ReLU outputs zero and sends a zero local derivative backward, so the corresponding weights receive no learning signal.

Inspect activation distributions and per-layer gradient norms. Try fitting one small batch; a healthy model should be able to nearly memorize it. If every layer has missing or zero gradients, look for a disconnected computation graph, a loss that is not connected to the prediction, or an update that never applies. For a tiny test model, compare backpropagation with a central finite-difference estimate such as (L(theta + epsilon) - L(theta - epsilon)) / (2 × epsilon). Finite differences are useful for debugging, but far too expensive for normal training because they require extra loss evaluations for parameters.

What they’ll ask next

Is backpropagation the same as gradient descent?

No. Backpropagation calculates dL/dtheta for each parameter. Gradient descent uses theta_new = theta - learning_rate × dL/dtheta. An optimizer such as momentum SGD or Adam changes how those gradients are scaled, accumulated, or adapted before the update.

Why not calculate every derivative independently?

That would repeat the same work. Reverse-mode automatic differentiation computes one downstream sensitivity, reuses it at each node, and shares intermediate results across all parameter gradients. It is especially effective when the output is one scalar loss and the input consists of many parameters.

How does this work for recurrent networks?

Backpropagation through time unrolls the recurrent network across its time steps and applies the same reverse process to that larger graph. Shared recurrent weights appear at every step, so their gradient contributions are added. Full unrolling captures long-range effects but uses more memory and can suffer from vanishing or exploding gradients; truncated backpropagation through time saves resources by limiting how far the gradient travels.

Say this in the interview

“Backpropagation runs the network forward to compute the loss, then applies the chain rule backward through the computation graph to obtain every parameter’s gradient; the optimizer separately uses those gradients to update the weights and biases.”