What is backpropagation and how does the chain rule make it work?

How to think about it

Backpropagation is reverse-mode automatic differentiation: it computes the gradient of a scalar loss with respect to every model parameter by applying the chain rule backward through the operations that produced the loss. The chain rule turns a long dependency into products of small local derivatives, while the reverse pass reuses each intermediate gradient instead of recomputing it for every weight; an optimizer then uses those gradients to update the parameters.

Why the chain rule is the key

A neural network is a composition of functions. An input passes through linear layers, activations, attention operations, and eventually a loss function. If a parameter changes slightly, the loss changes because that parameter changes its immediate output, which changes the next output, and so on until the loss changes.

The chain rule multiplies those effects. If L is the loss, z is an intermediate value, and w is a parameter, then:

dL/dw = (dL/dz) · (dz/dw)

The term dz/dw is the local derivative: how the immediate output changes when w changes. The term dL/dz is the upstream gradient: how the final loss changes when z changes. Multiplying them gives the total effect of w on the loss.

A computational graph is a record of these operations, with values at nodes and dependencies along edges. During the forward pass, the network computes predictions and usually saves intermediate values such as layer inputs and activations. During the backward pass, it starts with dL/dL = 1 and walks the graph in reverse order. At each operation, it multiplies the incoming gradient by that operation’s local derivative.

If a value affects the loss along two paths, the gradients from those paths are added. This matters in residual connections, shared embeddings, and recurrent computations. A parameter used in three places receives the sum of all three contributions, not just the contribution from the last place the algorithm visits.

The reverse direction is efficient because the usual training objective is one scalar loss with millions or billions of inputs to that function: the model parameters. Reverse-mode differentiation computes the derivatives of that one output with respect to all those inputs in one backward traversal. A naive finite-difference estimate would need separate forward evaluations for parameters one by one. Backpropagation instead performs one forward pass and one reverse pass, with a total cost of roughly the same order as a few forward passes.

That is not literally always twice the runtime. Backward matrix operations can have different costs from forward operations, and saved activations consume memory. But the important scaling result remains: the cost is tied to the computation graph, not multiplied by the number of parameters.

What happens inside a dense layer

For one example, using column-vector notation, consider a dense layer:

z = W @ a_prev + b
a = ReLU(z)

Here W is the weight matrix, b is the bias vector, and a_prev is the activation from the previous layer. Suppose g_a means the gradient of the loss with respect to a, and that gradient arrives from the layer above.

The backward calculation is:

g_z = g_a * (z > 0)       # elementwise ReLU derivative
g_W = g_z @ a_prev.T
g_b = g_z
g_a_prev = W.T @ g_z

The multiplication by (z > 0) is the chain rule through ReLU. A positive preactivation passes the gradient through. A negative preactivation has a local derivative of zero, so its gradient becomes zero. At exactly zero, ReLU is not differentiable; deep-learning libraries choose a defined subgradient, commonly zero.

The same g_z is used three times: to calculate the weight gradient, the bias gradient, and the gradient passed to the preceding layer. That reuse is the practical heart of backpropagation. It calculates an intermediate result once and lets every dependent parameter benefit from it.

For a batch, the formulas include summation or averaging across examples, and the matrix orientation depends on whether examples are stored as rows or columns. The principle does not change.

A concrete example

Take a tiny network with one input, one hidden ReLU unit, and one linear output:

x  = 2
w1 = 3
b1 = -1
w2 = 4
b2 = 0
target y = 10

The forward pass is:

z1   = w1*x + b1 = 3*2 - 1 = 5
h    = ReLU(z1) = 5
yhat  = w2*h + b2 = 4*5 + 0 = 20
L    = 0.5*(yhat - y)^2 = 0.5*(20 - 10)^2 = 50

The prediction is 20, so the model is 10 units above the target. The derivative of the loss with respect to the prediction is:

dL/dyhat = yhat - y = 10

Now move backward:

dL/dw2 = (dL/dyhat) * (dyhat/dw2)
       = 10 * h
       = 10 * 5
       = 50

dL/dh = (dL/dyhat) * (dyhat/dh)
      = 10 * w2
      = 10 * 4
      = 40

dL/dz1 = (dL/dh) * (dh/dz1)
       = 40 * 1
       = 40

The ReLU derivative is one because z1 is positive. The gradient for the first-layer weight contains the whole chain:

dL/dw1 = (dL/dyhat) * (dyhat/dh) * (dh/dz1) * (dz1/dw1)
       = 10 * 4 * 1 * 2
       = 80

Likewise, dL/db1 is 40, dL/db2 is 10, and dL/dw2 is 50. Notice that the value dL/dh = 40 was computed once and reused to obtain both first-layer gradients.

A small PyTorch version produces the same numbers:

import torch

x = torch.tensor(2.0)
w1 = torch.tensor(3.0, requires_grad=True)
b1 = torch.tensor(-1.0, requires_grad=True)
w2 = torch.tensor(4.0, requires_grad=True)
b2 = torch.tensor(0.0, requires_grad=True)

z1 = w1 * x + b1
h = torch.relu(z1)
yhat = w2 * h + b2
loss = 0.5 * (yhat - 10.0) ** 2

loss.backward()

print(loss.item())
print(w1.grad.item(), b1.grad.item(), w2.grad.item(), b2.grad.item())

It prints:

50.0
80.0 40.0 50.0 10.0

loss.backward() computes and stores the gradients. It does not update w1, b1, w2, or b2. An optimizer performs that separate step, usually using gradient descent:

parameter = parameter - learning_rate * gradient

What the production training loop does

A typical PyTorch training step is:

optimizer.zero_grad()
prediction = model(x)
loss = criterion(prediction, y)
loss.backward()
optimizer.step()

The forward pass builds a graph because model parameters require gradients. backward() traverses that graph in reverse and places gradients in each parameter’s .grad field. step() reads those gradients and applies the optimizer’s update rule. Adam, momentum SGD, and other optimizers alter the update rule, but they still need the gradients produced by backpropagation.

Common mistake: backpropagation is not gradient descent. Backpropagation answers, “Which direction would change the loss, and by how much?” The optimizer answers, “Given that information and my learning-rate policy, how should I change the parameters?”

PyTorch accumulates gradients by default. Calling backward() twice without clearing gradients adds the second result to the first. If this is accidental, the first symptom is often growing gradient norms and erratic or diverging loss. Clearing gradients before the next batch prevents that:

optimizer.zero_grad()

Accumulation is sometimes intentional when several microbatches are used to simulate a larger batch. In that case, the losses and learning rate must be scaled consistently. Otherwise, the effective gradient becomes larger simply because more microbatches were processed.

Trade-offs and failure modes

Backpropagation trades memory for speed. The forward pass saves intermediate activations because the backward pass needs them to calculate local derivatives. A large transformer can therefore run out of GPU memory during training even when the same model fits comfortably for inference. Activation checkpointing reduces memory by discarding selected activations and recomputing them during backpropagation; the trade-off is extra computation.

Deep networks also multiply many local derivatives. If most factors are smaller than one, the gradient can shrink toward zero as it travels backward. If factors are repeatedly larger than one, it can grow explosively. The first visible symptom may be a model whose early layers barely change, or a loss that suddenly becomes nan.

ReLU introduces a different failure mode. If a unit’s preactivation stays negative for every training example, its ReLU derivative stays zero. That unit receives no gradient and its incoming weights stop learning. The symptom is an activation channel that is always zero and a corresponding gradient that is exactly zero. Initialization, learning rate, activation choice, normalization, residual connections, and gradient clipping can all affect these problems.

Backpropagation is most natural when the computation is differentiable, or when useful subgradients are defined. It does not directly provide a gradient through a hard discrete choice such as “select item 7” or “take this branch and discard the other one.” A system built around genuinely discrete decisions may need a differentiable relaxation, a surrogate objective, policy-gradient methods, or a different optimization strategy. Adding backward() does not make a discontinuous operation informative.

For a deeper derivation, see the backpropagation walkthrough.

What they’ll ask next

Is backpropagation the same as automatic differentiation?

No. Automatic differentiation is the broader family of methods for mechanically applying the chain rule to a program. Backpropagation is reverse-mode automatic differentiation, especially useful when the program has many inputs and one scalar output. It is different from symbolic differentiation, which manipulates algebraic expressions, and numerical differentiation, which estimates derivatives with perturbations.

Why use reverse mode instead of forward mode?

It depends on the shape of the function. A training loss has millions of parameter inputs but usually one scalar output, so reverse mode computes all parameter gradients efficiently from one backward pass. Forward mode is attractive when there are few inputs and many outputs, or when computing directional derivatives. The choice is about input-output dimensions, not a claim that reverse mode is universally better.

Why might gradients vanish or explode?

Backpropagation multiplies local derivatives along a path. Repeated factors below one shrink the result; repeated factors above one enlarge it. I would inspect per-layer gradient norms and activation statistics first, then check initialization, learning rate, activation saturation, normalization, and residual paths. Gradient clipping can limit explosions, but it does not repair a fundamentally uninformative gradient.

Say this in the interview: Backpropagation is reverse-mode automatic differentiation: the chain rule propagates one upstream loss gradient through local derivatives in reverse, reusing intermediate results to compute every parameter gradient efficiently, while the optimizer performs the actual update.