Backpropagation (One Step)

Last lesson left the obstacle that froze neural networks for twenty years: an MLP carries thousands of weights, and gradient descent needs ∂L/∂w for every one of them — yet the loss reaches each weight only after threading through layer upon layer of mixing and squashing. Computing those gradients one weight at a time, from scratch, is hopeless. The escape is the chain rule from calculus, run backward, and it is called backpropagation.

The idea is to treat the network as a computation graph and notice that the gradient flows like blame down an assembly line. If you know how much the loss blames a layer’s output, the chain rule tells you cheaply how much to blame that layer’s inputs — and you hand that blame back to the previous layer, and repeat, sweeping right to left until every weight has its gradient. One backward pass, every gradient at once. GATE asks you to do this for a single weight through a tiny net, so the whole machine fits on a page.

The chain rule along a path

The forward pass sends values left to right. Backprop sends gradients right to left: each edge carries a local derivative — the derivative of one node with respect to its immediate input — and the gradient of the loss with respect to any quantity is the product of those local derivatives along the path back to it.

For a weight w on one path to the loss:

∂L/∂w = (∂L/∂output) · (∂output/∂hidden) · … · (∂·/∂w)

Each factor is a local derivative. The only non-obvious one here is the activation: ReLU’s local derivative is 1 if its input was positive, else 0. It behaves as a gate — passing the upstream gradient through unchanged, or blocking it entirely.

Play with the mechanism on a single neuron — run the forward pass to fill each value, then the backward pass to watch every gradient form as downstream gradient × the local derivative on the edge:

How GATE asks this

A NAT or MCQ: a tiny network is specified with concrete weights and an input, and you compute one partial derivative (often ∂y/∂w or ∂L/∂w) by the chain rule. The graph is small enough to trace by hand — the skill being tested is identifying the path and multiplying the local derivatives, plus remembering the ReLU gate. This pattern appeared on GATE DA 2025.

Worked example — one chain-rule step

A tiny net: input x = 2. Hidden unit h = ReLU(w₁·x) with w₁ = 1, so h = ReLU(2) = 2. Output y = w₂·h with w₂ = 3, so y = 6. Compute ∂y/∂w₁ by the chain rule.

The path from w₁ to y is w₁ → (w₁x) → h → y. Multiply the local derivative on each edge:

∂y/∂h        = w2 = 3                 (since y = w2·h)
∂h/∂(w1·x)   = 1                      (ReLU input is 2 > 0, so gate = 1)
∂(w1·x)/∂w1  = x = 2                  (linear in w1)

∂y/∂w1 = (∂y/∂h) · (∂h/∂(w1·x)) · (∂(w1·x)/∂w1)
       = 3 · 1 · 2
       = 6

So ∂y/∂w₁ = 6. The ReLU gate was open (input 2 > 0), so it contributed a factor of 1 and simply passed the gradient through — had the ReLU input been negative, the gate would be 0 and the whole gradient would vanish. The same three multiplications in code:

x, w1, w2 = 2.0, 1.0, 3.0

z = w1 * x                       # 2.0  (pre-activation)
h = max(0.0, z)                  # ReLU -> 2.0
y = w2 * h                       # 6.0

dy_dh   = w2                     # 3
dh_dz   = 1.0 if z > 0 else 0.0  # ReLU gate: input 2 > 0 -> 1
dz_dw1  = x                      # 2
dy_dw1  = dy_dh * dh_dz * dz_dw1
print("y      =", y)
print("dy/dw1 =", dy_dw1)

y      = 6.0
dy/dw1 = 6.0

In one breath

Backpropagation is the chain rule walked backward over the network’s computation graph: it sweeps gradients right to left, and the gradient to any weight on a single path is the product (never the sum) of the local derivatives along that path — ∂L/∂w = (∂L/∂out)·(∂out/∂h)·…·(∂·/∂w) — where the only tricky factor is the activation, since ReLU’s local derivative is a gate, 1 when its input was positive and 0 otherwise, so a unit whose pre-activation was ≤ 0 kills the gradient flowing through it.

Practice

A question to carry forward

With backprop, the whole supervised machine is complete: features and labels go in, gradients flow back, weights settle, a trained predictor comes out. Look back over every model since the start of this chapter — regression, the classifiers, the neural net — and one ingredient was always present, quietly indispensable: the labels. The answer key. Every method learned by comparing its guess against a known truth.

Now take the answer key away. You are handed a pile of points — customers, pixels, documents — and nothing tells you which belongs with which. Yet you suspect they fall into natural groups: a few customer types, a handful of topics. Here is the thread onward into unsupervised learning: with no labels at all to guide you, how could an algorithm discover those groups on its own — what would it even mean for a grouping to be “good,” and how do you find one by alternately guessing the groups and refining them?