Backpropagation (One Step)

Last lesson left the obstacle that froze neural networks for twenty years. An MLP carries thousands of weights, and gradient descent needs one number for every one of them: ∂L/∂w, read aloud as “how much the loss L moves when you nudge the weight w”. Yet the loss reaches each weight only after threading through layer upon layer of mixing and squashing.

Computing those gradients one weight at a time, from scratch, is hopeless. The escape is the chain rule from calculus — the rule that a derivative through a chain of steps is the product of the derivatives of the individual steps — run backward. That is backpropagation.

The idea is to treat the network as a computation graph: a picture of the forward pass in which every node is one value and every arrow is one small operation that produced it. Over that picture, the gradient flows like blame down an assembly line.

If you know how much the loss blames a layer’s output, the chain rule tells you cheaply how much to blame that layer’s inputs. You hand that blame back to the previous layer and repeat, sweeping right to left until every weight has its gradient. One backward pass, every gradient at once. GATE asks you to do this for a single weight through a tiny net, so the whole machine fits on a page.

The chain rule along a path

The forward pass sends values left to right. Backprop sends gradients right to left. Each edge carries a local derivative — the derivative of one node with respect to its immediate input. The gradient of the loss with respect to any quantity is the product of those local derivatives along the path back to it.

For a weight w on one path to the loss, write that product out. Each factor asks the same small question — how much does this node move when the node feeding it moves?

∂L/∂w = (∂L/∂output) · (∂output/∂hidden) · … · (∂·/∂w)

Why multiply and work backwards

Two things about that line trip up almost everyone on a first reading, so let us name them. Why multiply the factors, and why work backwards?

Multiply because effects along a chain compound instead of adding up. If nudging w moves the hidden value by a factor of 2, and moving the hidden value moves the output by a factor of 3, then nudging w moves the output by a factor of 6 — not 5. Work backwards because the factor you need at each edge is “how much the loss cares about this node”. That is unknowable until everything downstream has been handled. So the sweep starts at the loss and walks left, each layer reusing the number the layer on its right just produced. That reuse is the entire reason backprop is cheap.

Each factor is a local derivative. The only non-obvious one here is the activation. ReLU passes a positive input through unchanged and flattens anything negative to zero, so its local derivative is 1 if its input was positive, else 0. It behaves as a gate — passing the upstream gradient through unchanged, or blocking it entirely.

Play with the mechanism on a single neuron — run the forward pass to fill each value, then the backward pass to watch every gradient form as downstream gradient × the local derivative on the edge:

How GATE asks this

A NAT or MCQ gives you a tiny network with concrete weights and an input. You compute one partial derivative (often ∂y/∂w or ∂L/∂w) by the chain rule. The graph is small enough to trace by hand. The skill being tested is identifying the path and multiplying the local derivatives, plus remembering the ReLU gate. This pattern appeared on GATE DA 2025.

Worked example — one chain-rule step

A tiny net: input x = 2. Hidden unit h = ReLU(w₁·x) with w₁ = 1, so h = ReLU(2) = 2. Output y = w₂·h with w₂ = 3, so y = 6. Compute ∂y/∂w₁ by the chain rule.

The path from w₁ to y is w₁ → (w₁x) → h → y. Multiply the local derivative on each edge:

∂y/∂h        = w2 = 3                 (since y = w2·h)
∂h/∂(w1·x)   = 1                      (ReLU input is 2 > 0, so gate = 1)
∂(w1·x)/∂w1  = x = 2                  (linear in w1)

∂y/∂w1 = (∂y/∂h) · (∂h/∂(w1·x)) · (∂(w1·x)/∂w1)
       = 3 · 1 · 2
       = 6

So ∂y/∂w₁ = 6. The ReLU gate was open (input 2 > 0), so it contributed a factor of 1 and simply passed the gradient through. Had the ReLU input been negative, the gate would be 0 and the whole gradient would vanish.

If the number still feels abstract, check it by nudging. Raise w₁ from 1 to 1.1: the pre-activation becomes 2.2, the ReLU passes it, and y = 3 × 2.2 = 6.6. The output rose by 0.6 for an input change of 0.1 — a ratio of 6. That is exactly the derivative the chain rule handed you. The same three multiplications in code:

x, w1, w2 = 2.0, 1.0, 3.0

z = w1 * x                       # 2.0  (pre-activation)
h = max(0.0, z)                  # ReLU -> 2.0
y = w2 * h                       # 6.0

dy_dh   = w2                     # 3
dh_dz   = 1.0 if z > 0 else 0.0  # ReLU gate: input 2 > 0 -> 1
dz_dw1  = x                      # 2
dy_dw1  = dy_dh * dh_dz * dz_dw1
print("y      =", y)
print("dy/dw1 =", dy_dw1)

y      = 6.0
dy/dw1 = 6.0

In one breath

Backpropagation is the chain rule walked backward over the network’s computation graph. It sweeps gradients right to left, and the gradient to any weight on a single path is the product (never the sum) of the local derivatives along that path: ∂L/∂w = (∂L/∂out)·(∂out/∂h)·…·(∂·/∂w). The only tricky factor is the activation, since ReLU’s local derivative is a gate: 1 when its input was positive and 0 otherwise. A unit whose pre-activation was ≤ 0 kills the gradient flowing through it.

Practice

A question to carry forward

With backprop, the whole supervised machine is complete: features and labels go in, gradients flow back, weights settle, a trained predictor comes out. Look back over every model since the start of this chapter:

• regression
• the classifiers
• the neural net

One ingredient was always present, quietly indispensable: the labels. The answer key. Every method learned by comparing its guess against a known truth.

Now take the answer key away. You are handed a pile of points:

• customers
• pixels
• documents

Nothing tells you which belongs with which. Yet you suspect they fall into natural groups: a few customer types, a handful of topics. Here is the thread onward into unsupervised learning: with no labels at all to guide you, how could an algorithm discover those groups on its own? What would it even mean for a grouping to be “good,” and how do you find one by alternately guessing the groups and refining them?