Gradient Descent (One Step)

Last lesson left us needing a second way down to the best weights — one that never inverts a matrix, never even builds XᵀX, but simply creeps toward the bottom of the cost bowl.

Picture exactly that situation literally. You are standing somewhere on a smooth valley in thick fog. You cannot see the bottom, but you can feel which way the ground tilts under your feet. So you take a small step in the steepest downhill direction, feel the slope again, and step again. Keep going and the valley floor pulls you in.

That fog-walk is gradient descent, and it answers all three of last lesson’s questions at once:

• The slope under your feet is the gradient ∂L/∂w — the derivative of the loss with respect to the weight, i.e. how fast the loss changes when you nudge w.
• The size of your step is a number you choose.
• The rule that turns “step downhill” into arithmetic is a single line.

One fact to fix before anything else: the gradient points uphill, toward steeper loss. To go down you step the opposite way, against it. That same single move, repeated millions of times, is what trains every neural network you have ever used.

The update rule

Three pieces, and nothing more at exam level.

• Gradient ∂L/∂w — the slope of the loss with respect to the weight. A positive slope means loss rises as w rises, so we should decrease w — and the minus sign does that for us automatically.
• Learning rate η (eta) — a small positive number, the step size. It scales how far you move on each update.
• Subtraction — descent moves against the gradient. Step with its sign and you climb the loss instead (gradient ascent); that flipped sign is the classic error.

The sign is where most marks are lost, because the arithmetic reads backwards from the intuition: a positive gradient does not mean “increase w” — it means decrease it. A positive slope says the loss climbs as w climbs, so the repair is to pull w back, and the minus sign in the rule is exactly what performs that reversal. The gradient names the wrong way; the minus sign turns you around.

Drop a ball onto this loss bowl, change the learning rate, and step it downhill by hand. Watch a small η crawl, and a large one bounce clear across the valley:

The gradient for squared loss

The most common loss in GATE problems is the squared error of a single point. In words, see how far the prediction misses the target, square that miss so it counts the same whether you overshot or undershot, then halve it. The ½ is a convenience that cancels in a moment.

For a prediction ŷ = w·x and target y,

L = ½ (w·x − y)²

Differentiating with respect to w (chain rule — the ½ cancels the 2 that comes down):

∂L/∂w = (w·x − y) · x      ←  this is "(prediction − target) × input"

So the update for one data point becomes

w ← w − η · (w·x − y) · x

That residual (w·x − y) is just the prediction error; the gradient is that error re-weighted by the input. Memorise the shape — GATE drops numbers straight into it.

Run it once on real numbers so the shape stops being abstract. Take w = 0.5, x = 4, y = 3, η = 0.05. The prediction is w·x = 2, so the residual is 2 − 3 = −1 and the gradient is (−1)·4 = −4.

The update gives w ← 0.5 − 0.05·(−4) = 0.5 + 0.2 = 0.7. Notice the weight went up: the gradient was negative, and subtracting a negative adds. The new prediction is 0.7 × 4 = 2.8, closer to the target 3 than the old 2 — which is the whole point.

How GATE asks this

Almost always a NAT: you are handed a current weight, a gradient (or enough to compute it), and a learning rate, and asked for the weight after one update.

GATE DA 2026 (Q29) gave exactly this — one SGD step on f(x) = w·x, where the gradient works out to the input value. SGD means stochastic gradient descent: the identical rule, but with the gradient read off a single training sample rather than the whole dataset. It wanted the new weight to two decimals.

The whole task is one substitution into w − η·g. Occasionally it appears as an MCQ on what happens to η when it is too large or too small.

Worked example

The current weight is w = 10. The gradient of the loss at this point is 10. With learning rate η = 0.1, perform one gradient-descent update. What is w_new?

Substitute straight into the rule:

w_new = w − η · (∂L/∂w)
      = 10 − 0.1 × 10
      = 10 − 1
      = 9.00

The same three lines in Python land on the same number:

w, grad, eta = 10.0, 10.0, 0.1

step  = eta * grad        # 0.1 * 10 = 1.0
w_new = w - step          # 10 - 1 = 9.0

print(f"step  = {step}")
print(f"w_new = {w_new:.2f}")

step  = 1.0
w_new = 9.00

So w_new = 9.00. One step moved the weight down by η × gradient = 1. With a smaller η the step would be smaller and slower. With η large enough that η × gradient overshoots the minimum, the weight would jump clean past it and the loss could grow instead of shrink.

In one breath

Gradient descent reaches the bottom of a loss surface without ever inverting anything: it reads the slope ∂L/∂w (which points uphill), then steps the opposite way by the one-line rule w ← w − η·(∂L/∂w).

The learning rate η sets the step size — too small crawls, too large overshoots and diverges. For the squared loss L = ½(w·x − y)², the gradient is simply (w·x − y)·x, the prediction error times the input. One update by hand is a single clean substitution.

Practice

A question to carry forward

Now you have two ways to the bottom of a loss bowl: the one-shot normal equation, and this patient downhill walk. Both do the same faithful job — they find the weights that make the training loss as small as it can go.

But sit with that word, faithful. Last lesson’s first warning was that a model fitting the training data too perfectly fits its noise too — those huge, wild weights that swing on new data. Gradient descent, done well, walks you straight into that trap, because the trap is at the very bottom of the bowl it is descending.

So the cure cannot be a smarter optimizer. The cure is to change what you ask it to minimise — to add a clause that punishes oversized weights. Here is the thread onward: what does that extra clause look like, what single knob controls how hard it bites, and what is the model giving up in exchange for steadier predictions?