Gradient Descent (One Step)

Last lesson left us needing a second way down to the best weights — one that never inverts a matrix, never even builds XᵀX, but simply creeps toward the bottom of the cost bowl. Picture exactly that situation literally. You are standing somewhere on a smooth valley in thick fog. You cannot see the bottom, but you can feel which way the ground tilts under your feet. So you take a small step in the steepest downhill direction, feel the slope again, and step again. Keep going and the valley floor pulls you in.

That fog-walk is gradient descent, and it answers all three of last lesson’s questions at once: the slope under your feet is the gradient, which tells you which way is downhill; the size of your step is a number you choose; and the rule that turns “step downhill” into arithmetic is a single line. The gradient ∂L/∂w points uphill, toward steeper loss — so to go down you step the opposite way, against it. The same single move, repeated millions of times, is what trains every neural network you have ever used.

The update rule

Three pieces, and nothing more at exam level.

• Gradient ∂L/∂w — the slope of the loss with respect to the weight. A positive slope means loss rises as w rises, so we should decrease w — and the minus sign does that for us automatically.
• Learning rate η (eta) — a small positive number, the step size. It scales how far you move on each update.
• Subtraction — descent moves against the gradient. Step with its sign and you climb the loss instead (gradient ascent); that flipped sign is the classic error.

Drop a ball onto this loss bowl, change the learning rate, and step it downhill by hand. Watch a small η crawl, and a large one bounce clear across the valley:

The gradient for squared loss

The most common loss in GATE problems is the squared error of a single point. For a prediction ŷ = w·x and target y,

L = ½ (w·x − y)²

Differentiating with respect to w (chain rule — the ½ cancels the 2 that comes down):

∂L/∂w = (w·x − y) · x      ←  this is "(prediction − target) × input"

So the update for one data point becomes

w ← w − η · (w·x − y) · x

That residual (w·x − y) is just the prediction error; the gradient is that error re-weighted by the input. Memorise the shape — GATE drops numbers straight into it.

How GATE asks this

Almost always a NAT: you are handed a current weight, a gradient (or enough to compute it), and a learning rate, and asked for the weight after one update. GATE DA 2026 (Q29) gave exactly this — one SGD step on f(x) = w·x, where the gradient works out to the input value — and wanted the new weight to two decimals. The whole task is one substitution into w − η·g. Occasionally it appears as an MCQ on what happens to η when it is too large or too small.

Worked example

The current weight is w = 10. The gradient of the loss at this point is 10. With learning rate η = 0.1, perform one gradient-descent update. What is w_new?

Substitute straight into the rule:

w_new = w − η · (∂L/∂w)
      = 10 − 0.1 × 10
      = 10 − 1
      = 9.00

The same three lines in Python land on the same number:

w, grad, eta = 10.0, 10.0, 0.1

step  = eta * grad        # 0.1 * 10 = 1.0
w_new = w - step          # 10 - 1 = 9.0

print(f"step  = {step}")
print(f"w_new = {w_new:.2f}")

step  = 1.0
w_new = 9.00

So w_new = 9.00. One step moved the weight down by η × gradient = 1. With a smaller η the step would be smaller and slower; with η large enough that η × gradient overshoots the minimum, the weight would jump clean past it and the loss could grow instead of shrink.

In one breath

Gradient descent reaches the bottom of a loss surface without ever inverting anything: it reads the slope ∂L/∂w (which points uphill), then steps the opposite way by the one-line rule w ← w − η·(∂L/∂w), where the learning rate η sets the step size — too small crawls, too large overshoots and diverges; for the squared loss L = ½(w·x − y)² the gradient is simply (w·x − y)·x, the prediction error times the input, so one update by hand is a single clean substitution.

Practice

A question to carry forward

Now you have two ways to the bottom of a loss bowl: the one-shot normal equation, and this patient downhill walk. Both do the same faithful job — they find the weights that make the training loss as small as it can go.

But sit with that word, faithful. Last lesson’s first warning was that a model fitting the training data too perfectly fits its noise too — those huge, wild weights that swing on new data. Gradient descent, done well, walks you straight into that trap, because the trap is at the very bottom of the bowl it is descending. So the cure cannot be a smarter optimizer; the cure is to change what you ask it to minimise — to add a clause that punishes oversized weights. Here is the thread onward: what does that extra clause look like, what single knob controls how hard it bites, and what is the model giving up in exchange for steadier predictions?