Gradient descent from scratch

The last lesson handed us the gradient and the update rule w ← w − lr·∇L, then flagged the one thing it glossed over — that learning rate, and whether the walk settles or flies apart. Gradient descent has earned a lesson of its own, so here it is in full. Training a model is an optimization problem: the loss is a function that eats some weights and returns a number, and you want the lowest number it can give. If you could see the whole loss surface you would just point at the bottom — but in real ML that surface lives in millions of dimensions, and you can only feel it locally, one point at a time. Gradient descent is how you navigate it blind: at each step, work out which way is steepest downhill, take a step in that direction, repeat.

The gradient is the steepest-uphill direction

Picture a hilly landscape. You’re standing somewhere on it. The gradient at your spot is the vector that points uphill — specifically, in the direction the slope is steepest. Its length tells you how steep.

Why steepest? The slope in any direction u equals ∇L · u (dot product). That dot product is largest when u lines up with ∇L, so ∇L itself is the direction of maximum increase.

Since we want to go down, we step in the opposite direction:

w_new = w_old - learning_rate * gradient

That’s gradient descent. One line. Everything else is just choosing the loss, computing the gradient, and tuning the learning rate.

A 2D example you can watch

Let’s minimize loss(w) = w[0]² + 3 * w[1]² — a tilted bowl with its minimum at (0, 0). The gradient is [2*w[0], 6*w[1]] (just calculus).

import numpy as np

def loss(w):
    return w[0]**2 + 3 * w[1]**2

def grad(w):
    return np.array([2 * w[0], 6 * w[1]])

w = np.array([4.0, 2.5])      # starting point -- way off the minimum
lr = 0.1                      # learning rate

print(f"{'step':>4} {'w[0]':>8} {'w[1]':>8} {'loss':>10}")
for step in range(15):
    print(f"{step:>4} {w[0]:>8.4f} {w[1]:>8.4f} {loss(w):>10.4f}")
    w = w - lr * grad(w)

print(f"\nFinal w: {w}, loss: {loss(w):.6f}")

step     w[0]     w[1]       loss
   0   4.0000   2.5000    34.7500
   1   3.2000   1.0000    13.2400
   2   2.5600   0.4000     7.0336
   3   2.0480   0.1600     4.2711
   4   1.6384   0.0640     2.6966
   5   1.3107   0.0256     1.7200
   6   1.0486   0.0102     1.0998
   7   0.8389   0.0041     0.7037
   8   0.6711   0.0016     0.4504
   9   0.5369   0.0007     0.2882
  10   0.4295   0.0003     0.1845
  11   0.3436   0.0001     0.1181
  12   0.2749   0.0000     0.0756
  13   0.2199   0.0000     0.0484
  14   0.1759   0.0000     0.0309

Final w: [1.40737488e-01 2.68435456e-06], loss: 0.019807

Watch the loss fall and w march toward (0, 0). Each row is one step downhill — and notice w[1] collapses far faster than w[0] (it is essentially zero by step 5), because the bowl is three times steeper in that direction (the 6·w[1] gradient). The same uneven-curvature effect you glimpsed last lesson, seen here in full.

Learning rate is everything

The learning rate lr is the single most-tuned knob in ML. Too small and training takes forever. Too big and you overshoot — sometimes wildly. A common misconception: a bigger learning rate is always faster. Not so — beyond a threshold it causes divergence, not speed.

import numpy as np

def loss(w): return w[0]**2 + 3 * w[1]**2
def grad(w): return np.array([2 * w[0], 6 * w[1]])

def run(lr, steps=10):
    w = np.array([4.0, 2.5])
    history = [loss(w)]
    for _ in range(steps):
        w = w - lr * grad(w)
        history.append(loss(w))
    return history

for lr in [0.01, 0.1, 0.3, 0.5]:
    losses = run(lr)
    final = losses[-1]
    label = "diverging!" if final > losses[0] else "converging"
    print(f"lr={lr:<5} loss after 10 steps: {final:>12.4f}   {label}")

lr=0.01  loss after 10 steps:      16.1212   converging
lr=0.1   loss after 10 steps:       0.1845   converging
lr=0.3   loss after 10 steps:       0.2162   converging
lr=0.5   loss after 10 steps: 19660800.0000   diverging!

Read the four rows:

• lr=0.01 — converging, but barely begun after 10 steps (still at 16).
• lr=0.1 — the sweet spot: down to 0.18.
• lr=0.3 — still converging, but overshooting and slightly worse than 0.1 — past the ideal.
• lr=0.5 — diverging: the loss has exploded to ~20 million. Each step overshoots the valley and lands farther up the far wall than it started.

This is why nearly every modern optimizer (Adam, AdamW, SGD with schedulers) is really just gradient descent with smarter rules for picking the learning rate per parameter and per step.

Convex vs non-convex: do you need to worry about local minima?

A convex loss has a single valley — wherever you start, gradient descent reaches the global minimum, provided the learning rate is small enough (you saw above that even a perfect bowl diverges at lr = 0.5). Linear regression, logistic regression, and SVMs all have convex losses. That’s why they’re so reliable: there is no bad valley to get stuck in — the only way to fail is a bad step size.

A non-convex loss has multiple valleys. Neural networks are wildly non-convex. In principle you could land in a poor local minimum. In practice, deep networks have so many parameters that “most” local minima are fine — usually as good as the global one — and stochastic gradient descent + momentum tends to find them.

The deep-learning era basically discovered: “Non-convex is fine if you have enough data and a decent architecture.” Two decades of theory warnings turned out to be more pessimistic than reality.

A non-toy example: linear regression by hand

Fit y ≈ w * x + b to noisy data, with gradient descent updating both parameters:

import numpy as np

rng = np.random.default_rng(0)

# True relationship: y = 2.5 * x + 1.0 + noise
x = rng.uniform(-2, 2, size=100)
y_true = 2.5 * x + 1.0 + rng.normal(0, 0.3, size=100)

w, b = 0.0, 0.0      # start from zero
lr = 0.05

for step in range(200):
    y_pred = w * x + b
    err = y_pred - y_true
    # gradient of mean squared error
    grad_w = (2 / len(x)) * (err * x).sum()
    grad_b = (2 / len(x)) * err.sum()
    w -= lr * grad_w
    b -= lr * grad_b
    if step % 40 == 0:
        mse = (err ** 2).mean()
        print(f"step {step:>3}  w={w:.3f}  b={b:.3f}  mse={mse:.4f}")

print(f"\nLearned: y ~ {w:.2f} * x + {b:.2f}")
print(f"True:    y  = 2.50 * x + 1.00")

step   0  w=0.394  b=0.146  mse=11.3288
step  40  w=2.489  b=0.975  mse=0.0846
step  80  w=2.490  b=0.978  mse=0.0846
step 120  w=2.490  b=0.978  mse=0.0846
step 160  w=2.490  b=0.978  mse=0.0846

Learned: y ~ 2.49 * x + 0.98
True:    y  = 2.50 * x + 1.00

Starting from w=0, b=0, the fit reaches w≈2.49, b≈0.98 — all but converged by step 40, and the residual mse≈0.085 is just the noise we added. No black-box library: only element-wise multiplication, a sum, and the update rule. That is the entire training loop of any model; the only things that change for big models are how you compute the gradient (autograd) and which optimizer rule takes the step.

In one breath

Gradient descent minimises a loss by repeating one line — w ← w − lr·∇L(w) — stepping in the direction of steepest descent (the negative gradient, since ∇L itself points steepest uphill). The learning rate lr is the make-or-break knob: too small and training crawls, too large and the steps overshoot until the loss explodes — which is why every modern optimizer (SGD, Adam, AdamW) is really gradient descent with smarter, per-parameter, per-step rules for choosing it. On a convex loss (linear and logistic regression, SVMs) any sensible lr reaches the one global minimum; deep nets are wildly non-convex, yet with enough parameters and data most of their local minima turn out to be good enough.

Practice

A question to carry forward

Every line of this lesson rested on one quietly enormous assumption: that you already have the gradient ∇L. For the toy bowl and the one-line regression we simply wrote it down by hand — [2w₀, 6w₁], (2/n)·Σ err·x. But a real network has millions of weights buried under layer upon layer of matrix multiplies and nonlinearities. You cannot differentiate that by hand, and computing each of a million partials on its own would re-walk the entire network a million times over.

So here is the thread onward: what is the algorithm that computes all of those millions of partial derivatives in a single backward sweep — for barely more than the cost of the forward pass itself? It is backpropagation, and it is not new magic but the chain rule from a few lessons ago, applied to the network’s computation graph and walked deliberately backward. How does that one move make training a deep net feasible at all?