Jacobian, Hessian & Taylor: slope, curvature, and Newton

Backprop left us with two loose ends. A layer outputs a vector, not a number, so its derivative is something larger than a gradient. All through gradient descent we watched one coordinate sprint while another crawled, because the loss curved harder in one direction than the other. The gradient was blind to it. This lesson ties off both.

Gradient descent knows just one thing about your loss — which way is downhill. But the surface is also curving, bending sharply in some directions and gently in others. Reading that curvature is the whole leap from first-order to second-order optimization. It begins with the Taylor expansion.

Taylor: approximate any function with a slope and a bend

Near a point x₀, any smooth function is well-approximated by:

f(x₀ + Δ) ≈ f(x₀) + f′(x₀)·Δ + ½·f″(x₀)·Δ²
            └ value ┘ └ slope ┘   └ curvature ┘

The first two terms give the tangent line (linearization). Add the third — the second derivative — and you get a parabola that bends with the curve.

Scaling up: gradient → Jacobian → Hessian

With many variables, each derivative becomes an array.

• Gradient ∇f — for a scalar function, the vector of first partial derivatives. The direction of steepest ascent (you met it already).
• Jacobian J — for a vector-valued function f: ℝⁿ → ℝᵐ, the m×n matrix whose rows are the gradients of each output. Backpropagation is nothing but multiplying Jacobians together via the chain rule, layer by layer.
• Hessian H — the n×n matrix of second partials, Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ. It encodes curvature in every direction at once. It’s symmetric, so its eigenvalues are real — and they tell you everything:

The multivariate Taylor expansion ties them together:

f(x + Δ) ≈ f(x) + ∇f(x)ᵀ Δ + ½ Δᵀ H(x) Δ

Newton’s method: use the curvature

Gradient descent steps −η∇f and guesses the step size η. Newton’s method uses the Hessian to compute the ideal step: Δ = −H⁻¹∇f. On a quadratic bowl it lands on the minimum in a single jump.

import numpy as np

# A quadratic bowl  f(x) = ½ xᵀ A x − bᵀ x
A = np.array([[3.0, 0.5], [0.5, 1.0]])   # the (constant) Hessian
b = np.array([1.0, 2.0])
grad = lambda x: A @ x - b

x = np.array([4.0, -3.0])
print("start:", x)

# One Newton step uses the Hessian (here, A) -- lands on the minimum exactly
x_newton = x - np.linalg.solve(A, grad(x))
print("after ONE Newton step:", x_newton.round(4))

# Gradient descent needs many small steps and a tuned learning rate
x_gd = x.copy()
for _ in range(50):
    x_gd = x_gd - 0.25 * grad(x_gd)
print("after 50 GD steps:    ", x_gd.round(4))

start: [ 4. -3.]
after ONE Newton step: [0. 2.]
after 50 GD steps:     [0. 2.]

One Newton step lands exactly on the minimum [0, 2] — because on a quadratic, the second-order Taylor model is the function, so solving it once is solving the whole problem.

Gradient descent reaches the same point, but only after all 50 small, learning-rate-tuned steps. It feels the slope and never the curvature.

The catch is that the Hessian is n×n, so for a billion-parameter model it is unthinkable to form or invert. That is why deep learning uses first-order methods (Adam) plus approximations of curvature (L-BFGS, K-FAC) instead.

In one breath

Gradient descent reads only the slope, but a loss surface also curves, and the Taylor expansion captures both: f(x + Δ) ≈ f(x) + ∇fᵀΔ + ½ ΔᵀHΔ — value, then slope, then curvature.

The slope climbs a ladder:

• the gradient (first derivatives of a scalar function).
• the Jacobian (the m×n matrix of a vector-valued function’s partials — and backprop is just a product of these, layer by layer).
• the Hessian (the symmetric matrix of second partials, whose eigenvalues classify a critical point: all-positive = minimum, all-negative = maximum, mixed = saddle).

Newton’s method uses the Hessian for the ideal step Δ = −H⁻¹∇f, exact in one jump on a quadratic. But it is n×n and so infeasible at deep-learning scale, where saddle points, not bad minima, dominate.

Practice

A question to carry forward

The Hessian’s eigenvalues just gave us a classifier for a single point: all-positive means a bowl, mixed means a saddle. But that was a purely local verdict. It described the curvature exactly where we stood and said nothing about the rest of the surface. So ask the bolder question: what if a function’s Hessian were positive everywhere, in every direction, at every point?

Then something remarkable follows. The surface could never bend downward to open a second valley. There would be exactly one bottom, and every downhill path — from any starting point — would have to arrive at it. No saddles to stall on, no bad local minima to fear. That property has a name, convexity, and it is the quiet dividing line between the losses that train effortlessly and the ones that need babysitting.

Here is the thread onward: what exactly makes a function convex? How does the Hessian test for it in a single line? And why does knowing a loss is convex turn optimization from an art back into a guarantee?