Jacobian, Hessian & Taylor: slope, curvature, and Newton
Gradient descent only uses the slope. But every loss surface also has curvature — and the Jacobian, the Hessian, and the Taylor expansion are how you read it. Curvature is the difference between crawling and converging.
What you'll learn
- Taylor expansion: linear (slope) and quadratic (curvature) approximations of a function
- The Jacobian: stacking the gradients of a vector-valued function — and why backprop is a product of them
- The Hessian: the matrix of second derivatives, and what its eigenvalues say about minima, maxima, and saddles
- The multivariate Taylor formula and how Newton's method uses curvature to take smarter steps
- Why saddle points — not local minima — dominate deep-learning loss surfaces
Before you start
Gradient descent knows one thing about your loss: which way is downhill. But the surface is also curving — bending sharply in some directions, gently in others. Reading that curvature is the leap from first-order to second-order optimization, and it all starts with the Taylor expansion.
Taylor: approximate any function with a slope and a bend
Near a point x₀, any smooth function is well-approximated by:
f(x₀ + Δ) ≈ f(x₀) + f′(x₀)·Δ + ½·f″(x₀)·Δ²
└ value ┘ └ slope ┘ └ curvature ┘The first two terms give the tangent line (linearization). Add the third — the second derivative — and you get a parabola that bends with the curve.
Scaling up: gradient → Jacobian → Hessian
With many variables, each derivative becomes an array.
- Gradient
∇f— for a scalar function, the vector of first partial derivatives. The direction of steepest ascent (you met it already). - Jacobian
J— for a vector-valued functionf: ℝⁿ → ℝᵐ, them×nmatrix whose rows are the gradients of each output. Backpropagation is nothing but multiplying Jacobians together via the chain rule, layer by layer. - Hessian
H— then×nmatrix of second partials,Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ. It encodes curvature in every direction at once. It’s symmetric, so its eigenvalues are real — and they tell you everything:
| Hessian eigenvalues | Critical point |
|---|---|
| all positive | local minimum (bowl up) |
| all negative | local maximum (bowl down) |
| mixed signs | saddle point (up one way, down another) |
The multivariate Taylor expansion ties them together:
f(x + Δ) ≈ f(x) + ∇f(x)ᵀ Δ + ½ Δᵀ H(x) ΔNewton’s method: use the curvature
Gradient descent steps −η∇f and guesses the step size η. Newton’s
method uses the Hessian to compute the ideal step: Δ = −H⁻¹∇f. On a
quadratic bowl it lands on the minimum in a single jump.
One Newton step nails it; gradient descent is still creeping in after 50.
The catch: the Hessian is n×n, so for a billion-parameter model it’s
unthinkable to form or invert — which is why deep learning uses first-order
methods (Adam) plus approximations of curvature (L-BFGS, K-FAC) instead.
Quick check
Quick check
Practice this in an interview
All questionsSigmoid's derivative peaks at 0.25 and approaches zero in both tails, so the chain of gradient multiplications collapses exponentially in deep networks. Tanh's derivative peaks at 1 and is zero-centered, which helps weight update symmetry, but it still saturates at large magnitudes and the gradient still shrinks to near-zero in both tails.
Logistic regression minimizes binary cross-entropy (log-loss), which is the negative log-likelihood of the Bernoulli distribution given the sigmoid-transformed linear predictions. The Hessian of log-loss is positive semi-definite everywhere, guaranteeing a convex surface with a unique global minimum.
The normal equation gives an exact closed-form solution in O(p³) time but becomes impractical when the number of features p is large (typically above ~10,000) because matrix inversion is cubic. Gradient descent scales as O(np) per iteration, making it the only viable option for large feature spaces or online learning.
Vanishing gradients occur when repeated multiplication of small derivatives during backpropagation drives gradients toward zero, starving early layers of learning signal. The main fixes are better activations (ReLU/GELU), residual connections, batch normalization, and careful weight initialization.