Quadratic Forms & Definiteness

The last lesson kept running into symmetric matrices and promised a payoff: feed a direction x into the expression xᵀAx and one number pops out — a warped “size” of x in that direction.

A symmetric matrix hides a whole landscape this way. Plug in a direction, read off a height; sweep the direction around and a surface emerges:

• maybe a bowl
• maybe an upside-down bowl
• maybe a saddle

That expression is a quadratic form, and its shape is set entirely by the eigenvalues of A. Reading the curvature off them is what definiteness means — the engine behind PCA and behind every “is this minimum really a minimum?” in optimisation.

The form and its curvature

For a symmetric A, the quadratic form is the scalar xᵀAx. With A = [[a, b], [b, c]] in two dimensions it expands to a pure degree-2 expression:

xᵀAx = a·x₁² + 2b·x₁x₂ + c·x₂²

Run that on real entries once, so the symbols have somewhere to land. Take A = [[6, 2], [2, 6]], so a = 6, b = 2, c = 6, and the form is 6x₁² + 4x₁x₂ + 6x₂². At x = (1, 1)/√2 each of x₁², x₁x₂, x₂² equals ½, giving 3 + 2 + 3 = 8.

At x = (1, −1)/√2 only the cross term flips sign, giving 3 − 2 + 3 = 4. Two unit directions, two very different heights on the same surface. Hold on to those numbers, because 8 and 4 are precisely the eigenvalues of A.

The geometry of the whole surface is fixed by the signs of the eigenvalues:

Definiteness names the form by those signs (always with A symmetric):

• Positive definite: xᵀAx > 0 for all x ≠ 0 ⇔ all eigenvalues > 0 (a bowl).
• Positive semidefinite (PSD): xᵀAx ≥ 0 for all x ⇔ all eigenvalues ≥ 0.
• Negative definite / semidefinite: flip the inequalities (all < 0 / ≤ 0).
• Indefinite: eigenvalues of both signs — a saddle.

The prefix semi deserves a moment, because it is not a hedge: semidefinite is definite plus permission to hit zero. Positive definite means no non-zero direction is ever flattened, so A is invertible.

Positive semidefinite allows some v ≠ 0 with vᵀAv = 0. Such a v is an eigenvector with eigenvalue 0, which makes A singular. One zero eigenvalue is the entire difference between the two words.

A Gram matrix XᵀX is always PSD, because xᵀ(XᵀX)x = (Xx)ᵀ(Xx) = ‖Xx‖² ≥ 0 for every x. It is never guaranteed definite, for exactly the reason above: if X has a non-trivial null space, some x ≠ 0 gives ‖Xx‖² = 0. Covariance matrices are Gram-like, which is exactly why they come out PSD.

The optimisation fact behind PCA

Here is the result GATE tests most. Constrain x to the unit sphere (‖x‖ = 1) and ask how large xᵀAx can grow. Diagonalise A = QΛQᵀ and write x in the orthonormal eigenbasis with coordinates y = Qᵀx (so ‖y‖ = 1):

xᵀAx = λ₁y₁² + λ₂y₂² + … + λₙyₙ²,   with  y₁² + … + yₙ² = 1

That is a weighted average of the eigenvalues, weights yᵢ² summing to 1. A weighted average is biggest when all the weight piles onto the largest eigenvalue, smallest when it piles onto the smallest:

So max₍‖x‖=1₎ xᵀAx = λ_max (at the top eigenvector) and the minimum is λ_min. PCA is exactly this: the first principal direction maximises the variance xᵀΣx on the unit sphere, so it is the top eigenvector of the covariance Σ.

Reshape the cloud below. The longest arrow, the direction of maximum spread, always lands along the top eigenvector. The variance along it is the maximum of xᵀΣx.

A worked example — the 2026 maximisation question

Let A be symmetric with eigenvalues 5 and 2. Find the maximum and minimum of xᵀAx subject to ‖x‖ = 1.

Write x in A’s orthonormal eigenvectors, coordinates y₁, y₂ with y₁² + y₂² = 1, then substitute y₂² = 1 − y₁²:

xᵀAx = 5·y₁² + 2·y₂² = 5y₁² + 2(1 − y₁²) = 2 + 3y₁²

This grows with y₁², largest (= 1) at the eigenvector for λ = 5 → maximum 5. It is smallest (y₁² = 0) at the eigenvector for λ = 2 → minimum 2. The constrained range is exactly [2, 5], smallest to largest eigenvalue — no calculus, no matrix entries.

A question to carry forward

To prove this we leaned on writing A as a product, A = QΛQᵀ — a decomposition that laid its action bare. Factoring a matrix into a useful product is a recurring trick.

Here is the thread onward: the row-elimination you ran back in lesson four can itself be packaged as a product of two triangular matrices, so that the expensive part is done once and reused. What is that factorisation, and why would you ever want it?

In one breath

• A quadratic form xᵀAx (symmetric A) is a warped “size” — sweep x and you trace a bowl, saddle, or dome set by eigenvalue signs.
• Definiteness: positive definite ⇔ all λ > 0; PSD ⇔ all λ ≥ 0; indefinite ⇔ mixed signs (saddle).
• Gram matrix XᵀX is always PSD (xᵀXᵀXx = ‖Xx‖² ≥ 0) — why covariance matrices are PSD.
• On ‖x‖ = 1: max xᵀAx = λ_max, min = λ_min (at the matching eigenvectors). PCA’s top direction = top eigenvector of Σ.
• Trap: it’s the eigenvalues, not the entries — [[2,−1],[−1,2]] is positive definite despite a negative entry.

Practice