Eigenvalues and eigenvectors

The last lesson closed with a question asked from the matrix’s own point of view: are there directions a transform simply refuses to turn? Here is the answer, and it is yes. Most vectors, pushed through a matrix, come out rotated and stretched — pointing somewhere new. But for any given matrix there are a special few directions where it only stretches the vector and leaves its heading untouched. Those directions are the eigenvectors, and the stretch factors are the eigenvalues.

It sounds abstract, and it is the single most useful idea you will meet for understanding PCA, covariance structure, PageRank, and the stability of any system that loops its own output back in as the next input.

The defining equation

A vector v is an eigenvector of matrix A if applying A only scales it:

A v = λ v

λ (lambda) is the scalar that says “by how much.” It can be positive (same direction, scaled), negative (flipped to point the other way), or zero (collapsed). The direction of v is unchanged.

Visually:

v came out longer but pointing the same way — it’s an eigenvector with λ = 2. w came out rotated — it isn’t.

Play with it. The fan of faint arrows shows where a matrix sends every input direction — most of them swing to a new angle, but the highlighted lines stay put and only scale by their eigenvalue λ. Drag the test vector onto one to feel a direction survive; switch to the rotation preset to watch the eigenlines disappear entirely.

Solving a 2×2 by hand (briefly)

For a small matrix you can find eigenvalues from det(A - λI) = 0.

Take A = [[2, 1], [0, 3]]. Then:

det( [ 2-λ   1  ] ) = (2-λ)(3-λ) - 0 = 0  →  λ = 2  or  λ = 3
     [  0   3-λ ]

For each λ, solve (A - λI) v = 0 to get the eigenvector direction. For λ = 2: the eigenvector is [1, 0]. For λ = 3: it’s [1, 1] (up to scale).

You’ll do this twice in your life by hand, and call np.linalg.eig for the rest. Knowing the shape of the calculation helps you understand what the library is returning.

np.linalg.eig

import numpy as np

A = np.array([[2.0, 1.0],
              [0.0, 3.0]])

eigvals, eigvecs = np.linalg.eig(A)

print("eigenvalues:", eigvals)
# Columns of eigvecs are the eigenvectors (one per eigenvalue)
print("eigenvectors (columns):")
print(eigvecs.round(3))

# Verify: A @ v should equal lam * v for each pair
for i, lam in enumerate(eigvals):
    v = eigvecs[:, i]
    print(f"lam={lam:.2f}  A@v={A @ v}   lam*v={lam * v}")

eigenvalues: [2. 3.]
eigenvectors (columns):
[[1.    0.707]
 [0.    0.707]]
lam=2.00  A@v=[2. 0.]   lam*v=[2. 0.]
lam=3.00  A@v=[2.12132034 2.12132034]   lam*v=[2.12132034 2.12132034]

The hand-computed answer (λ = 2 with [1, 0], λ = 3 with [1, 1]) matches — the library returns the λ = 3 eigenvector normalised to unit length, [0.707, 0.707], and the verify lines confirm A @ v equals λ · v exactly. Two important details:

1. Eigenvectors are columns of the returned matrix, not rows.
2. Eigenvectors are unit-length by convention. Any non-zero scalar multiple is still an eigenvector — the library picks one.

Where this lives in ML: covariance and variance directions

The big payoff is the covariance matrix. Given centered data X of shape (n, d), the covariance matrix is:

Σ = (1/n) Xᵀ X        # shape (d, d)

Σ describes how features vary with each other. Its eigenvectors are the directions of variance in your data. The eigenvalues tell you how much variance points in each direction.

Why the eigenvectors? Projecting the data onto a unit vector v gives variance equal to vᵀ Σ v. The directions that make this variance stationary (neither increasing nor decreasing as you rotate v) are exactly the eigenvectors of Σ, and the corresponding eigenvalue is the variance in that direction.

Sort eigenvalues largest to smallest, project onto the top-k eigenvectors, and you’ve done PCA.

import numpy as np

rng = np.random.default_rng(0)

# Generate 2D data that's elongated along the (1, 1) direction
n = 500
direction = np.array([1.0, 1.0]) / np.sqrt(2)
spread_along = rng.normal(0, 3.0, n)      # large variance along direction
spread_across = rng.normal(0, 0.5, n)     # small variance perpendicular
perp = np.array([-1.0, 1.0]) / np.sqrt(2)

X = spread_along[:, None] * direction + spread_across[:, None] * perp
X = X - X.mean(axis=0)                     # center

cov = (X.T @ X) / n                        # 2x2 covariance
print("Covariance matrix:")
print(cov.round(3))

vals, vecs = np.linalg.eig(cov)
order = np.argsort(-vals)                  # sort descending
vals = vals[order]
vecs = vecs[:, order]

print("\nEigenvalues (variances per direction):", vals.round(3))
print("Top eigenvector (direction of max variance):", vecs[:, 0].round(3))
print("(should be +/-[0.707, 0.707] -- the (1,1)/sqrt(2) we built in; sign is arbitrary)")

Covariance matrix:
[[4.726 4.513]
 [4.513 4.741]]

Eigenvalues (variances per direction): [9.246 0.22 ]
Top eigenvector (direction of max variance): [-0.707 -0.708]
(should be +/-[0.707, 0.707] -- the (1,1)/sqrt(2) we built in; sign is arbitrary)

The top eigenvector recovers the direction we baked into the data (its sign is arbitrary — an eigenvector and its negative point along the same line). The top eigenvalue, 9.25, is the variance along that direction; the tiny 0.22 is the variance across it. That is the whole algorithm — next lesson we wrap it up as PCA.

Other places eigenvalues show up

• PageRank. Google’s original ranking algorithm finds the top eigenvector of the web’s link matrix. Each entry of that eigenvector is a page’s rank.
• Spectral clustering. Use the eigenvectors of a graph’s Laplacian to find natural clusters.
• Stability of linear systems. A discrete-time system x_{t+1} = A x_t is stable when all eigenvalues of A have magnitude less than 1. Bigger than 1 and the system blows up. This explains why RNN training can explode or vanish based on the spectrum of the recurrent matrix.
• Determinant and trace. det(A) = product of eigenvalues, trace(A) = sum of eigenvalues. Handy sanity checks.

In one breath

An eigenvector of A is a direction the matrix only stretches, never turns: A v = λ v, where the eigenvalue λ is the stretch factor (positive keeps the direction, negative flips it, zero collapses it to nothing). You find them from det(A − λI) = 0 by hand, or call np.linalg.eig (eigenvectors come back as columns, unit-length, sign arbitrary). The payoff is the covariance matrix Σ = (1/n)XᵀX: its eigenvectors are the directions of variance in your data and its eigenvalues are how much variance lies along each — sort them, keep the top k, and you have done PCA. The same idea drives PageRank, spectral clustering, and the stability test |λ| < 1 for any system that feeds its output back in.

Practice

A question to carry forward

Eigenvectors handed us a matrix’s skeleton — the directions it merely scales — and we saw the prize: eigendecomposing a covariance matrix sorts a cloud of data into its directions of variance, which is very nearly PCA. But notice the quiet condition we leaned on the whole time. A v = λ v only makes sense when A is square: input and output live in the same space, so “came out pointing the same way” is even a question you can ask.

Your data is almost never square. A design matrix X is n × d — five hundred rows, three columns — rectangular, carrying one space into a different-sized one, and eigenvectors have nothing to say about it. Here is the thread onward: is there a decomposition that does for any rectangular matrix what eigendecomposition does for a square one — still finding special orthogonal input directions, special orthogonal output directions, and one set of stretch factors linking them? There is. It is the singular value decomposition, and it is the single factorization quietly underneath PCA, recommender systems, and least squares all at once.