PCA & Dimensionality Reduction
Rotate to the axes of maximum variance: PCA reads off the eigenvectors of the covariance matrix as new orthogonal components, keeping the top few to shrink dimensions.
What you'll learn
- PCA finds orthogonal directions (principal components) of maximum variance
- The recipe: centre the data, form the covariance matrix, take its eigenvectors and eigenvalues
- Variance explained by a component = its eigenvalue divided by the sum of all eigenvalues
- Principal components are mutually orthogonal; PCA is unsupervised (unlike LDA)
Before you start
Last lesson left clustering stranded in high dimensions, where distances blur and no plot can show you the data — and pointed at the cure: stop shrinking the rows and start shrinking the columns.
A table with fifty features is hard to see and slow to model, yet the data often really lives along just two or three directions, the rest being noise or redundancy. PCA (Principal Component Analysis) finds those directions. And it is, at last, the very method LDA was forever measured against.
PCA rotates the axes so the first new axis points along the direction of maximum variance — the way the cloud is most spread out — the next along the most variance left over, perpendicular to the first, and so on. Keep the first few of these axes and you have fewer dimensions carrying almost all the information.
Where LDA used the labels to pull classes apart, PCA ignores labels entirely and chases raw spread; that single difference is the whole contrast the earlier lesson promised.
From covariance to components
PCA’s new axes are the principal components, and the recipe to find them is short:
- Centre the data — subtract the mean of each feature (usually standardise too).
- Form the covariance matrix of the features. It is the square table whose
(i, j)entry records how strongly featureiand featurejmove together, with each feature’s own variance sitting down the diagonal. - Take its eigenvectors (these are the principal components, the new axis directions) and eigenvalues (the variance along each component).
Two facts fall straight out of this, and they are exactly what GATE tests:
- The components are mutually orthogonal — the covariance matrix is symmetric, so its eigenvectors are perpendicular. PC1 ⟂ PC2 ⟂ PC3 …, every pair at 90°.
- The variance explained by a component is its eigenvalue as a fraction of the total:
Components are not selected features
One picture is worth fixing before it hardens into the wrong one: PCA does not pick the best features — it builds new ones. A principal component is an eigenvector, which means it is a direction, and a direction in feature space is a weighted blend of every original feature at once.
Reduce fifty columns down to two and the two you keep are not two of the original fifty; they are two mixtures, each drawing a little from all fifty. That is why PCA is simultaneously so effective and so awkward to explain to a stakeholder — “component 1” has no name, only a recipe.
Choosing an actual subset of the original columns is a different job entirely, called feature selection, and PCA is not it.
Drag the points below: PC1 (the long axis) and PC2 (the short, perpendicular one) re-fit live, and the panel shows each component’s explained-variance share.
Drag points — watch the principal axes re-fit live
How GATE asks this
There are two recurring shapes. An MCQ on the geometry: because components are orthonormal, the angle between any two of them is 90° — GATE DA 2026 asked precisely this.
And a NAT on variance explained: given the eigenvalues, compute the fraction one component captures. The arithmetic is always eigenvalue over the sum.
Worked example — real GATE DA questions
(1) Orthogonality — a real GATE DA 2026 question. Principal components are orthonormal, so any two distinct components are perpendicular. The angle between PC1 and PC10 is therefore 90° — no calculation needed; it follows from orthogonality alone.
(2) Variance explained. Suppose the covariance matrix has eigenvalues [12, 3, 1]. The fraction of total variance captured by the first component is its eigenvalue over the sum:
total variance = 12 + 3 + 1 = 16
fraction for PC1 = 12 / 16 = 0.75 → 75%So PC1 alone explains 0.75 (75%) of the variance — keeping just that one component retains three-quarters of the spread, exactly as the lopsided eigenvalues suggested.
In one breath
PCA shrinks dimensions by rotating to new orthogonal axes of greatest variance. Centre the data, form the covariance matrix, and take its eigenvectors (the principal components, every pair at 90° because the matrix is symmetric) and eigenvalues (the variance along each).
The variance explained by a component is λᵢ / Σλ. Keeping the top-k components retains most of the spread in fewer dimensions. PCA is unsupervised, using only feature variance and ignoring the labels that its supervised rival LDA exploits.
Practice
Quick check
A question to carry forward
That closes the machine-learning chapter — and it is worth seeing what every single method in it shared. Regression, the classifiers, the neural net, clustering, PCA: each one learned from data. Hand it enough examples and it generalised, fitting weights or finding structure that the data itself revealed. The data was always the teacher.
But picture a problem with no data to learn from at all:
- Solve this maze.
- Win this game of chess.
- Find the cheapest flights from Delhi to Lisbon.
There is no dataset of “solved mazes” to train on; there are only rules and a goal, and the answer must be reasoned out by exploring possibilities, not generalised from examples. This is the other great branch of artificial intelligence, and the next chapter opens it.
Here is the thread onward: how do you turn a maze, a puzzle, a route map, or a game into something a computer can explore systematically? What are the handful of pieces every such problem reduces to, and what does the space of all possibilities look like once you lay it out?
Practice this in an interview
All questionsPCA centers data and finds orthogonal directions that maximize variance, usually through the covariance matrix's eigenvectors or an SVD, then projects observations onto the leading directions. Choose the component count using cumulative explained variance or reconstruction needs for compression, and cross-validated downstream performance for prediction; standardize first only when feature scales should contribute equally.
PCA finds the orthogonal directions of maximum variance in the data and projects onto a lower-dimensional subspace, reducing features while retaining most information. It is most useful before distance-based models or when training is bottlenecked by dimensionality. Its main limits are loss of interpretability, sensitivity to scale, and an assumption of linear structure.
PCA is an unsupervised linear projection that preserves high-variance directions, not necessarily target-predictive directions. It can hurt when the signal is low-variance or nonlinear, scaling or outliers dominate, too many components are dropped, or component mixing damages the model or its interpretability; preprocessing must be fit on training data only.
Feature selection keeps a subset of the original features, while dimensionality reduction such as PCA creates new features by combining the originals. Use selection when named, explainable inputs matter, and PCA when compactness and handling correlated numeric data matter more than direct interpretability.