PCA & dimensionality reduction

The curse of dimensionality says too many features hurt. PCA (principal component analysis) is the classic cure: it finds the handful of directions your data actually varies along and lets you throw the rest away — keeping most of the information in far fewer dimensions. It’s the most important unsupervised technique after clustering, and a guaranteed interview topic.

The idea: rotate to where the variance is

Real features are usually correlated, so the data doesn’t fill its space evenly — it stretches along certain directions. PCA finds those directions, called principal components:

• PC1 is the direction of maximum variance (the long axis of the cloud).
• PC2 is the direction of most remaining variance, orthogonal to PC1.
• …and so on, each capturing less.

Mathematically, the principal components are the eigenvectors of the covariance matrix, and each one’s eigenvalue is the variance along it. PCA is just a rotation onto those axes. See it — and project away the low-variance direction:

Once you’re on the principal axes, the low-variance components carry little information, so you keep the top k and drop the rest. That’s dimensionality reduction.

How many components to keep

You don’t guess — you read the explained-variance ratio. Each component explains some fraction of the total variance; sum them up and keep enough components to retain, say, 95%.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits

X, _ = load_digits(return_X_y=True)   # 64 features (8x8 pixel images)
Xs = StandardScaler().fit_transform(X)

pca = PCA().fit(Xs)
cum = np.cumsum(pca.explained_variance_ratio_)
for k in [2, 10, 20, 30, 40]:
    print(f"  {k:2d} components keep {cum[k-1]*100:5.1f}% of the variance")

# How many components to reach 95%?
k95 = int(np.argmax(cum >= 0.95)) + 1
print(f"\n{k95} of 64 components retain 95% of the information.")

   2 components keep  21.6% of the variance
  10 components keep  58.9% of the variance
  20 components keep  79.3% of the variance
  30 components keep  89.3% of the variance
  40 components keep  95.1% of the variance

40 of 64 components retain 95% of the information.

Two components already capture 21.6% — a usable thumbnail — but you need 40 of the 64 to preserve 95%. The digits genuinely live in many dimensions; the explained-variance ratio tells you exactly how many you can afford to drop.

What PCA is good (and bad) at

• Good: speeding up downstream models (fewer features), denoising (dropping low-variance components removes noise), decorrelating features, and compression.
• Visualization: projecting to 2 components to eyeball structure — though for seeing clusters, the nonlinear methods t-SNE and UMAP usually reveal more.
• The catch: PCA is linear — it can only find straight-line directions. If your structure is curved (a spiral), PCA misses it. And the components are combinations of all original features, so they’re less interpretable than the raw columns.

In one breath

• Correlated features make data stretch along a few directions; PCA finds them — the principal components, eigenvectors of the covariance matrix, ordered by variance.
• PC1 is the direction of maximum spread, PC2 the most remaining variance orthogonal to it, and so on.
• Keep the top k by reading the cumulative explained-variance ratio (e.g. enough for 95%) and drop the rest — that is the dimensionality reduction.
• Always standardize first, or PCA just chases your largest-scaled column.
• It’s linear and its components aren’t nameable — for curved structure use t-SNE/UMAP, and to know which feature matters use feature selection or SHAP.

Quick check

Next

For seeing clusters in high-dimensional data, the nonlinear projections t-SNE & UMAP often beat PCA. And to group what you’ve reduced, revisit k-means and DBSCAN.