PCA & dimensionality reduction

Your model has 500 columns. Many are cousins: monthly_spend and annual_spend, five temperature sensors bolted to the same pipe, or one-hot columns that mostly repeat the same fact. Training takes longer. Distances become less useful. A two-dimensional plot has become a 500-dimensional argument that nobody can see.

Worse, the raw columns may not be the shape the data naturally occupies. A customer’s income and spending often rise together, so the observations form a long, thin cloud rather than filling a rectangle. The useful variation is mostly along one slanted direction.

PCA, or principal component analysis, rotates that cloud onto new axes, orders those axes by how much the data varies along them, and lets you keep only the first few. It is compression by changing coordinates, not by deleting a particular named column.

The idea: find the long directions

Use one small example. An analyst records the floor area and bedroom count for five apartments:

The two columns are related. Larger apartments tend to have more bedrooms. Plot the rows and you get a stretched cloud running from the lower left to the upper right.

PCA asks a very specific question:

If I draw a line through this cloud, which direction makes the projected points spread out the most?

That line is the first principal component, or PC1. A component is a new axis, represented by a weighted combination of the original features. PC1 is not “the area feature.” It is a direction that might say, roughly, “more area and more bedrooms.”

The second principal component, PC2, must be perpendicular to PC1. Among all perpendicular directions, it captures the most remaining variation. Further components repeat the same idea. Each one explains less variance than the previous one.

Here, “variance” means squared spread around the average. PCA does not know that the rows are apartments, and it does not know a target such as rent. It only sees the geometry of the input columns.

The mechanism underneath

Before finding directions, PCA centers each feature: it subtracts that feature’s mean from every row. Without centering, the origin’s arbitrary location can make “far from zero” look like variation. Scikit-learn’s PCA centers by default, but it does not standardize the units.

For the apartment data, the column means are 60 square metres and 1.8 bedrooms. Apartment A therefore becomes (-20, -0.8) after centering, while apartment E becomes (20, 1.2).

The centered data has a covariance matrix, a table describing how each feature varies on its own and how pairs of features move together. Using sample covariance, the matrix is approximately

[ 250.0   12.5 ]
[  12.5    0.7 ]

The top-left value says that area has variance 250. The bottom-right says bedroom count has variance 0.7. The positive off-diagonal value says that apartments above the average area also tend to be above the average bedroom count.

PCA finds the eigenvectors of this matrix. An eigenvector is a direction that the covariance matrix can stretch without changing its orientation. Its matching eigenvalue is the amount of variance along that direction.

For this matrix, the eigenvalues are approximately 250.625 and 0.075. Their sum is 250.7, which is the total variance across both features. PC1 therefore explains

250.625 / 250.7 = 0.9997, or 99.97%

of the total variance. Its direction is approximately (0.999, 0.050): mostly area, with a small bedroom contribution. PC2 is the perpendicular direction, approximately (-0.050, 0.999). It captures the tiny residual pattern: rows with unusually many or unusually few bedrooms for their area.

Projecting a centered row onto a component means taking a dot product. For a row x and a unit component direction w, the new coordinate is x · w. That coordinate is called a score. Every apartment now has a PC1 score and a PC2 score instead of an area value and a bedroom value.

If you keep only PC1, you store one number per apartment. You have reduced two dimensions to one. The apartment cloud still lies almost where it did because PC2 contained only 0.03% of the total variance.

This is not merely a plausible-sounding visual trick. Among all possible one-dimensional linear projections, PC1 gives the smallest average squared reconstruction error when the projected data is mapped back into the original space. In plain English: if squared distance is your measure of information lost, the top k PCA directions are the best k linear directions for compression.

PCA is commonly computed with a singular value decomposition, or SVD, of the centered data rather than by explicitly constructing the covariance matrix. SVD is usually more numerically stable, especially when you have many features. The conceptual result is the same: orthogonal directions ordered by their captured variance.

One small detail causes needless debugging: the sign of a component is arbitrary. If one library reports PC1 as (0.7, 0.7) and another reports (-0.7, -0.7), they found the same axis. Every score merely changed sign. The direction matters; which end you call positive does not.

Scaling changes the question

Area is measured in square metres. Bedrooms are measured in counts. Their raw variances are therefore not directly comparable. In the example, area’s variance of 250 overwhelms bedroom variance of 0.7, so unscaled PCA mostly discovers area.

That may be exactly what you want. If your features are all measurements in the same unit and their absolute spread has meaning, preserving that spread is sensible.

Often it is not what you want. Standardization subtracts each feature’s mean and divides by its standard deviation, giving each column mean zero and variance one. The covariance matrix then behaves like a correlation matrix. For the apartment data, the area-bedroom correlation is about 0.945, so the standardized covariance matrix is approximately

[ 1.000   0.945 ]
[ 0.945   1.000 ]

Its eigenvalues are about 1.945 and 0.055. PC1 now explains about 97.25% of the standardized variance, and its direction is close to an equal blend of standardized area and standardized bedroom count.

Notice what happened. Standardization did not “make PCA correct.” It changed the question from “where is the largest spread in the original units?” to “where is the strongest pattern after giving each feature equal scale?” Those are different questions.

Choosing how many components

Each component has an explained-variance ratio: its eigenvalue divided by the sum of all eigenvalues. The cumulative ratio after k components tells you how much total input variance those k directions retain.

A target such as 95% is a compression budget, not a law of nature. If 95% takes you from 1,000 columns to 850, PCA has not delivered much compression. If it takes you to 30, the trade may be attractive. For a noisy sensor system, 90% might preserve the useful signal. For a reconstruction task, you may need 99.9%.

The missing percentage is also useful. With centered data and squared-error reconstruction, keeping 95% of the variance leaves 5% of the total squared energy outside the retained subspace. That does not mean every row loses 5%, and it says nothing directly about classification accuracy.

Here is a concrete inspection using scikit-learn’s 8 by 8 handwritten digits dataset. The 64 pixel columns are standardized, PCA reports the cumulative variance, and the code finds the smallest component count reaching 95%. The split comes first, so both learned transforms use only the training partition. This is a training-only inspection of input variance, not a final supervised choice of k.

import numpy as np

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)  # 1,797 rows, 64 pixel features
X_train, X_holdout, y_train, y_holdout = train_test_split(
    X, y, test_size=0.2, random_state=7, stratify=y
)

X_scaled = StandardScaler().fit_transform(X_train)
pca = PCA().fit(X_scaled)
cumulative = np.cumsum(pca.explained_variance_ratio_)

for k in [2, 10, 20, 30, 40]:
    print(f"{k:2d} components: {cumulative[k - 1] * 100:5.1f}%")

k95 = int(np.argmax(cumulative >= 0.95)) + 1
print(f"Smallest 95% representation: {k95} components")

Do not fit this scaler or PCA on all rows and then reuse the resulting k95 or transform in supervised evaluation. For selecting k for a supervised model, put scaling, PCA, and the estimator inside a cross-validation pipeline. That keeps each validation fold from influencing the preprocessing fit.

Two components are useful for a rough picture. They are not automatically enough to recognise every digit accurately. The 95% threshold gives a different answer because it is preserving global pixel variation, not optimising a digit classifier.

In a supervised task, evaluate the downstream objective as well. Fit several candidate values of k, measure validation performance, latency, memory, and reconstruction quality, then choose the smallest representation that meets the actual requirement. A variance threshold is a useful first sweep. It is not a substitute for validation.

The production pattern

PCA must be learned from the training set. Learning means calculating the means, standard deviations, covariance structure, and component directions. If validation or test rows help calculate any of those, information has crossed the evaluation boundary.

The safe pattern is a pipeline. The scaler and PCA are fitted inside it, so cross-validation gives each training fold its own preprocessing fit.

from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=7, stratify=y
)

model = make_pipeline(
    StandardScaler(),
    PCA(n_components=0.95, svd_solver="full"),
    LogisticRegression(max_iter=2000)
)

model.fit(X_train, y_train)
test_accuracy = model.score(X_test, y_test)
print(f"test accuracy: {test_accuracy:.3f}")

n_components=0.95 asks PCA to retain enough components for at least 95% of the variance in the training data. The actual number is learned during fitting. The test set is transformed using those same learned means, scales, and directions; it does not influence them.

A common first symptom of leakage is a validation score that looks suspiciously excellent, followed by a disappointing score on genuinely new data. The fix is not to lower the threshold. Put every learned preprocessing step inside the cross-validation pipeline. See train, test, and cross-validation and model selection for the evaluation boundary.

Where PCA helps

PCA is a good fit when several conditions line up:

• Features are correlated or redundant, and a linear low-dimensional shape is plausible.
• The downstream model is slowed by many columns, or suffers from noisy distances.
• You care about prediction, compression, denoising, or a compact visual summary more than naming each input.
• When drift justifies retraining, you must refit and version the entire scaler–PCA–model pipeline together; never replace PCA alone beneath an old estimator.

It can help linear models because correlated columns become orthogonal coordinates. It can reduce memory and matrix-operation cost. It can also remove small-variance measurement noise, provided the noise really is in the small-variance directions.

PCA is not automatically a better representation. A low-variance feature can carry the entire target signal. Imagine fraud affecting only 0.2% of transactions and appearing as a subtle pattern in one direction. That direction may be discarded by a 95% variance cutoff because ordinary customer behaviour produces far more spread.

PCA is also sensitive to outliers. Variance squares deviations, so one sensor reading 100 units from the mean can pull PC1 toward itself. The first symptom is often a component dominated by one row, followed by a sudden change in components when that row is removed. Inspect extreme rows, correct data quality problems, and consider a robust preprocessing strategy before fitting PCA. Do not hide a bad sensor inside a “denoising” step.

PCA is linear. A spiral, a curved manifold, or two clusters wrapped around each other may have no useful straight axis. For exploratory visualisation, t-SNE and UMAP can reveal local structure that PCA cannot, but their pictures are not interchangeable with a stable feature transform for a production model.

There is an interpretability cost too. PC1 might be 0.61 times income, minus 0.48 times debt, plus 0.33 times age, after scaling. That can be mathematically useful and politically awkward to explain. If the requirement is “which original columns should we keep?”, use feature selection instead. If the requirement is “how did this fitted model use the inputs?”, look at SHAP. PCA answers a different question: which combinations best describe the geometry of the inputs?

For sparse text or clickstream matrices, ordinary PCA is often a poor engineering choice because centering a sparse matrix makes it dense. A truncated SVD approach is commonly considered instead because it can work without explicitly centering the matrix. The right choice depends on the representation and the downstream model, not on the word “dimensionality.”

Failure modes worth catching

PC1 is just the biggest-numbered column. The first component has a huge loading on income, bytes, or a timestamp, while other columns barely appear. That is the symptom of incompatible scales. Decide whether raw scale is meaningful. If not, put StandardScaler before PCA and inspect the loadings.

Validation is strong; deployment is weak. This usually means leakage, especially a scaler or PCA fitted before the split. Refit preprocessing inside each training fold and persist the fitted pipeline, not just the classifier.

The model gets worse after “compression.” The retained components preserved input variance but discarded information useful for the target. Try larger values of k, compare against the uncompressed baseline, and select using the real validation metric. Also check whether the target relationship is nonlinear.

Components change dramatically between retrains. Check for outliers, changed feature distributions, and small eigenvalue gaps. When two eigenvalues are nearly equal, their individual directions are not strongly identified; the subspace may be stable even though PC2 and PC3 rotate within it. Track the transform and the downstream metric rather than demanding identical component columns.

PCA is a tool for a particular geometry. It is excellent when “most of the important structure” really does mean “most of the variance in a linear subspace.” That sentence contains the entire warranty.

In one breath

PCA centers data, finds orthogonal directions of maximum remaining variance, and expresses each row as scores along those directions. The directions are the covariance matrix’s eigenvectors, usually computed with SVD. Their eigenvalues determine explained variance. Keep the first k components when the resulting compression still serves your actual task. Standardize when raw units should not choose the directions, fit every preprocessing step on training data only, and remember that variance is not the same thing as predictive signal.

Next

For nonlinear visual structure, read t-SNE and UMAP. For choosing original columns rather than blended axes, use feature selection. If PCA is feeding a classifier, evaluate the whole transform-and-model pipeline with model selection.

Quick check