How does PCA work, and how do you choose the number of components?

How to think about it

The direct answer

Principal component analysis, or PCA, replaces correlated features with a smaller set of new, orthogonal features called principal components. It chooses the first component in the direction of greatest variance, the second in the greatest remaining direction, and so on, then projects the data onto the components you keep.

Choose the number using cumulative explained variance for compression, reconstruction error when information loss has a measurable cost, or cross-validated performance when PCA feeds a predictive model. Standardize features first when their units or ranges should have equal influence.

Why PCA works

Suppose a customer dataset has annual_income, monthly_spend, number_of_orders, and average_order_value. These four columns may contain less than four genuinely distinct kinds of information. Income and spending are correlated. Orders and average order value partly explain total spending. Feeding every column to a model can therefore add redundancy, memory use, and sometimes noise.

PCA rotates the coordinate system so that the new axes follow the important patterns in the data.

A linear combination is a weighted sum of the original features. For example, a component might roughly look like:

0.70 × standardized_income + 0.68 × standardized_spend + 0.12 × standardized_orders

That expression is not a selected original column. It is a new feature.

The first component points along the direction where observations spread out most. The second captures as much of the remaining spread as possible while being perpendicular to the first. This continues until every feature direction has been used.

The usual steps are:

1. Center each feature by subtracting its training-set mean.
2. Optionally scale each feature to a comparable standard deviation.
3. Compute the covariance matrix, or decompose the centered data directly with singular value decomposition.
4. Keep the leading directions.
5. Project each row onto those directions.

If there are p original features, the covariance matrix has p rows and p columns. Its diagonal contains feature variances. Its off-diagonal entries contain covariances, which measure whether two features tend to move together.

For a unit direction v, the variance of the projected data is vᵀ C v, where C is the covariance matrix. Maximizing that quantity leads to the eigenvector equation C v = λ v. The vector v is a principal direction, and its eigenvalue λ is the variance captured along that direction.

That is the mechanism behind the common interview answer: eigenvectors give the directions, and eigenvalues rank them.

In practice, software often uses SVD rather than explicitly forming the covariance matrix. If the centered data matrix is decomposed as X = U S Vᵀ, the rows of Vᵀ give the component directions. SVD is usually numerically safer, especially when features have very different scales or the matrix is large.

The transformed value for one row is its coordinate on a component. These coordinates are often called component scores. Keeping the first k scores gives a k-dimensional representation instead of the original p dimensions.

PCA is not merely deleting columns. It is rotating the space and then dropping directions. That distinction matters for interpretation.

A concrete example

Take four students with two measurements:

Student	Hours studied	Exam score
A	1	2
B	2	4
C	3	5
D	4	7

The means are 2.5 hours and 4.5 score points. After centering, the sample covariance matrix is approximately:

C = [[1.667, 2.667],
     [2.667, 4.333]]

Its eigenvalues are approximately 5.981 and 0.019. The total variance is their sum, 6. Therefore the first component explains about 99.7 percent of the variance.

The first direction is approximately (0.526, 0.851). It combines hours studied and exam score because those measurements move together. Student A receives a strongly negative first-component score, while Student D receives a strongly positive one. The four two-dimensional rows can therefore be represented almost perfectly by one coordinate.

If you standardize both columns first, the correlation is approximately 0.992. The component directions become roughly equal combinations of the two standardized variables, and the first component explains about 99.6 percent of the variance.

This example also exposes an important detail. PCA does not know that an exam score is more meaningful than study time, or that one feature is a possible cause of the other. It sees only numerical variation and relationships between columns.

How to choose the number of components

The most common measure is the explained variance ratio: the variance of one component divided by the total variance in all components. Add these ratios from the beginning to get the cumulative explained variance.

Suppose a fitted PCA reports the following:

Components kept	Individual variance	Cumulative variance
1	72%	72%
2	18%	90%
3	6%	96%
4	2%	98%
5	1%	99%
6	1%	100%

A 95 percent rule would keep three components. The remaining three dimensions contain only 4 percent of the variance.

That is a reasonable starting point for compression, but 95 percent is not a scientific constant. If a model must fit in a 50 MB memory budget, you choose the largest k that fits. If reconstructing an image with visible quality requirements, you inspect reconstruction error. If PCA feeds a classifier, you evaluate the classifier rather than trusting a variance threshold.

A practical calculation in scikit-learn looks like this:

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

pca = PCA()
X_train_pca = pca.fit_transform(X_train_scaled)

cumulative = pca.explained_variance_ratio_.cumsum()
k = (cumulative >= 0.95).argmax() + 1

X_valid_pca = pca.transform(scaler.transform(X_valid))

The k calculation finds the first component count whose cumulative ratio reaches 95 percent. In a real project, compare that choice with nearby values and with no PCA at all.

For a supervised task, put scaling and PCA inside the cross-validation pipeline. Try several values of k, such as 5, 10, 20, and 40, and select using validation performance and operational constraints. Do not select the value that happens to produce the best score on the final test set. That turns the test set into a training signal.

The nuance that earns the senior signal

Standardization is conditional, not automatic. PCA measures variance, and variance depends on units. If income is stored in dollars and then changed to cents, every value is multiplied by 100 and its variance becomes 10,000 times larger. The underlying people have not changed, but unscaled PCA now gives that feature far more influence.

Standardize when features such as income, age, counts, and distances should contribute on a comparable basis. Do not standardize blindly when absolute scale is itself meaningful and all variables already share a sensible unit. A process engineer may deliberately want a temperature variation of 20 degrees to outweigh a pressure variation of 2 units. That is a domain decision, not a PCA decision.

PCA is also sensitive to outliers because variance uses squared deviations. A single transaction recorded as 10 million instead of 10,000 can pull the first component toward the bad row. The first symptom is often a suspiciously dominant loading or a scatter plot where one point determines the apparent direction. Investigate, correct, transform, cap, or use a robust method when appropriate. Standardization alone does not make an outlier harmless.

PCA is unsupervised. It preserves variation in X, not predictive information about y. A low-variance feature can be the best fraud signal in the dataset. Conversely, a high-variance feature can reflect harmless customer size. For prediction, explained variance is only a diagnostic; validation performance is the decision criterion.

PCA also changes interpretability. A component may mix 30 original columns, so explaining “component 3 increased by 1.4” is harder than explaining “account age decreased.” If the requirement is to select a small set of original business features, use feature selection rather than PCA. PCA is feature extraction, not feature selection.

Finally, the transformed components are uncorrelated on the fitted data, but they are not necessarily statistically independent. Zero linear correlation does not mean that one component contains no nonlinear information about another.

For sparse text data, ordinary centered PCA can be a poor fit because subtracting column means makes a mostly-zero matrix dense. A truncated SVD approach is commonly used instead because it can work without centering the sparse matrix in the same way.

What they will ask next

Does PCA always require standardization?

No. PCA requires centering for the usual covariance-based interpretation, but scaling depends on the meaning of the units. Standardize when a feature with a larger numeric scale should not dominate merely because of that scale. Keep the original scale when the magnitudes are intentionally comparable and meaningful.

How would you choose components for a classifier?

I would fit scaling and PCA only on each training fold, test several component counts, and compare cross-validated performance against a model without PCA. I would also check latency, memory, calibration, and interpretability. The best k is the smallest one that preserves the required business performance, not automatically the one that preserves 95 percent of variance.

How is PCA different from t-SNE or UMAP?

PCA is a deterministic linear transformation designed to preserve global variance and support compression or downstream features. t-SNE and UMAP are nonlinear methods mainly used to visualize local neighborhood structure. A visually separated two-dimensional UMAP plot is not evidence that those coordinates are safe production features. For a stable, inexpensive baseline, start with PCA; use nonlinear visualization methods for the question they actually answer.

Say this in the interview: “PCA centers and usually scales the features, finds orthogonal directions of maximum remaining variance through eigenvectors or SVD, and projects the data onto the leading directions; I choose the component count by explained variance for compression, but by cross-validated downstream performance when prediction is the goal.”