What are the assumptions and limitations of PCA, and when would it hurt your model?

How to think about it

PCA can absolutely hurt a model. It is an unsupervised, linear projection that preserves directions of greatest feature variance; that helps when high variance represents useful structure, but hurts when the target signal is low-variance, nonlinear, outlier-driven, or hidden by the feature mixing.

The mechanism an interviewer is probing

PCA, or principal component analysis, is not a predictive model. It does not look at the target y. It looks only at the input features X and tries to compress them while losing as little squared reconstruction information as possible.

A principal component is a new feature formed as a weighted combination of the original features. PCA first centers each feature, meaning it subtracts that feature’s training-set mean. It then chooses the first component as the unit-length direction w that maximizes the variance of the projected values Xw. The second component captures as much remaining variance as possible while being orthogonal, meaning perpendicular under the ordinary dot product, to the first. The process continues.

The result is a rotation of the feature space. Keeping only the first k components, where k is smaller than the original number of features, also makes it a compression step.

That objective explains both the appeal and the danger:

• If ten features measure nearly the same underlying quantity, PCA can replace them with one component and remove redundancy.
• If a noisy direction has the largest spread, PCA preserves the noise.
• If a small-variance direction predicts the target, PCA may discard it.
• If the useful relationship is curved rather than straight, ordinary PCA cannot represent it well.

The key distinction is between reconstruction quality and prediction quality. A model that retains 95 percent of the input variance has preserved most of the geometry needed to approximately rebuild the input. It has not necessarily preserved 95 percent of the information needed to classify a customer, detect fraud, or forecast demand.

What the usual “assumptions” really mean

Interview shorthand	What PCA actually imposes	How it can fail
Relationships are linear	Each component is a linear combination of the input features	Curved or manifold-shaped structure is compressed poorly
High variance means importance	The optimization explicitly rewards variance	Predictive signal can live in a low-variance direction
Components are orthogonal	Each new direction must be perpendicular to earlier directions	Real latent factors may not be naturally perpendicular
Feature scale is meaningful	Variance is measured in the units supplied	A dollar-valued column can dominate a small-scale feature
Outliers are not dominant	Covariance uses squared deviations	One extreme observation can rotate the components

These are mostly practical conditions, not all formal statistical assumptions. PCA does not require normally distributed data for the standard linear algebra calculation. It also does not require independent features. In fact, correlated features are often exactly why PCA is useful.

Orthogonality is another common source of imprecise answers. PCA does not discover that the real world contains perpendicular factors. It imposes perpendicular directions because that gives a convenient, non-redundant basis and the best linear reconstruction under its objective. If the underlying factors are non-orthogonal, PCA can still reconstruct the data when enough components are retained, but the resulting components may be awkward to interpret.

Common misconception: “PCA assumes normality” is not a good default answer. Normality matters for some probabilistic interpretations of PCA, not for ordinary PCA as a deterministic transformation.

A concrete example: the 99.7 percent trap

Suppose a fraud team has 10,000 transactions, including 200 fraudulent ones. Two numerical features, a and b, measure closely related account behaviour.

For legitimate transactions, imagine:

• A shared activity variable z has standard deviation 10.
• A small mismatch variable e has standard deviation 0.5.
• The features are a = z + e and b = z - e.

The two useful PCA directions are approximately:

• u = (a + b) / sqrt(2), which measures shared activity.
• v = (a - b) / sqrt(2), which measures mismatch.

The standard deviation of u is about 14.14. The standard deviation of v for legitimate transactions is about 0.71. So almost all the variance lies along the shared-activity direction.

Now suppose fraud shifts e upward by 2. That subtle mismatch is highly predictive, even though it contributes very little to total variance. With 2 percent fraud, the first component still explains roughly 99.7 percent of the total variance in this simplified example.

If the team keeps one component because it exceeds a 95 percent explained-variance threshold, it keeps u and discards v. The classifier sees normal account activity but loses the billing-behaviour mismatch that separates fraud.

This is the central PCA failure mode: the largest direction in X is not necessarily the most useful direction for predicting y.

A model trained on both raw features can calculate the difference between a and b. A model trained only on the first principal component cannot recover that difference, because the information was removed. The explained-variance report can look excellent while fraud recall quietly collapses.

Where PCA hurts in practice

1. The target signal is low variance

This is the most important limitation. PCA is unsupervised, so it has no reason to preserve a feature combination that separates classes if that combination does not account for much overall spread.

For supervised learning, choose the number of components with cross-validated model performance, not explained variance alone. A curve showing 99 percent reconstruction variance is evidence about compression, not evidence that the classifier will retain its recall.

Supervised feature selection, regularized models, or a supervised projection such as partial least squares may be better when the target is the priority.

2. Relationships are nonlinear

PCA finds straight axes. It cannot unwrap a spiral, flatten a curved manifold, or represent a class boundary that depends on a radius or another nonlinear interaction.

Kernel PCA and autoencoders can learn nonlinear representations, although they add hyperparameters and can be harder to validate. A nonlinear predictive model may be the simpler answer: use a tree ensemble or neural network directly rather than compressing first. UMAP and t-SNE are useful for visualisation, but they should not be treated as automatic production feature transformers because their geometry is not designed to preserve every predictive relationship.

3. Scaling changes the answer

PCA operates on variance, and variance depends on units. A transaction amount might have a standard deviation of 75 dollars, while a purchase-count feature might have a standard deviation of 2. On raw values, the amount contributes far more numerical variance. That may be appropriate if the units reflect genuine importance. It may also mean that the count is ignored simply because it was measured on a smaller scale.

Standardization, which subtracts the mean and divides by the standard deviation, puts features on comparable scales. But it is not automatically correct. It can give a noisy, rare feature the same initial weight as a stable, meaningful one. The choice should follow the measurement semantics and validation results.

4. Outliers dominate covariance

Covariance squares deviations from the mean. One transaction worth 1,000,000 dollars can influence the components far more than thousands of ordinary transactions worth 20 to 200 dollars.

The first symptom is often a component whose loadings are surprising: it is mostly the column containing a handful of extreme values. A second symptom is unstable components across folds or across monthly retrains.

Inspect the records first. An extreme value may be a data error, but it may also be the fraud signal. Depending on the domain, a log transformation, a justified cap, robust preprocessing, or a method designed for outliers may help. Blindly deleting outliers is a fine way to delete the business problem.

5. Interpretability disappears into mixtures

A component may combine age, income, region, account history, and product usage. Its loading is the weight assigned to an original feature in that combination. Loadings can be inspected, but “component 3 increased” is usually less useful to an analyst than “late payments increased.”

PCA also does not provide a causal explanation. A large loading means a feature contributes to the direction, not that the feature caused the prediction. If an auditor needs feature-level reasoning, raw features, supervised feature selection, or a sparse representation may be preferable.

6. The downstream model may not benefit

PCA often helps distance-based models, linear models with many correlated inputs, and systems where memory or inference cost matters. It is less predictably useful for decision trees. A tree can split directly on an original feature, but after PCA each component mixes many features, so an originally simple axis-aligned rule may require several awkward splits.

PCA is also not automatically regularization. If all nonzero components are retained, ordinary PCA is essentially an invertible change of coordinates, so it has not removed information. Dimension reduction begins when components are dropped. Different penalties can then behave differently: an L1 penalty is not invariant to rotating the features, and interpretability changes even when predictive performance does not.

The safe production pattern

Fit every learned preprocessing step on the training data only. That includes the mean used for centering, the scale used for standardization, and the PCA directions themselves.

from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

model = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95, svd_solver="full")),
    ("classifier", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)

Here, n_components=0.95 asks PCA to retain enough components to explain 95 percent of the feature variance. It does not ask for 95 percent classification performance. In a real project, I would treat that value as a starting point and tune the component count against the business metric, such as fraud recall at a fixed false-positive rate.

Putting the steps in a pipeline also prevents a common leakage mistake during cross-validation. If PCA is fit once on the entire dataset before the split, validation rows influence the mean, scale, and component directions. The first symptom may be an implausibly good validation score that falls on genuinely future data. The fix is to fit the complete pipeline separately inside each training fold.

For sparse text features, ordinary centered PCA is often a poor fit because centering a sparse matrix can destroy its sparsity. Truncated SVD is commonly used instead because it can work without centering. That is a different numerical operation, so the choice should still be validated rather than swapped in by reflex.

For a deeper treatment of the decomposition and reconstruction objective, see PCA and dimensionality reduction.

What they’ll ask next

How do you choose the number of components?
Start with explained variance to understand the compression trade-off, then choose the number using cross-validation on the actual downstream metric. Also consider inference latency, memory, stability across time, and whether the resulting representation remains usable to the team.

How do you prevent leakage when using PCA?
Split first. Fit scaling and PCA on each training fold only, then transform the validation or test fold with those fitted parameters. A pipeline is the safest implementation pattern. For time-dependent data, use a time-aware split rather than allowing future rows to influence the transformation.

What would you use for nonlinear structure or sparse text?
For nonlinear structure, consider a nonlinear model, kernel PCA, or an autoencoder, depending on the operational constraints. For sparse text matrices, consider truncated SVD. For visual exploration, UMAP or t-SNE may help, but I would not use either as a production feature transformation without task-specific validation.

Say this in the interview: PCA is a linear, unsupervised compression method, so it can hurt whenever high variance is not the same as predictive signal, especially with low-variance targets, nonlinear structure, outliers, bad scaling, or an interpretability requirement.