What's the difference between feature selection and dimensionality reduction like PCA?

How to think about it

The crisp answer

Feature selection keeps some of the original columns and discards the rest. Dimensionality reduction, such as PCA, creates new columns by combining the originals. Both can turn 42 input columns into 5, but feature selection leaves you with named business variables while PCA leaves you with components.

Why the distinction matters

The interviewer is testing whether you understand what happens to the data, not whether you can recite two definitions.

A feature is one input variable, such as annual income or number of late payments. If the input is a vector called x, feature selection might produce:

x_selected = [annual_income, debt_to_income, late_payments]

Those are still the original measurements. A model can report that its input included annual_income, and a lender can explain that choice to a reviewer.

PCA produces something different:

z_1 = 0.58 income + 0.57 credit_limit + 0.58 balance

z_1 is a new coordinate, or component, whose value is calculated from several original features. The coefficients are weights learned from the data. Component 1 is not “income,” “credit risk,” or any other naturally named quantity. It is a direction through the original feature space.

That difference controls interpretability. After feature selection, you can say, “The model uses income and debt-to-income ratio.” After PCA, you can say, “The model uses component 1, which has strong positive weights for income, credit limit, and balance.” The second statement is useful to a technical audience, but it is not the same as a plain-language explanation.

One important qualification: selecting a feature does not prove that the feature causes the outcome. If a selection method keeps income, it means income helped prediction under that data and model. It does not mean changing someone’s income would produce the predicted change.

Dimensionality reduction is the broad category. PCA is one dimensionality-reduction method. Other methods create compact representations in different ways, including random projections, matrix factorization, and neural autoencoders. Feature selection can also reduce the number of dimensions, but it does so by choosing existing dimensions rather than inventing new ones.

A concrete example

Imagine a lender has 100,000 historical applications and 42 numeric input columns. The columns include annual income, debt-to-income ratio, credit utilization, credit limit, account age, recent inquiries, and several payment-history aggregates. The target, the outcome being predicted, is whether the applicant defaulted.

Suppose the business wants a model that a credit analyst can discuss with a customer. A feature-selection approach might keep 8 columns and discard the other 34. The final model still sees variables such as:

• annual income
• debt-to-income ratio
• credit utilization
• late-payment count
• account age
• recent credit inquiries

The model is smaller, and its inputs remain recognizable. It may also be cheaper to collect data for future applications if the discarded fields are not needed elsewhere.

Now use PCA and keep 8 components. The model still receives 8 numbers, but each number may combine many of the 42 columns. A first component might represent overall account scale: high income, high credit limit, and high balance all contribute positive weights. Another might contrast account age with recent inquiries.

For a particular applicant, the model can calculate the 8 component values and make a prediction. But explaining the prediction requires tracing those component values back through their weights to the original columns. That is possible, but it is less direct and easier to misunderstand.

A typical scikit-learn pipeline for the PCA version might look like this:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

pca_model = make_pipeline(
    StandardScaler(),
    PCA(n_components=8),
    LogisticRegression(max_iter=1000)
)

pca_model.fit(X_train, y_train)

The StandardScaler puts numeric columns on comparable scales. PCA then converts the 42 scaled columns into 8 components, and logistic regression uses those components. The pipeline matters because the scaler and PCA must be fitted using training data, not the test data.

What PCA is actually doing

PCA first centers each numeric feature by subtracting its training-set mean. It often also scales each feature by its standard deviation, which is a measure of its typical spread.

It then finds a direction that captures as much variance as possible. Variance here means spread: how much the observations differ along that direction. The first component captures the greatest spread. The second captures the greatest remaining spread while being mathematically independent in direction from the first, and so on.

If the first 8 components capture 95 percent of the variance, PCA has kept directions that preserve most of the variation in the input data. It has discarded the remaining directions.

That last sentence is where many answers go wrong. PCA preserves variance, not necessarily predictive signal. PCA does not look at the default label in the example. It does not know which direction helps distinguish defaulters from non-defaulters. A low-variance direction can still be highly predictive.

PCA is especially useful when several numeric features carry overlapping information. If credit limit and balance move together, PCA can combine them into a smaller number of coordinates. The retained components are uncorrelated with one another under the covariance structure used to fit PCA. They are not necessarily independent, and they are not automatically better features.

How feature selection chooses columns

Feature selection has several families of methods:

• Filter methods score features before fitting the final model. Examples include removing constant columns, examining correlation, or measuring how strongly each feature relates to the target. They are fast, but a feature that looks weak alone may become useful alongside another feature.
• Wrapper methods try different subsets and evaluate a model for each one. They can find useful combinations, but testing many subsets becomes expensive as the number of features grows.
• Embedded methods select features during model fitting. For example, an L1 penalty encourages some linear-model weights to become exactly zero. Tree-based models can also provide importance scores, though those scores are not a universal measure of truth.

Selection is therefore not one algorithm with one definitive answer. The chosen subset depends on the model, the evaluation metric, the data sample, and whether the target was used during selection.

The practical trade-off

Question	Feature selection	PCA
What comes out?	Original columns	New component columns
Can it use the target?	Yes, depending on method	Standard PCA does not
Handles correlated features by	Keeping or dropping them	Combining them
Interpretation	Direct	Indirect through component weights
Preserves sparse data?	Usually	Often produces dense data
Best fit	Explainable, operational inputs	Compact numeric representation

Choose feature selection when feature names matter. This includes regulated decisions, models reviewed by domain experts, debugging, and systems where unused inputs cost money or create data-governance work.

Choose PCA when you have many correlated numeric features and the downstream model benefits from a compact representation. Distance-based models such as nearest neighbors, and some linear models, can benefit when redundant variables make the geometry difficult. PCA can also reduce storage and computation.

Do not assume PCA improves every model. A tree model can often ignore irrelevant columns on its own, and rotating the data into dense components may make its splits harder to use. With a modest number of columns, regularization may solve the problem more simply.

The nuance that earns the senior signal

PCA is sensitive to scale. An annual-income column measured in dollars may have much larger numerical variance than a credit-utilization column measured between zero and one. Without scaling, PCA can give income more influence merely because of its units. That does not mean income is more predictive. It means the numbers are measured on a larger ruler.

The choice is not automatic, though. If all features have the same units and their absolute scale is meaningful, scaling may not be appropriate. The decision should follow the data-generating process, not a ritual.

PCA is also sensitive to extreme observations. One unusually large income or balance can affect the estimated variance and therefore the component directions. Inspect unusual values and compare the result with a sensible preprocessing strategy.

For sparse data, meaning data represented mostly by zeros, PCA may create a dense matrix and increase memory use. This matters for text data with tens of thousands of word columns. A sparse-friendly method such as truncated singular value decomposition is often considered instead.

You can combine the approaches. For example, first remove constant, unavailable, or clearly invalid columns, then apply PCA to the survivors. That can reduce junk before compression. But if the final model uses PCA components, the result is still not directly interpretable as a model using the selected original features. A two-stage pipeline does not magically preserve explainability.

Finally, both methods must be fitted inside the training portion of each validation split. If you use the full dataset to select features, the selection step has already seen the validation labels. If you use the full dataset to fit PCA, the components contain information about the validation distribution. The resulting score can look reassuring while measuring a process that will not exist in production.

Common failure modes

A common symptom of leakage is an unusually strong validation score followed by a disappointing production result. The fix is to put scaling, feature selection, and PCA inside one pipeline and fit that pipeline separately in each training split.

Another symptom is excellent PCA reconstruction but worse prediction. Reconstruction error measures how well the compressed components can approximate the original inputs. It does not measure how well they preserve the target signal. A model can retain 95 percent of input variance and still discard a small direction that carries most of the useful information about fraud or default.

A third symptom is a first principal component dominated by one dollar-valued column. That usually points to inconsistent scaling rather than a profound discovery about the business.

What they’ll ask next

Can I use feature selection and PCA together?

Yes. You might remove unusable or obviously redundant columns first and then apply PCA. The final representation is still made of components, so this is appropriate when compactness matters more than named inputs. If explanations are required, keep a separate interpretable model or use feature selection alone.

Does PCA always improve model accuracy?

No. PCA may help when correlated numeric inputs make a downstream model unstable or unnecessarily expensive. It may hurt when predictive information lies in low-variance directions, when the dataset is small, or when the model already handles irrelevant features well. Compare the original model, a selected-feature model, and a PCA model using the same validation procedure.

Should I standardize before PCA?

Usually, when features use different units or have very different scales. Otherwise, large numerical units can dominate the variance calculation. Fit the scaler only on the training data, and make the scaling decision based on the meaning of the measurements rather than applying it blindly.

Say this in the interview: “Feature selection chooses a smaller set of the original columns, while PCA creates new columns by combining them; I use selection for interpretability and PCA for compact, correlated numeric data when named features are not required.”