Compare filter, wrapper, and embedded feature-selection methods. When would you use each?

How to think about it

The crisp answer

Feature selection means keeping some original input columns and discarding the rest. Filter methods score columns independently of the predictive model, wrapper methods search subsets by repeatedly training and evaluating a model, and embedded methods select features during model fitting. I use filters for a cheap first pass, wrappers when the candidate set is small and accuracy matters most, and embedded methods as the usual production default.

What differs mechanically

The dividing line is simple: when does selection happen, and what judges a feature?

A filter asks, “Does this column have a statistical relationship with the target on its own?” A wrapper asks, “Does this particular model perform better with this subset?” An embedded method lets the model answer that question while it is learning its parameters.

Here, p means the number of candidate features and n means the number of labelled examples. When p is tens of thousands and n is only a few thousand, the computational difference is not academic.

Filter methods

A filter scores features before fitting the final model. Common scores include:

• Variance threshold, which removes columns that barely change. It does not use the target.
• Correlation, which measures linear association between a numeric feature and the target.
• Chi-square, which tests association between nonnegative or categorical values and a class label.
• ANOVA F-test, which compares the average feature value across target groups.
• Mutual information, which measures statistical dependence and can detect some nonlinear relationships.

The filter keeps the top k features or those above a threshold. Its main advantage is speed: calculating thousands of univariate scores is far cheaper than training thousands of models.

The important limitation is that it sees features mostly one at a time. Two columns can each look useless but become powerful together. It also tends to keep redundant columns. If monthly_logins and weekly_logins both measure the same behaviour, both may receive excellent scores even though the final model needs only one.

Common trap: model-independent does not mean target-independent. Correlation, chi-square, ANOVA, and mutual information all use the target. They are safe only when calculated on the training portion of the data.

Wrapper methods

A wrapper puts a model inside a search procedure. The procedure proposes a subset, trains the model, measures its validation score, and proposes another subset.

Typical searches are:

• Forward selection, which starts with no features and adds them.
• Backward elimination, which starts with all features and removes them.
• Recursive feature elimination, or RFE, which repeatedly removes the least useful features according to the fitted model.

A wrapper can optimise the metric that actually matters. If the product cares about recall at a fixed false-positive rate, the wrapper can use that metric rather than a generic correlation score.

Its cost grows quickly. There are 2^p possible subsets, so exhaustive search becomes impossible even at modest p. Practical wrappers use greedy searches such as forward selection or RFE. They are still expensive because every candidate subset requires another model fit, and cross-validation multiplies the work.

A wrapper is not automatically interaction-aware. RFE using logistic regression can only exploit interactions that are explicitly provided as features. A wrapper using a random forest can capture nonlinear interactions because the underlying model can represent them. The search inherits the strengths and weaknesses of its model.

Embedded methods

An embedded method selects features as part of training. The model and the selector are not separate stages.

The classic example is logistic regression with an L1 penalty, also called lasso regularisation. The training objective is prediction loss plus lambda times the sum of the absolute coefficient values. The penalty has a sharp corner at zero, so some coefficients are driven exactly to zero. Those zero-coefficient columns are natural candidates for removal.

Elastic net combines L1 and L2 penalties. It is often more stable than pure lasso when several features are highly correlated, although it still does not guarantee a unique or causal selection.

Tree models provide another embedded example. At each split, a tree chooses a feature and threshold that improve the objective. A boosted-tree model therefore performs a kind of feature selection while building its trees. Its reported importance values can be used to create a smaller feature set, but that thresholding step is technically a second stage rather than the embedded training itself.

Embedded methods are usually a good compromise. They are model-aware without requiring a separate subset search. They are not free, though. You still need to tune the regularisation strength or tree complexity, check stability, and validate the resulting model.

A concrete example

Suppose we predict whether a customer will cancel a subscription in the next 30 days.

The dataset has 200,000 customer-month records, a 12 percent churn rate, and 5,000 candidate columns. Most are one-hot encoded plan, region, and support categories; the rest describe recent usage, billing events, and account age. The features are computed as of the prediction date. We train on the first eight months, validate on month nine, and keep month ten untouched as the final test period.

A sensible sequence might look like this:

1. A filter removes constant columns and keeps the best 500 candidates using mutual information and simple redundancy checks.
2. An RFE wrapper with logistic regression tries to reduce those 500 columns to 50 or 100.
3. An L1 logistic model is trained on the filtered set and keeps the 68 columns with nonzero coefficients.

An illustrative result could be:

Approach	Columns	Validation AUC	Future-test AUC
All features	5,000	0.842	0.839
Filtered set	500	0.844	0.841
RFE-selected set	70	0.846	0.840
L1-selected set	68	0.843	0.843

AUC, the area under the ROC curve, measures how often the model ranks a randomly chosen churner above a randomly chosen non-churner.

The wrapper looks best on validation but loses on the future test period. That is a classic sign that the selection process has started fitting quirks of the validation data. The embedded model is slightly less impressive in validation but generalises better and uses nearly the same number of columns.

The exact numbers are not universal. The lesson is the pattern: selection is part of model tuning, so a smaller feature set is not automatically a better model.

When I would use each

Use a filter when the feature space is huge, the first screen must be cheap, or you want a model-agnostic reduction. For example, screening 100,000 text or one-hot features down to 5,000 before model training can make every later experiment cheaper. Filters are also useful when several different model families will consume the same candidate set.

Use a wrapper when the candidate set is small enough to search and the final model and metric are already known. If there are 30 carefully engineered features and a model evaluation takes seconds, forward selection may be reasonable. It is especially useful when the business wants the best performance from a compact, fixed model and the extra compute is affordable.

Use an embedded method when the final estimator supports it. L1 or elastic-net linear models are natural choices for sparse, high-dimensional inputs. Tree-based models are useful when nonlinearities and interactions matter. Embedded selection usually aligns better with the model’s actual loss than a generic filter does.

In production, I often combine them: filter 5,000 columns down to 500, then use an embedded model to make the final choice. I reserve wrappers for the last, small search. If there are only 20 features and no latency, memory, or interpretability problem, I may skip selection entirely. Fewer columns can mean less signal and more selection variance.

Feature selection is also different from dimensionality reduction. PCA, for example, creates new columns that are combinations of the originals. It does not select original features.

Failure modes to watch for

The most damaging mistake is selecting features on the whole dataset before cross-validation. Suppose you calculate target correlations using all 200,000 customer-month records, choose the top 500, and only then run cross-validation. Every validation fold has already influenced which columns exist. The reported score is optimistic because the validation data helped design the model.

The first symptom may be a validation AUC of 0.91 followed by a future AUC of 0.78, along with a feature list that changes dramatically from fold to fold.

Put every target-dependent transformation inside the cross-validation pipeline:

from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
    ("select", SelectKBest(mutual_info_classif, k=500)),
    ("scale", StandardScaler(with_mean=False)),
    ("model", LogisticRegression(max_iter=2000, solver="liblinear")),
])

auc = cross_val_score(
    pipe,
    X_train,
    y_train,
    cv=5,
    scoring="roc_auc",
)

The selector is fitted separately inside each training fold. For time-ordered churn data, the splitter must also respect time; ordinary random folds can let later customer behaviour influence earlier predictions.

A second failure is using the wrong score. Correlation can miss a U-shaped relationship. A univariate filter can miss a useful feature pair. L1 selection can choose one of several correlated billing signals arbitrarily. None of these outcomes means the selected feature is important in a causal sense.

A third failure is ignoring scale before L1 regularisation. If one numeric column is measured in cents and another in dollars, their coefficient magnitudes are not comparable. Because the penalty acts on coefficients, inconsistent feature scales change which columns are shrunk toward zero. Scale numeric inputs using training data only, and preserve the exact selector, scaler, and column order at serving time.

For a rigorous performance estimate, use nested cross-validation when the dataset is small and you are tuning k, the regularisation strength, the search strategy, and the model at once. The inner loop chooses those settings; the outer loop estimates generalisation. Otherwise, the selection procedure gets to keep peeking at the same score.

What they’ll ask next

“Do filter methods always miss interactions?”

They miss interactions during their ordinary univariate scoring step. You can explicitly create an interaction feature, such as late_payment_count multiplied by support_ticket_count, and then filter that new column. But the filter will not discover arbitrary interactions by itself. A wrapper or embedded method can use interactions only if its underlying model can represent them.

“Why is RFE expensive?”

RFE repeatedly fits the estimator as it removes features. Reducing 500 columns to 50 in steps of 50 creates about 10 candidate subset sizes. If each size is evaluated with five-fold cross-validation, that is roughly 50 model fits before the final fit, not counting preprocessing and hyperparameter tuning. With a deep tree ensemble or neural model, the bill grows quickly.

“Are tree feature importances an embedded selector?”

Tree training does choose split features during fitting, so tree models are commonly described as embedded methods. But an importance ranking is not yet a selected feature set. If you keep features above an importance threshold and retrain, you have added a post-hoc selection stage. Also, impurity importance can favour continuous or high-cardinality columns, while correlated features can split credit between themselves.

Say this in the interview

“Filters are fast and model-agnostic, wrappers search model-scored subsets, and embedded methods select during fitting; I usually use a leakage-safe filter-plus-embedded pipeline and reserve wrappers for small, high-value feature sets.”