Feature selection

A support team wants to predict whether a new ticket will be escalated. The table has 100,000 rows and 80 plausible columns: customer plan, message length, previous escalations, product area, time of day, and so on.

It also has ticket_id, an internal routing code, 12 nearly duplicated customer-history columns, and resolution_code—a field filled in after the ticket is closed.

Throw every column into a model and the validation score rises from 0.81 to 0.91 ROC AUC. The model looks brilliant. Six weeks after launch, it quietly falls back to roughly 0.80 because resolution_code was not available at prediction time and the routing code captured an old team assignment.

This is the problem feature selection exists to solve: not “how do I make the table skinny?” but “which information is available, useful, and worth making the model depend on?”

Feature selection means keeping a subset of the original input columns and discarding the rest. It is different from feature extraction, which combines columns into new representations, and from explanation, which describes how an already-trained model uses them.

The three families

There are three broad strategies. They differ in what they optimise, so there is no universal winner.

Filter methods

A filter method scores each feature without fitting the final predictive model, then keeps the best-scoring columns. Pearson correlation measures linear association with a numeric target; chi-squared tests work for suitable categorical or count data; mutual information measures more general dependence.

Filters are fast because columns can be scored before an expensive model runs. The catch is that they score columns independently. message_length and contains_angry_words might be weak alone but decisive together, so a filter can discard both. Correlation also misses curved relationships; mutual information detects more shapes but can be noisy with small samples or high-cardinality columns.

Wrapper methods

A wrapper method uses the model as its measuring instrument: try a subset, fit the estimator, measure validation performance, and compare it with another subset. It can detect interactions because the model evaluates features in context, but repeated fitting is expensive.

Recursive Feature Elimination (RFE) repeatedly fits an estimator, removes the weakest features according to coef_ or feature_importances_, and stops at the requested number. RFECV adds cross-validated scoring to choose how many to keep. A wrapper is tied to its estimator and metric: a subset that suits a random forest may not suit logistic regression.

Embedded methods

An embedded method selects features during model training. With L1 regularization, the objective is:

loss + λ × (|w1| + |w2| + ... + |wp|)

A large enough λ drives some coefficients exactly to zero, selecting those columns while the model is fitted. Regularization explains the broader bias–variance trade-off.

Tree models offer another embedded signal through impurity-based feature importance. It is useful for a first look, but high-cardinality continuous columns can look important by chance, and correlated columns can share or steal importance. Treat it as a clue, not a verdict.

Why fewer columns can generalize better

Training performance rewards memorisation. Generalization means performing well on new rows from the same process. Selection helps when it removes opportunities to learn accidents while retaining recurring signal.

Suppose 10,000 tickets contain five predictive fields and 995 random ones. A flexible estimator can test many relationships among those columns. Some random relationships will look impressive in the training sample simply because there are so many chances to get lucky. They do not repeat next month, so the model has spent capacity fitting quirks instead of escalation behaviour.

This is one face of the curse of dimensionality: data becomes sparse relative to the space of possible inputs. Neighbourhood models have fewer genuinely similar neighbours, trees have more candidate splits, and linear models have more coefficients to estimate. The common problem is limited data spread across too many degrees of freedom.

Fewer columns also reduce collection, validation, documentation, serving memory, and explanation costs. But fewer is not automatically better. Discarded columns may contain real signal, and a strongly regularized model or a large dataset may tolerate irrelevant inputs cheaply. Selection can add instability without improving accuracy.

The useful question is: “Does removing these columns improve the metric, cost, reliability, or maintainability that matters here?”

A worked example: what does “important” mean?

Imagine the ticket model has a held-out set of 20,000 tickets. Its baseline ROC AUC is 0.86: the probability that it ranks a randomly chosen escalated ticket above a randomly chosen non-escalated ticket.

Now permute one feature by randomly reordering it among held-out rows. Its distribution remains, but its connection to the correct ticket is broken:

This is permutation importance. For prior_escalations_90d, 0.86 - 0.74 = 0.12; a larger drop means the fitted model depended more on that feature for this evaluation set. The method is model-agnostic because it needs only predictions and a scoring rule.

It measures model reliance, not truth, causation, or business value. agent_id may be predictive because some agents receive harder tickets. Removing it could lower AUC while making the system fairer or more portable. A high score is a reason to investigate, not permission to deploy the column.

Correlated features create another problem. If prior_escalations_30d and prior_escalations_90d carry nearly the same history, shuffling either one alone may produce a drop of 0.04: the model can use its surviving twin. Shuffle both together and the drop may be 0.11. Assess correlated groups together, or choose the cleaner representation. Repeat permutations and examine the average drop and spread, not one ranking.

import numpy as np

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFE, SelectKBest, mutual_info_classif
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline

# Twenty columns: five informative, the rest noise.
X, y = make_classification(
    n_samples=1500,
    n_features=20,
    n_informative=5,
    n_redundant=0,
    n_repeated=0,
    random_state=0,
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

model = RandomForestClassifier(n_estimators=200, random_state=0)
model.fit(X_train, y_train)

permutation = permutation_importance(
    model, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0
)
top_six = np.argsort(permutation.importances_mean)[::-1][:6]
print("top six:", top_six.tolist())

rfe = RFE(model, n_features_to_select=5)
rfe.fit(X_train, y_train)
print("RFE selection:", np.flatnonzero(rfe.support_).tolist())

# The filter is part of the pipeline, so each CV training fold
# learns its own scores and chooses its own 10 columns.
pipeline = make_pipeline(
    SelectKBest(mutual_info_classif, k=10),
    LogisticRegression(max_iter=2000),
)
scores = cross_val_score(pipeline, X, y, cv=5, scoring="roc_auc")
print("CV AUC mean:", scores.mean())

The forest receives test rows only for held-out permutation importance. RFE uses training rows. The final pipeline learns SelectKBest separately inside each cross-validation training fold, then applies that choice to that fold’s validation rows.

An independent test set has a second job. Use cross-validation to choose k, selectors, and other settings, then evaluate once on a test set that stayed untouched. If you repeatedly choose the test winner, it has become training data. Nested cross-validation separates tuning from evaluation when the comparison is high-stakes; see model selection.

A production pattern that survives contact with data

Start with an all-feature baseline using only columns genuinely available at prediction time. Record its metric, latency, memory, and operational cost. Then remove obvious non-features:

• A unique row identifier usually identifies a row, not its outcome.
• A post-outcome field is not valid for an earlier prediction.
• Timestamps must be represented using information available at decision time.
• Duplicate fields may be consolidated for reliability.

Investigate suspicious IDs rather than deleting them automatically: an ID can encode geography, product version, or a real batch effect. The question is whether that relationship will exist when the model runs.

Choose the method according to the constraint:

If serving cost motivates selection, measure serving cost. A feature that takes 200 milliseconds to fetch may deserve removal despite a score drop of only 0.002; deleting a cheap feature for an insignificant accuracy change may be pointless.

Finally, check stability across folds or training seeds. Track selection frequency and groups of equivalent columns. A feature chosen in five of five folds is more reassuring than one chosen in one of five, though neither proves future usefulness. If the all-feature model has AUC 0.86, the selected model 0.861, and 40 percent of selected columns change across folds, that is not a meaningful victory.

Failure modes you can see first

Production performance collapses. Inspect selected columns for leakage and train–serve skew; a pipeline prevents cross-validation leakage but cannot make a post-resolution field available at prediction time.

A filter removes a feature and accuracy falls. Look for interactions. Use a model-aware selector, retain a domain-mandated interaction, or engineer it explicitly.

Correlated features all have small permutation scores. Permute the group together and choose the cheaper, earlier, more stable, or less sensitive representation.

Selection changes wildly between folds. Weak signal, small samples, and correlated predictors cause instability. Check whether performance differences are real; then consider stronger regularization, grouped selection, more data, or no hard selection.

A model becomes easier to explain but less fair. Predictive features can encode unequal treatment or access, and proxies may preserve the same information. Evaluate subgroup performance and policy consequences alongside the main metric. Feature selection is not a substitute for fairness.

Quick check

Next

If you want to combine columns rather than discard them, see PCA. For explanations of individual predictions, see SHAP and model interpretability. For a deeper treatment of leakage, see data leakage.