Feature engineering & encoding

Here’s the open secret of tabular machine learning: the algorithm is rarely what wins. Swap XGBoost for LightGBM and your score barely moves. But change how you represent the data — how you encode a category, whether you scale, what features you build — and the score can jump. Survey the people who actually win Kaggle and the refrain is the same: feature engineering, not algorithm choice, is what separates the top tabular solutions. The model is a commodity. The features are the edge.

Encoding categoricals — the daily decision

Models eat numbers, but data is full of categories (city, product, device). Three ways to turn a category into numbers, and the right one depends on cardinality and the model:

• One-hot — one 0/1 column per category. Perfect for low cardinality (device ∈ {ios, android, web}). For high cardinality it explodes the feature count into a sparse mess.
• Ordinal — map each category to an integer. Only correct when the category is genuinely ordered (size ∈ {S, M, L}). For unordered categories it invents a false ranking a linear model takes literally.
• Target encoding — replace each category with the mean target for that category. Compact and powerful for high cardinality (zip_code, user_id) — but it peeks at the label, so it must be fit inside cross-validation or it leaks.

Try the same table under each encoding, with a linear vs tree model:

Scaling numerics — matters, except when it doesn’t

Should you standardize features to mean-0, variance-1? It depends entirely on the model:

• Linear models, SVMs, k-NN, neural nets — yes. They compare feature magnitudes directly, so an unscaled income (tens of thousands) drowns out age (tens). Standardize or min-max scale.
• Trees and tree ensembles (random forest, XGBoost) — no effect. A tree splits on a threshold (income > 50000?), so multiplying a feature by 1000 changes nothing. Don’t bother.

The right way: ColumnTransformer + Pipeline

The professional pattern applies different transforms to different columns and bundles everything into one Pipeline — so the exact same transformation is re-fit inside each CV fold and re-applied at inference. This is what makes feature engineering leak-proof.

Beyond encoding — the features that actually win

Encoding gets the data in; these build the edge:

• Aggregations (groupby). “Average spend per user,” “transactions in the last 7 days.” Group-level statistics are the single most powerful tabular feature family — they inject context a single row can’t see.
• Interactions. debt / income, price × quantity. Ratios and products capture relationships the model would otherwise have to discover.
• Datetime features. From a timestamp, extract day-of-week, hour, is-weekend, days-since-event. A raw timestamp is nearly useless; its parts are gold.
• Binning & transforms. Bucketing a skewed feature, or log(income), can linearize a relationship for linear models.

Quick check

Next

Now that your features are honest, you need to read whether the model is underfitting or overfitting them — bias–variance & learning curves — and validate it without leaking, in train/val/test & CV.