datarekha
Patterns June 10, 2026

Feature engineering still beats the algorithm (even after TabPFN)

Tree-based models remain state-of-the-art on tabular data, and the biggest gains come from the features, not the model. Even as a Nature-published foundation model finally challenges gradient boosting, the data-centric lesson holds.

6 min read · by datarekha · machine-learningfeature-engineeringtabulardata-centric

Ask anyone who’s won a tabular competition what made the difference, and they won’t say “I found a better algorithm.” They’ll talk about features. Swap one gradient boosting library for another and the score barely moves; change how the data is represented and it can jump. In 2026, with deep learning everywhere, the tabular world still runs on this stubborn truth — and the research backs it.

Trees still win on tabular data

This isn’t nostalgia. Grinsztajn, Oyallon, and Varoquaux (2022) benchmarked across 45 datasets and concluded tree-based models like XGBoost and random forests remain state-of-the-art on medium-sized (~10K-sample) tabular data, even before accounting for their speed. The why is instructive: they attribute trees’ edge to inductive biases neural nets lack — robustness to uninformative features, preserving data orientation, and learning irregular target functions. Those are exactly the properties that make good feature engineering pay off: a tree can ignore your junk columns and exploit the one ratio that matters.

Data-centric AI says the same thing

Andrew Ng turned this into a movement. He defines data-centric AI as “the discipline of systematically engineering the data needed to successfully build an AI system,” arguing the model architecture is largely solved and effort should shift to improving the data. The workflow is a mirror image of the usual one: hold the model fixed and iterate on the data — using error analysis to re-curate and improve examples rather than changing model code. For tabular ML, “improving the data” largely means feature engineering: aggregations, ratios, and the parts of a timestamp that inject context a single row can’t see.

The TabPFN twist — and why the lesson holds

There’s a genuine 2026 plot point worth knowing. TabPFN, published in Nature (2025), is a tabular foundation model that in 2.8 seconds outperforms an ensemble of the strongest baselines tuned for 4 hours, on classification datasets up to 10,000 samples — and its authors note gradient-boosted trees have dominated tabular data for roughly 20 years, framing this as a notable shift on small data. It’s real, and it’s exciting. But it’s a small-data result that reopens the question rather than closing it — and a better model still can’t invent the domain feature that isn’t in your columns.

Skip to content