Feature engineering still matters on many tabular problems

At 8:17 on a Monday morning, a model can look perfectly healthy. Its random holdout AUC is 0.82. AUC, the area-under-the-ROC-curve score, can be read here as a chance. Specifically, it is the chance that a randomly chosen defaulter receives a higher risk score than a randomly chosen non-defaulter.

Then the model meets applications from next quarter and starts approving people it should reject.

The usual response is algorithm shopping:

• Try XGBoost.
• Try another gradient boosting library.
• Try a neural network.
• Add a larger hyperparameter sweep.

This often produces a few attractive decimal places and very little understanding.

My engineering heuristic is narrower: on many medium-sized tabular problems, engineered, point-in-time features—features computed only from information available when the prediction is made—can matter more than switching among strong algorithms.

That is not a universal result, and this page does not pretend to prove it from a leaderboard. Test it with a fair ablation. A better model can extract more from good columns. It cannot reliably recover the question your columns failed to ask.

TabPFN does not erase that heuristic. It is an impressive tabular foundation model, and its benchmark results make “deep learning will replace boosting” a respectable argument rather than a conference hallway prediction.

A model can only use the shape you give it

A data representation is the set of variables and values the model receives. Raw database columns are one representation. Ratios, counts, recency measures, and customer-level aggregates are another.

Feature engineering means constructing those useful variables from available data. It is not magic, and it is not automatically “adding information.” Often it is changing the coordinates so that a pattern becomes easy for a learner to express.

Consider a loan application. The raw row contains:

• annual income: $60,000
• existing monthly debt payments: $1,800
• proposed monthly payment: $400

Gross monthly income is $5,000. If the relevant business concept is total monthly payment burden, the useful feature is:

(1,800 + 400) / 5,000 = 0.44

Now consider another applicant:

• annual income: $120,000
• existing monthly debt payments: $3,600
• proposed monthly payment: $800

The raw numbers are twice as large, but the burden is also 0.44. If repayment risk depends mainly on the share of income committed to debt, these applicants should look similar on that dimension.

The model can, in theory, discover this relationship from the raw columns. But the route matters.

A decision tree makes rules by splitting one variable at a time. A split might ask whether annual income is below $72,000, then whether monthly debt is above $2,100. The true boundary, however, is a diagonal relationship: debt rises with income. A collection of axis-aligned splits can approximate that boundary, but it needs a staircase of rectangles. Give the tree total_burden, and one threshold such as total_burden <= 0.44 expresses the relationship directly.

Gradient boosting, which builds many small trees that correct one another’s errors, can eventually approximate the ratio from the raw columns. With finite data, finite tree depth, and regularization, “eventually” is doing a lot of work. A useful feature reduces the complexity of the function the model has to learn. That can mean fewer examples, fewer trees, and fewer accidental rules.

This is why feature engineering can help even when the feature is a deterministic calculation. A perfect learner does not gain new information from a ratio that was already implied by two columns. A real learner working with limited data gains a more convenient shape.

The bigger gains often come from changing the row’s context.

Suppose the application table contains income and loan details, while a credit-account table contains balances, limits, and dated missed payments. The application row does not contain “missed payments in the last 90 days” until you calculate it. At the decision time of 10 January, that count must use records available up to 10 January, not the customer’s complete history as it exists today.

You might construct:

• utilization: current balance divided by available credit limit
• missed payments in the last 90 days
• number of open accounts
• months since the most recent delinquency
• variation in monthly income over the previous year

Those variables express domain questions. How much of the available credit is being used? Is the problem recent? Is income stable? The raw tables contain pieces of the answers, but not necessarily answers in the grain the model needs.

That last phrase matters. A row grain is what one row represents: an application, a customer, a transaction, or something else. Many weak models are not missing a fancy algorithm. They are trying to predict a customer outcome from a transaction row with no customer history attached.

The tree benchmark explains the practical default

This is not merely folklore from competition winners. Grinsztajn, Oyallon, and Varoquaux’s 2022 study compared neural networks with tree-based models across 45 tabular datasets. On medium-sized data, roughly around 10,000 examples, models such as gradient-boosted trees and random forests remained highly competitive and often state of the art in the benchmark.

The interesting part is not the leaderboard. It is the explanation.

Tabular data tends to contain a mixture of useful columns, irrelevant columns, different scales, missing values, thresholds, and irregular interactions. Tree models have useful inductive biases, meaning built-in preferences about what kinds of patterns are easy to learn. They can be relatively robust to uninformative features. They do not require every numeric column to occupy the same scale. They naturally represent “if this value crosses a threshold, risk changes” and combinations of such thresholds.

Neural networks can learn these patterns too. They may need one or more of the following to do so reliably:

• more data
• more careful normalization
• stronger regularization
• a better architecture

A dense network initially treats every input as something whose relevance it must learn. A tree can simply decline to split on a useless column. Neither behavior is guaranteed, but the starting geometry favors trees on many medium-sized tables.

This is also why feature engineering and tree models fit together so well. Trees are good at ignoring some noise and exploiting a clean, domain-shaped variable. They do not need the feature to be sophisticated. A count, a ratio, or “days since” can be more valuable than a new model family because it puts the useful distinction on one axis.

The benchmark is a strong prior, not a law. It does not say trees win on images, raw language, very large datasets, or every modern tabular architecture. It says that if you have 20,000 business records and a spreadsheet-shaped problem, starting with a strong tree baseline is usually more rational than starting with a neural architecture diagram.

Data-centric AI changes what “improvement” means

Andrew Ng’s description of data-centric AI is a useful correction to the model-centric reflex. He describes it as the systematic engineering of the data needed to build a successful AI system. The practical workflow is to hold the model and evaluation procedure relatively steady while improving labels, examples, consistency, and representation.

For tabular work, “the data” is not only the raw values. It includes:

• whether two tables were joined at the right grain
• whether a missing value means zero, unknown, or not applicable
• whether a label was recorded consistently
• whether a feature was available when the decision was made
• whether a timestamp was interpreted in the correct time zone
• whether an aggregate describes the past or quietly includes the future

Feature engineering is valuable when it is driven by error analysis rather than imagination.

Suppose the loan model’s false negatives cluster among applicants with seasonal income. Annual income hides the relevant instability. A trailing income range, minimum monthly income, or a measure of month-to-month variation may address the failure. If false positives are concentrated among applicants with otherwise healthy income but a recent delinquency, recency and delinquency-count features are more plausible than another round of tuning.

A feature should be treated as a hypothesis: “this observable relationship explains these errors.” Then test that hypothesis while keeping the split, metric, and model fixed. That turns feature engineering from a formula zoo into an investigation.

The feature engineering and encoding guide covers the mechanics. The important operating principle is that every new column should have a reason to exist.

TabPFN is a real challenge, not a rebuttal

TabPFN deserves more than a defensive footnote. The Nature paper reports TabPFN outperforming the strongest baselines on its small-tabular-classification benchmark. The benchmark includes datasets with up to 10,000 training examples. That is precisely the territory where gradient-boosted trees have been unusually hard to displace. It is a benchmark result, not evidence that TabPFN wins on every tabular problem.

What TabPFN can and cannot infer

TabPFN is a pretrained tabular model rather than a model trained from scratch on only your dataset. During pretraining, its weights—the learned numerical parameters—are trained offline on many synthetic tabular tasks sampled from a task prior, a probability model describing which kinds of tabular tasks are likely. Those synthetic rows are generated examples, not your customers’ records.

At prediction time, TabPFN receives the labeled training rows from your current task and the features of the rows needing predictions. It conditions its predictions on those examples without updating its weights. The prior transfers regularities across the synthetic tasks; it does not provide missing customer-specific history. TabPFN has learned how to adapt to a small table, not the facts that were never joined into yours.

That can make it remarkably effective when your dataset is small and the task fits its intended setting.

The fair response to TabPFN

The strongest objection to this article is therefore fair:

“If TabPFN can discover interactions and ratios from raw columns, why spend a week constructing them? Just use the foundation model.”

Sometimes you should. Using TabPFN is a good engineering decision if it is:

• accurate enough on your raw data
• fast enough for your workflow
• acceptable on memory, calibration, and operational constraints

Do not manufacture 200 fragile features to beat a model that already meets the requirement.

But three facts survive the objection:

• First, TabPFN cannot infer a variable that is absent from its inputs. If the application row omits recent delinquency history, no architecture can hallucinate the customer’s payment record into existence. You must retrieve and aggregate that history.
• Second, a strong model may infer a deterministic ratio, but that does not guarantee it will do so with the available sample size and inductive bias. Test the engineered column. If it adds nothing, remove it. If it improves several honest validation slices, the measurement wins.
• Third, the comparison must be fair. Evaluate boosting and TabPFN on the same raw feature set, then on the same engineered feature set. Comparing a carefully curated table for one model against a raw table for another tells you almost nothing about either algorithm.

TabPFN changes the Monday-morning shortlist. It does not change what the model is allowed to know.

The cost of a feature is paid after the notebook

Feature engineering has an honest downside: every useful feature becomes a production obligation.

A 90-day delinquency count requires a reliable historical query. A utilization ratio requires a current balance and a valid credit limit. A monthly-income statistic requires consistent timestamps and a policy for months with no income record. If the online system cannot reproduce the training calculation, the model sees one world in development and another at serving time.

A feature can also drift. Customers change behavior. A bank changes how it records account limits. A data provider changes a field definition while keeping the column name. The model may continue returning confident predictions because nothing in the API says the meaning changed.

Feature engineering is a poor choice in three cases:

• when the raw input is unstructured and a learned representation is the natural tool
• when the dataset is enormous and the model has enough data to learn the relationships safely
• when the feature cannot be computed consistently at decision time

It is also unnecessary when a simple baseline already satisfies the business need. Accuracy is not the only objective. These also count:

• latency
• freshness
• interpretability
• privacy
• maintenance

The right rule is not “always engineer features.” It is: engineer the representation when you can state the domain relationship, observe it before prediction, and compute it reliably.

The failure mode you will see first

The most dangerous feature often produces the most impressive validation score.

A loan team computes missed_payments_last_90_days by joining applications to the current credit table. The current table includes payment events that happened after each application. A random split mixes time periods, so the leaked column looks predictive in both training and validation. The model’s holdout score rises sharply.

Then a forward-looking test collapses. In production, those future payment events do not exist yet.

The first symptom is usually not a dramatic error message. It is a suspiciously dominant feature, a random-split score that looks too good, or a large gap between random validation and a time-based validation. A model can be perfectly reproducible and still be reproducing the future.

Unit errors are quieter. If one pipeline supplies annual income and another supplies monthly income, the burden ratio can be wrong by a factor of 12 while remaining numerically plausible. A sudden spike in the feature distribution, a shift in missingness, or nearly every prediction moving in one direction after deployment is worth investigating before retraining.

This is training-serving skew: the model was trained with one feature computation and served with another. A feature’s name is not a contract. Its source, grain, units, cutoff time, null behavior, and computation code are the contract.

What I would do on Monday morning

Start by writing down the prediction timestamp. For the loan example, that might be the moment the application is submitted. Draw a hard line there. Every feature must be calculated from information available on the earlier side of that line.

Use a time-based or grouped validation split when the business problem is time-based or customer-based. For example, train on applications from January through June, validate on July, and keep August untouched for the final test. Do not let applications from the same customer leak across the split. The right bias–variance and learning-curve diagnosis is impossible if the evaluation setup is already flattering you.

Next, establish one strong tree baseline on the raw columns. Record:

• the metric
• calibration
• inference time
• missing-value behavior
• performance on important slices

Do not record only the single best score. A model that gains 0.01 AUC but doubles latency and fails self-employed applicants may be a regression in the product.

Read a fixed sample of errors. One hundred false positives and false negatives is enough to reveal patterns that aggregate metrics hide. Group them by income type, account age, geography, or any other business-relevant slice. Ask what information the wrong predictions lack.

Then build a small set of features tied to those observations. For the running example, that might be total payment burden, utilization, and recent delinquency count. For these three candidate features, compare:

• the raw columns
• the raw columns plus each of the three singletons
• the raw columns plus each of the three pairs
• the raw columns plus all three

That is eight feature sets: one raw baseline, three one-feature additions, three two-feature additions, and one full set.

This matters because features can be redundant or interact. If utilization helps alone but adds nothing beside delinquency count, it may be carrying overlapping signal. If the pair helps while neither feature helps alone, the effect is an interaction. Comparing only each feature separately with the full set can misattribute both kinds of gain. If you only need each feature’s marginal value in the chosen full representation, an alternative is to compare the full set with each leave-one-out version.

Fit every preprocessing step—such as imputing missing values, scaling, or encoding—inside each training split. Keep the model, split, metric, and preprocessing procedure unchanged across the feature-set comparisons. Report score spread or uncertainty across honest splits, not only one flattering split. This is an ablation, meaning a controlled comparison that shows what each feature subset contributes.

Only after that should you compare algorithms. Try your tuned boosting baseline, a competing tree implementation, and TabPFN when the dataset fits its small-classification setting. Give each model the same feature sets. Measure not just predictive quality, but also:

• runtime
• memory
• calibration
• how gracefully it handles missing or delayed data

Finally, write a feature contract before shipping. For every feature, record:

• its source table
• row grain
• units
• point-in-time cutoff
• missing-value meaning
• refresh schedule
• owner

Replay the computation as it would have run on the original decision date. Monitor feature freshness and distributions after deployment.

The algorithm still matters. A weak learner can waste a good representation, and TabPFN has shown that architecture can produce a meaningful leap on small tabular classification. But model choice is downstream of what the model can see.

A good feature does not make the algorithm irrelevant. It makes the algorithm’s job possible.

Related lessons

• Feature engineering and encoding — constructed variables, transformations, and encodings that carry domain signal
• Bias–variance and learning curves — finding out whether data, representation, or model capacity is limiting you
• Data-centric AI — improving the data instead of reflexively replacing the model