AutoML raised the floor, not the ceiling

At 4:47 on Thursday afternoon, a team has a loan-default dataset, a deadline, and a spreadsheet full of columns nobody fully trusts. The first model takes three days to assemble. Missing values need decisions. Categorical fields need encoding. Several algorithms need tuning. Then somebody notices that the validation score is worse than the old rules engine, and the whole argument starts again.

There is a better first move now. Give the table to a good AutoML system and let it produce a serious baseline before anyone spends a week hand-tuning a gradient booster.

That does not make the human work obsolete. It makes the human work more important.

AutoML has raised the floor: the minimum respectable result is now cheap. It has not raised the ceiling by the same amount, because the highest-value decisions are usually about what the rows mean, which information was available at prediction time, and which features express the business process.

AutoML can generate and search candidate features, but it cannot reliably infer or validate the domain semantics (what a field means in the business), prediction-time availability (whether it exists before the prediction), and business framing (what is being predicted and when) that make a feature trustworthy.

A strong baseline is no longer a toy

AutoML means software that automates several parts of the machine-learning workflow:

• data preprocessing
• model selection
• hyperparameter tuning
• often ensembling

It can also automate evaluation inside a validation protocol, meaning the rules for splitting data and scoring candidates, that a human specifies. It does not decide whether that protocol should be chronological, grouped, point-in-time, or otherwise deployment-valid, meaning it matches how predictions will actually be made. Choosing that split remains a human responsibility.

For tabular data, meaning data arranged as rows and columns, that automation is now genuinely capable.

AutoGluon’s official release notes report a result under the name AutoML Benchmark 2025: a five-minute budget beat rival frameworks given a full hour.

“Report” matters here. The result comes from AutoGluon’s own release notes, not from a current independent leaderboard. The old headline also failed to name the exact AutoGluon version, benchmark run date, benchmark revision, metric, dataset list, hardware, per-system budget, and compared systems. Each detail can change the ranking. AutoML Benchmark 2025 names a benchmark revision; it is not, by itself, a run date.

The cited evidence contains no current independent rerun for 28 August 2026, so this page does not claim that AutoGluon is the current benchmark leader. Rather than invent the missing metadata, treat the release-note result as a useful, vendor-reported historical comparison.

A defensible “current leader” claim would require an independent result that names all of those details.

The independent AutoML Benchmark evaluation is historical context. Its 2022 evaluation found AutoGluon had the highest average model-performance rank across its tested systems, while noting slower inference as a trade-off. “Average rank” means it was often near the front of the field across datasets; it does not mean every dataset had the same winning score or that the result remains current.

For tighter compute budgets, FLAML takes a different approach, searching efficiently over established learners such as XGBoost and LightGBM.

The practical implication is less glamorous than a leaderboard headline. You can ask a useful question on day one:

Can a robust, mostly automated pipeline already solve enough of this problem?

If the answer is yes, you may ship it. If the answer is no, you have a baseline that tells you what your custom work must beat. Either outcome is better than arguing about algorithms before measuring anything.

A concrete example: loan default

Imagine a lender with 250,000 historical applications and 60 columns. The target is whether an applicant defaults within 90 days of receiving a loan. The data includes income, requested amount, employment category, existing monthly debt, credit utilization, application timestamp, and a few customer identifiers.

The numbers below are illustrative, not a benchmark claim:

Model	Input	Honest chronological validation AUC
Logistic regression	Raw cleaned columns	0.71
Gradient-boosted trees	Raw cleaned columns	0.75
AutoML ensemble	Raw cleaned columns	0.78
Ensemble plus domain features	Raw columns plus engineered features	0.83

AUC, the area under the receiver operating characteristic curve, measures how well a model ranks risky applications above safe ones. An AUC of 0.78 means the model is substantially better than random ranking, not that 78 percent of its predictions are correct.

All four rows are illustrative development results on the same deployment-valid validation partition. The locked final test set is not part of this comparison.

The AutoML result is valuable. It establishes a strong floor. A custom model that scores 0.76 after two weeks has not demonstrated progress; it has demonstrated that the team spent two weeks rebuilding a weaker version of the baseline.

The jump from 0.78 to 0.83 is where the ceiling appears. The difference is not necessarily a cleverer algorithm. It may be a better description of the lending process.

What AutoML is actually doing

A tabular AutoML system starts with a rectangular training table. It tries a collection of reasonable ways to turn it into predictions.

It may fill missing values, encode categories, choose model families, tune their parameters, and compare them on a validation split that follows the protocol you supplied.

It also tries combinations of models. One model may be good at smooth numeric relationships. Another may handle categorical splits. Another may capture a different set of interactions.

A weighted ensemble combines their predictions. Stacking goes one step further: predictions from earlier models become inputs to a later model, called a stacker or meta-model, which learns when to trust each one.

Stacking has an important leakage safeguard. The stacker must be trained on out-of-fold predictions, meaning predictions for each training row made by a base model that did not train on that row, or on otherwise held-out base-model predictions.

In-sample base predictions are unsafe: a base model can memorize its training rows, and the stacker then learns from unrealistically polished predictions that will not exist at inference time. After the stacker is trained, the base models must be refit appropriately on the development data allowed by the protocol before producing predictions for real cases.

This works because different models make different mistakes. If model A misses applicants with unusual employment categories while model B handles those categories but misses nonlinear income effects, their errors are not identical. Combining them can reduce variance without discovering a new fact about lending.

AutoGluon’s approach is notable partly because it gets a lot from robust defaults, bagging, stacking, and ensembles rather than spending all its compute exhaustively tuning one model. The search is broad and practical.

It is not magic. The system still receives the table, the target, the metric, and the evaluation protocol from a human.

That last sentence is the boundary.

A leaderboard is also not a neutral oracle. If the system tries enough candidates and repeatedly chooses the one with the best validation score, the validation set becomes part of the search target.

Create three partitions:

• a training set
• a deployment-valid validation set
• a locked final-test set

Use the training and validation partitions for AutoML search and feature iteration. Freeze the pipeline before opening the final test, then evaluate the final candidate on that test once. If a comparison set is reused to guide decisions, call it validation, not test.

The ceiling is usually representation

Features that encode the process

Suppose defaults become much more common when monthly debt exceeds 45 percent of income. A tree can learn that relationship from separate debt and income columns, but it has to approximate a diagonal boundary with a series of rectangular splits.

A new debt_to_income feature states the relationship directly:

debt_to_income = monthly_debt / monthly_income

For an applicant earning $4,000 per month with $1,800 in monthly debt, the ratio is 0.45. The feature gives a linear model a useful signal immediately and gives a tree a simpler split to discover.

A powerful model can sometimes learn the same relationship. That is the important nuance. Feature engineering is not a guarantee of improvement, and a ratio can be noisy, redundant, or dangerously defined.

Modern AutoML can generate and search candidate transformations and features. But it cannot reliably infer or validate the domain semantics, prediction-time availability, and business framing that make a feature trustworthy. The system has no reliable way to know that this ratio represents the lender’s concept of affordability, rather than an accidental correlation in the table.

Point-in-time data

The same problem appears with event data. The raw table may contain one row per card transaction, while the prediction table needs one row per loan application.

A point-in-time, or as-of, join means that for each application you retain only events whose timestamps are before that application’s prediction timestamp and fall inside the preceding 90-day window. You then aggregate those retained rows by application.

For an application at 10:00 on 10 August, a transaction at 12:00 that day is excluded; a qualifying event on 15 May can be included.

A groupby may perform the final aggregation, but grouping transaction rows by customer alone is not sufficient. It commonly aggregates all of a customer’s history, including events after the application being predicted.

Most tabular AutoML systems will not reliably decide that the rows should be joined to applications, that 90 days is the right window, or that only events before the application timestamp are legal evidence.

That is domain knowledge expressed as data.

Problem framing and validation

Problem framing matters even more than feature names. Before training, someone must decide:

• the prediction moment
• the unit of prediction
• the label

Are we predicting default when the application is submitted, or after an underwriter has already called the applicant? Is the row an application, a customer, or a customer-month? Is the target default within 30 days, 90 days, or the lifetime of the loan?

AutoML should not be expected to establish those answers from the table. It will accept the framing you give it and optimize within it.

Splits, leakage, and deployment

A random train-test split can create the same illusion when time or customers matter. But time and customer grouping are not interchangeable. The right protocol depends on the deployment population, and the same protocol must govern AutoML validation as well as the final test.

If the service predicts future applications for customers already known to the lender, use a chronological split: earlier applications for training, later applications for validation and testing.

Preserve each customer’s legitimate, pre-prediction history when that history would really be available at prediction time. A random group split that forbids a known customer from appearing across partitions measures a different problem: performance on entirely unseen customers.

If the service must predict for genuinely unseen customers or other entities, use a group split, keeping every row for one entity on the same side of the split. This prevents the model from learning identity or repeated-history shortcuts.

A group split alone does not protect against temporal shift, however. If both future prediction and unseen entities matter, use a chronological group split so the entities are disjoint and the evaluation data is later in time.

This is why the data leakage guide belongs next to any AutoML notebook.

The metric is another human decision. If only 50 of every 1,000 applications can receive manual review, a model’s ranking among the top 50 matters more than its average accuracy.

A model that predicts “no default” for everyone can look accurate when defaults are rare. A model with a good AUC can still produce probabilities that are badly calibrated, meaning its predicted 20 percent risks do not default roughly one time in five.

Those decisions include:

• thresholds
• review capacity
• false-positive cost
• regulatory constraints

They do not emerge from a leaderboard.

The strongest objection is partly right

Modern AutoML systems can perform:

• feature selection
• transformations
• feature generation
• more sophisticated search than the early “try a few classifiers” tools

An agent can even write a candidate aggregation after reading a schema and a description of the business. On a clean competition-style dataset, generic automation may find enough structure to beat a hand-built pipeline.

That objection should change the claim, not destroy it. The ceiling is not a mystical wall that software can never cross. Automation improves, and some systems will discover useful features when the raw data and the search space make them discoverable.

But the system still needs a trustworthy definition of the task. It needs answers about:

• which events existed at prediction time
• which customers may be grouped together
• what mistakes cost
• whether a feature is legally or operationally usable

An LLM-generated recent_payment_count is not useful if “recent” includes payments made after the prediction. Generated code makes bad assumptions faster; it does not remove the assumptions.

There are also cases where you should stop at the baseline. If the AutoML model meets the business target, fits the latency budget, is calibrated, passes fairness checks, and can be monitored, shipping it is not a failure. Custom features are not a moral virtue.

Conversely, a large ensemble may be the wrong production choice. Multiple models can increase memory use, inference latency, and operational complexity.

The historical AMLB inference-speed caveat matters when the service has a strict 20-millisecond budget or must run on a small device. A simpler model with a small performance sacrifice may be the better system.

What to do on Monday morning

Start with a one-page data contract before opening the AutoML notebook. Write down:

• the row unit
• the prediction timestamp
• the label window
• the allowed source tables
• the business action
• the metric

For the loan example: one row per application, features available at submission time, default within 90 days, and review capacity of 50 applications per 1,000.

Then create three partitions:

• training
• deployment-valid validation
• locked final test

Use the deployment population to choose their protocol. For future predictions on known customers, preserve chronological order and legal pre-prediction history. For genuinely unseen entities, keep groups disjoint. When both constraints apply, use a chronological group split.

AutoML search and feature iteration may reuse the validation partition; the final test gets no labels exposed to those decisions.

Run a short AutoML job first. A 300-second run is enough to learn whether the table is tractable and to establish an initial floor; it is not enough to certify a model.

Record the library version, data snapshot, hardware, time budget, metric, and split. Benchmark claims are only reproducible when those details are visible.

Read the leaderboard as a diagnostic, not a trophy. Ask:

• Which model families are competitive?
• How much does the ensemble add?
• Is the best model much larger or slower?
• Does performance change across new versus old applications, employment categories, regions, or missingness levels?

Now add a small number of domain features. For the lender, that might be:

• debt-to-income
• utilization relative to the applicant’s own historical maximum
• the count of missed payments in the 90 days before application

Define every window using the prediction timestamp. Rerun the comparison on the same deployment-valid validation set. Do not use the locked final test to choose features or tune settings.

The feature-engineering guide is useful here because it treats features as hypotheses about the process, not decorative columns.

Once these are frozen:

• the feature set
• the model
• the hyperparameters
• the threshold
• the calibration procedure

evaluate the final candidate on the locked final test once. If you reopen that set to guide another decision, it has become validation data; call it that and create a new locked test if an unbiased final estimate is still required.

Finally, check whether the output can support the decision:

• calibration
• top-of-queue precision
• subgroup performance
• latency

If the model’s probability says 0.8 risk, find out whether similar cases actually default about 80 percent of the time. If the operations team can review only 50 cases, evaluate those 50 rather than admiring a global average.

The calibration guide covers the distinction between ranking well and producing trustworthy probabilities.

The first failure you will usually observe is not a dramatic crash. It is a suspiciously excellent validation score, followed by a disappointing production score.

The next is a model that performs well overall but fails on a small, important slice. Another is an ensemble that wins offline and misses the service’s latency budget.

Each symptom points back to a different missing decision: leakage, measurement, or deployment constraints.

AutoML is a force multiplier because it removes routine search. Use that saved time to ask better questions about the data.

If you cannot beat its baseline, ship it when it is safe and adequate. If you can beat it, the improvement should come from a defensible idea about the problem, not from trying the 47th variation of a parameter grid.

Related reading:

• AutoML in practice — how to use the baseline without mistaking it for a finished system
• Feature engineering and encoding — turning domain knowledge into usable columns
• Data leakage — why a perfect score can be the most dangerous result in the notebook