Most ML failures are silent: the case for data contracts

The most expensive ML incidents don’t announce themselves. A crashed pipeline pages someone in minutes. But when an upstream team renames a column, starts sending nulls, or quietly switches a field from dollars to cents, the model doesn’t error — it keeps serving predictions, now subtly wrong, and nobody notices until the business metrics sag weeks later. This is the defining failure mode of production ML, and in 2026 the standard defense has a name: the data contract.

Why ML eats bad data without complaining

The trouble is that a model can’t tell “corrupt data” from “the world changed.” It faithfully maps whatever arrives to a prediction. As one engineering write-up puts it, when the downstream consumer of data is an automated ML system rather than a human reading a report, a schema or freshness change can fail silently and degrade predictions — which is exactly why you need to validate data before it reaches the model rather than discovering the problem in a dashboard later.

That urgency has grown as more data flows straight into retraining loops. The failure everyone now fears is the silent one: the producer redefines a column, dashboards still look fine, but a retraining job is quietly poisoned.

A contract is schema plus semantics, enforced

A data contract goes beyond a schema definition. Soda describes it as an enforceable agreement between data producers and data consumers that specifies how data should be structured, validated, and governed as it moves through a pipeline. The key word is enforceable — the contract is checked every time data flows, so a violation blocks the pipeline at the source. The responsibilities are explicit, too: consumers specify the guarantees they depend on and producers take responsibility for meeting them, with contract validation acting as a control point before data is released.

The tools that enforce it

You don’t hand-roll this in production. Four tools cover most needs:

• Great Expectations — an open-source framework for describing data with expressive, unit-test-like Expectations and then validating that data meets them, with Checkpoints for production pipelines.
• pandera — a Python library with a flexible, declarative API for validating dataframe-like objects (pandas, polars, dask, pyspark) via schemas, ideal for inline checks in a training script.
• dbt tests — for warehouse data, dbt ships four built-in generic tests (unique, not_null, accepted_values, relationships) that are SQL assertions passing only when zero rows fail.
• Soda — declarative checks with first-class data-contract support.

The mindset shift is to treat incoming data with the same rigor as code. A contract that fails the build on a bad batch is worth far more than a dashboard you check after the damage is done.

Related lessons

• Data contracts & quality — the full treatment, with a runnable contract-gate example
• Testing ML & the ML Test Score — where data tests fit in the four-category rubric
• Training-serving skew — the failure data contracts help prevent