Many ML data failures are silent: the case for data contracts

At 03:12 on a Tuesday, the churn pipeline can be completely green and still be having a terrible night.

Picture a SaaS company retraining a customer-churn model every Monday. The billing team changes an export. Yesterday, monthly_spend_cents contained 7999 for a $79.99 customer. Today, it contains 79.99. The column still exists. The rows still arrive. The warehouse accepts the values. The training job completes.

Nothing crashes because the values remain acceptable to the warehouse’s numeric column and the model’s numeric input.

Other failures can still make an ML pipeline fail loudly:

• Missing files
• Parse errors
• Schema mismatches
• Timeouts
• Resource failures

This lesson is about the quieter data-quality failures.

The model simply sees most customers as 100 times poorer than they are.

The first visible symptom may be a sharp jump in predicted churn among low-spend customers. The actual retention metric arrives weeks later, after the bad batch has been joined to labels, used in a new model, and promoted to production. By then, someone is opening dashboards and asking why a perfectly healthy pipeline produced a sick model.

That is the argument for data contracts: do not make the model responsible for discovering that its inputs changed meaning. An enforced contract can block declared, testable violations at a chosen boundary before they become training data or a prediction.

But a declaration alone is not a semantic lie detector. Guarantees about meaning require explicit producer-side checks against one of these:

• Canonical data
• Paired representations
• Trusted baselines
• An equivalent mechanism

When valid data means the wrong thing

Many ML data-quality failures are silent when inputs remain type-compatible but their meaning changes. A model is doing exactly what it was built to do. It maps an array of values to an output. It does not know that one value represents dollars, another represents days, and a third is supposed to be a customer identifier.

An ordinary application often fails loudly when its input is malformed. A missing required field can trigger a parser error. An invalid enum can stop a request. A model usually gets a vector of valid floating-point numbers and produces another valid floating-point number. Arithmetic has no opinion about units.

This is a narrower claim than “ML fails silently.” Missing files, parse errors, schema mismatches, timeouts, and resource failures often make ML pipelines fail loudly. Ordinary software can also silently accept semantically wrong data. The special risk here is a type-compatible input whose meaning changed.

That makes a semantic change especially dangerous. A schema is the machine-readable shape of data. It includes:

• Column names
• Types
• Required fields
• Nesting

Semantics are what those fields mean. They include:

• Dollars or cents
• UTC or local time
• Event time or ingestion time
• One row per customer or one row per payment

A schema check can confirm that monthly_spend_cents is numeric. It cannot, by itself, confirm that the number is still cents.

This is also why natural data drift and broken data must be separated. Data drift means the real-world distribution changes, such as customers genuinely spending less. A producer sending dollars in a field named cents is not the world changing. It is an interface breach. A useful system should alert on the first and block the second when the evidence is strong.

What a data contract actually promises

A data contract is an enforceable agreement between a producer, the team or job publishing data, and a consumer, the pipeline or service relying on it. It states what the data looks like, what it means, how complete and recent it must be, and who is responsible when those promises fail.

That agreement becomes a control only when a check is deployed at a chosen boundary. An enforced contract can block the violations it declares and can test; a declaration alone cannot detect or stop them. A plausible value can still carry the wrong meaning if the contract has no rule or trusted comparison capable of exposing it.

The useful contract is not just a schema file. It covers several different guarantees:

Guarantee	Example for the churn model
Shape	monthly_spend_cents exists and is numeric
Meaning	Values are non-negative USD cents, not dollars
Quality	The customer key is present and unique
Time	The newest partition is no more than 26 hours old
Categories	plan_tier is one of free, pro, or enterprise

A small contract might look like this:

dataset: billing.customer_features
owner: billing-data
version: 3
columns:
  customer_id:
    type: string
    nullable: false
    unique: true
  monthly_spend_cents:
    type: integer
    unit: USD cents
    nullable: false
    min: 0
    max: 1000000
  plan_tier:
    type: string
    nullable: false
    allowed_values:
      - free
      - pro
      - enterprise
freshness:
  max_age: 26 hours

This is deliberately plain YAML, not a universal library format. The important part is the promise. The producer must publish integer cents, not merely a value that happens to fit inside a numeric database column. An enforced producer-side check plus a boundary gate gives the consumer something stronger than reconstructing the unit from old examples; a declaration alone does not.

The max value is not decoration. It encodes a domain boundary: if the largest legitimate monthly charge is $10,000, a value above 1,000,000 cents deserves investigation. The freshness rule matters because a perfectly shaped feature table from three days ago can be just as harmful as a malformed one when customers’ plans change daily.

Not every guarantee should block the pipeline. A missing required key is usually a hard failure. A small change in the distribution of spend may be legitimate and should usually create an alert first. The contract needs a severity for each rule, or engineers will eventually turn every check into a warning to get the morning build moving.

The cents-to-dollars failure, with numbers

Suppose the churn model standardises monthly_spend_cents before prediction. On its training data, the mean is 8,000 cents and the standard deviation is 3,000 cents. A normal $79.99 customer has a standardised value of roughly:

(7,999 - 8,000) / 3,000 = -0.0003

That is essentially average. After the export change, the same customer contributes:

(79.99 - 8,000) / 3,000 = -2.64

The model now sees a customer 2.64 standard deviations below the training mean.

Take a deliberately simple logistic model where lower spending is associated with churn and the spend coefficient is -0.4. The feature contribution changes from approximately zero to -2.64 × -0.4 = 1.06 log-odds, where log-odds are the model’s internal scale before it converts a result into a probability. If the other features cancel out and the baseline probability was 50 percent, that one unit mistake moves the toy prediction to about 74 percent churn probability.

The exact output of a real model will differ. The mechanism will not. The model does not recognise a unit conversion. It sees a large feature shift and applies the relationship it learned.

Catch the obvious violation

A row-level check can catch some versions of this incident:

SELECT COUNT(*) AS bad_rows
FROM billing.customer_features
WHERE monthly_spend_cents IS NULL
   OR monthly_spend_cents < 0
   OR monthly_spend_cents > 1000000
   OR monthly_spend_cents <> CAST(monthly_spend_cents AS BIGINT);

The exact cast syntax varies by warehouse, but the logic is portable. A value such as 79.99 is not equal to its integer cast, so it is counted as bad. That query only returns a scalar count; running it successfully does not fail the task. Wire the result to the task outcome:

# Pseudocode: these names stand for your warehouse/orchestrator hooks.
bad_rows = run_scalar(validation_sql)

if bad_rows > 0:
    quarantine_batch(batch_id)
    raise ContractViolation(f"{bad_rows} invalid rows")

run_scalar, quarantine_batch, and ContractViolation are placeholders for the corresponding operations in your warehouse and orchestrator. The required behavior is concrete: read the count, quarantine the batch, and raise a contract violation when the count is greater than zero. No model artifact should be produced after that exception.

Evidence beyond row rules

That check is useful, but it is not a lie detector. If the producer rounds dollars to whole numbers, 79.99 may become 80. The value is an integer, non-negative, and within range. Every row-level rule above can pass while the unit is still wrong. No validator can infer a unit from one plausible integer.

You need a second line of defence: producer-side tests against the canonical billing amount, a distribution check against a trusted baseline, or both. If the median spend was 8,200 cents last week and becomes 82 today, that is strong evidence of a scale change. It is evidence, not proof, so the producer still needs to own the unit guarantee.

The failure is worse during retraining. If the bad rows enter both the training and validation slices, offline evaluation can remain respectable because both sets describe the same corrupted world. The new model learns relationships from the wrong scale, gets registered, and may serve plausible-looking scores. A rollback restores the previous model, but it does not remove the contaminated dataset or explain why the next run failed again.

Why the gate belongs at the boundary

A contract check is valuable because of when it runs.

If the billing batch fails before feature materialisation, the bad data is still a batch. It can be quarantined, corrected, and replayed. The blast radius is a failed job and an alert to the billing-data owner.

If the same batch passes into a feature table, the consequences multiply. It may feed:

• A training dataset
• A model artifact
• An online feature store
• A customer-facing decision

Each downstream system can preserve the mistake while making the original cause harder to find.

Batch and online checks

For batch inference and retraining, run the contract before the model reads the data.

For online inference, keep the request check lightweight. At ingress, check:

• Required fields
• Types
• Ranges
• Allowed values

When an online feature has an age or expiry timestamp, check its TTL, or time-to-live. Do that at feature-store write or read, or at ingress when available. Reject or fall back on stale values. Monitor table or partition freshness and population distributions asynchronously. Those checks do not belong in the latency-sensitive request path.

The same contract must cover both training and serving inputs. Otherwise the training table can correctly describe cents while an online feature service sends dollars. That is training-serving skew with a very tidy schema.

The strongest objection is valid

The best objection is that strict contracts slow teams down.

Data evolves. Producers add columns. Business rules change. A hard gate can page someone for a real product change, and a model team may not even control the upstream table. If every small change requires a meeting between three teams, people will bypass the contract or weaken every rule until it becomes ceremonial.

That objection is not solved by pretending contracts are free. It is solved by making compatibility explicit.

An additive, optional column should normally be non-breaking. The following changes should be versioned:

• Renaming a field
• Changing its unit
• Changing its time grain
• Changing how missing values are encoded

For a migration, publish both forms for a defined period, update the consumer, compare them, and then retire the old field. Do not silently repurpose monthly_spend_cents because changing the name would require paperwork.

Be strict about facts that make the data unusable. Be cautious about facts that might represent a genuine change in the world. A missing customer key should block. A 12 percent rise in average spend may deserve an alert and an investigation, not an automatic outage.

Monitoring is the other serious counter-argument. Why not let the model run and watch:

• Score distributions
• Feature drift
• Business metrics

Because monitoring is downstream and often late. Labels for churn may arrive 30 days after a prediction. A score shift can tell you that something changed, but not whether the cause was a billing-unit error, a new customer segment, or a successful marketing campaign. Monitoring remains essential for unknown failures and real-world change. It is not a substitute for rejecting a known-invalid input at the door.

A contract also cannot guarantee that the model is good. It will not detect concept drift, label bias, or a feature that was legally collected but poorly chosen. It guarantees an interface. Model evaluation, drift monitoring, and business checks still have to do their jobs.

What to do on Monday morning

Start with one boundary, not the entire data lake. Pick the model feature set with the largest blast radius. Write down:

• Its source
• Owner
• Unit
• Time zone
• Row grain
• Null policy
• Valid range
• Freshness target

For the churn model, that might be the table containing customer_id, monthly_spend_cents, plan_tier, and last_payment_at.

Ask the producer to confirm each meaning in writing. “Numeric” is not a sufficient answer. The useful answer is “integer USD cents from the ledger, one row per customer, updated by 05:00 UTC.” That sentence gives a future reviewer something to test.

Put the contract in version control beside the pipeline configuration. Make the gate run before feature materialisation and again before training if those are separate boundaries. Store the contract version and the check results with the dataset or model run. A failed batch should remain available for inspection; dropping it silently makes debugging much harder.

Then test the gate with deliberately bad fixtures:

• Rename the column.
• Remove a required value.
• Insert a duplicate customer.
• Send 79.99 where cents are expected.
• Send an old timestamp.
• Add an unknown plan tier.

By the end of the day, each mutation should fail before a model artifact is created. This is more convincing than a green test written only for healthy data.

Choose the enforcement tool based on where the boundary lives. Common choices include:

• Warehouse teams commonly express row and relationship assertions with dbt data tests.
• Python pipelines can validate dataframe-like data with pandera.
• Great Expectations and Soda provide declarative checks and validation runs for broader data workflows.

The tool is secondary. A beautifully described rule that runs after training is still in the wrong place.

Finally, assign a human owner and a change path. The contract should specify:

• Who receives the alert
• How long a failed batch can wait
• Which changes require a new version
• Which checks are warnings

Review the rules after the first real incident. The goal is not to produce a document nobody reads. It is to make the next unit conversion fail at 05:01, while the producer still remembers what changed.

Data contracts are not a promise that the world will stay still. They are a promise that your model will not quietly mistake a changed interface for a changed world.

Related lessons

• Data contracts & quality — the full treatment, with a runnable contract-gate example
• Testing ML & the ML Test Score — where data tests fit in the wider testing picture
• Training-serving skew — why identical feature meanings must hold in both paths
• Drift — how to monitor changes that a contract should not automatically block