Data contracts & quality

At 09:07 on Monday, the loan model starts receiving customer incomes like 5_200_000 and 6_100_000. Nothing crashes. The columns are present. The values are numbers. The API returns a probability for every request.

The upstream billing team has changed dollars to cents.

The model was trained on incomes such as 52,000 and 61,000 dollars. It now sees values one hundred times larger. A model may turn those inputs into absurdly confident decisions, and nobody gets an exception. The first visible symptom might be a two-point fall in approved-loan conversion three weeks later.

A crashed pipeline is an inconvenience. A model that keeps running on wrong data is an incident with a tidy dashboard.

A data contract is an explicit, enforced agreement about a dataset’s structure, meaning, and operating expectations. It tells the producer what it must send, the consumer what it may rely on, and the pipeline what to do when reality breaks the agreement.

Garbage in, wrong out — quietly

A model sees values, not their meaning. It does not know whether a column has the right unit, whether zero means zero or “unknown”, or whether a country code shifted one column to the left.

Suppose a churn model learned from monthly_spend in dollars. During training, 80 dollars appears as 80.0; if serving sends cents, the same customer appears as 8000.0.

If the feature was standardized:

z = (x - μ) / σ

With a training average of 60 and standard deviation of 20, the real value produces:

z = (80 - 60) / 20 = 1

The cents version produces:

z = (8000 - 60) / 20 = 397

That is not a slightly unusual customer. It is a value from a different universe. Depending on the model, it may be clipped, pushed into a saturated probability, or combined with other features in an unintended way.

The failure’s location matters:

• A missing column often causes an immediate error.
• A wrong type may stop a parser.
• A wrong unit is usually a valid number.
• A stale feature can be perfectly shaped and completely wrong.

The model cannot distinguish corrupt input from an unusually rich customer. Data must be checked before the model interprets it.

A contract is more than a schema

A schema describes shape: column names, data types, keys, and sometimes nullability. It answers “can I parse this?” A contract also answers “does this mean what we agreed it means?”

For a customer feature table, useful rules cover:

1. Identity and structure: user_id exists, is an integer, and identifies one row.
2. Value semantics: age is measured in years and stays in a plausible range.
3. Relationships: a timestamp is not later than ingestion time; the table has the expected row grain.
4. Operations and governance: partitions arrive on time, volumes are plausible, and sensitive columns have owners, access rules, and retention.

An invariant is a rule that should remain true, such as unique user_id values. A contract records these expectations in a machine-checkable form.

The producer creates or publishes the data. The consumer relies on it, such as a feature pipeline or model service. The contract should name an owner, define compatible changes, and specify how breaches are reported. An alert should normally reach the team that changed the source, with enough evidence to reproduce the failure.

The running example: catch the cents bug

Imagine a credit model with one row per customer. Its contract says:

• user_id is a non-null integer and unique within the batch.
• age is a non-null integer from 0 through 120.
• income is a non-null number measured in dollars, from 0 through 1,000,000.
• country is non-null and one of US, UK, IN, or DE.

The maximum income reflects the product’s population: values above one million are more likely to indicate a unit or parsing error than a genuine customer. Documenting that reason keeps a later engineer from removing an apparently arbitrary check.

The incoming batch is:

Structurally, it is fine: columns exist, values parse, there are no nulls, countries are allowed, and IDs are distinct. Semantically, it violates the agreed unit. The likely intended values are 52,000 and 61,000 dollars. A schema check cannot distinguish those numbers; the range rule can.

import pandas as pd

from pandas.api.types import (
    is_bool_dtype,
    is_integer_dtype,
    is_numeric_dtype,
    is_object_dtype,
    is_string_dtype,
)

CONTRACT = {
    "user_id": dict(dtype="integer", nullable=False, unique=True),
    "age":     dict(dtype="integer", nullable=False, min=0, max=120),
    "income":  dict(dtype="number", nullable=False, min=0, max=1_000_000),
    "country": dict(dtype="string", nullable=False,
                    allowed={"US", "UK", "IN", "DE"}),
}

def matches_dtype(series, expected):
    dtype = series.dtype
    if expected == "integer":
        return is_integer_dtype(dtype) and not is_bool_dtype(dtype)
    if expected == "number":
        return is_numeric_dtype(dtype) and not is_bool_dtype(dtype)
    if expected == "string":
        return is_object_dtype(dtype) or is_string_dtype(dtype)
    return False

def enforce(df, contract):
    errors = []

    for column, rule in contract.items():
        if column not in df:
            errors.append(f"missing column: {column}")
            continue

        series = df[column]
        type_ok = matches_dtype(series, rule["dtype"])

        if not type_ok:
            errors.append(f"{column}: wrong type")
        if rule.get("nullable") is False and series.isna().any():
            errors.append(f"{column}: nulls")
        if rule.get("unique") and series.duplicated().any():
            errors.append(f"{column}: duplicates")
        if not type_ok:
            continue

        values = series.dropna()
        if "min" in rule and (values < rule["min"]).any():
            errors.append(f"{column}: below min")
        if "max" in rule and (values > rule["max"]).any():
            errors.append(f"{column}: above max")
        if "allowed" in rule and not set(values) <= rule["allowed"]:
            errors.append(f"{column}: bad category")

    return errors

def compute_features(df):
    return df[["age", "income"]]

def train_model(features):
    print(f"training on {len(features)} rows")

batch = pd.DataFrame({
    "user_id": [1, 2],
    "age": [34, 29],
    "income": [5_200_000.0, 6_100_000.0],
    "country": ["US", "UK"],
})

try:
    violations = enforce(batch, CONTRACT)
    if violations:
        raise ValueError(f"contract blocked: {violations}")

    features = compute_features(batch)
    train_model(features)
except ValueError as exc:
    print(exc)
    print("\nThe model never trains on the corrupted batch.")

contract blocked: ['income: above max']

The model never trains on the corrupted batch.

The function checks type, missingness, range, allowed values, and uniqueness. It records a wrong type before skipping comparisons that could raise a TypeError. The causal chain is:

1. The producer emits the wrong unit.
2. The gate compares it with a domain constraint.
3. The batch is blocked before feature computation or training.
4. The producer owner receives a useful error.
5. The model does not learn from corrupted examples.

A good contract shortens the path from symptom to owner.

Write rules that describe meaning

A weak contract says “income is a float.” A useful one says “income is a non-negative amount in US dollars for the customer’s latest completed month.”

That definition covers type, unit, domain, time meaning, and missingness. A null means “no value is present”; zero means “the measured value is zero.” Replacing null income with zero may pass a non-null check while changing the model’s interpretation. Define imputation as a separate, explicit step.

Ranges need reasons. An age range of 0 through 120 catches negative ages and accidental timestamps, but not a teenager represented as 340 because the source used months. A range is a useful net, not proof of meaning.

Contracts should also state the dataset’s grain: what one row represents. “One row represents one customer at the end of one completed billing month” determines whether duplicate IDs are errors, which timestamps matter, and whether a join is allowed to multiply rows.

Allowed values need an evolution policy. A new country may be a legitimate backward-compatible change, a change requiring model support, or a value to quarantine pending approval. Say which rather than forcing someone to disable an assertion.

Freshness and volume are part of quality

Valid values can still be unusable if data is late or incomplete. A table that normally contains 10 million rows may pass all row-level checks after arriving with only 200,000 rows.

Contracts can specify:

• the partition must arrive by a deadline;
• event timestamps must cover the expected period;
• row counts must stay within a justified range;
• major regions must contribute plausible shares.

Use a severity appropriate to the decision:

• Block: missing required columns, duplicate keys, impossible timestamps, or high-confidence unit violations.
• Quarantine: suspicious volume, an unapproved category, or a distribution change that may be real.
• Warn: a small shift that deserves tracking but does not make the batch unusable.

Thresholds depend on context. An 80% row-count threshold may suit a stable nightly export and fail for a growing product.

This is related to drift. A contract defines what is allowed; drift monitoring compares data with a reference period. Valid population changes can trigger drift without violating a contract, while a unit conversion can violate a contract before a coarse monitor notices.

Where to enforce the contract

Use layers:

• Producer: validate before publication, where the source team can fix renamed fields or changed units.
• Ingestion: protect every downstream consumer.
• Feature and training boundary: recheck the exact snapshot after transformations, joins, and imputation.
• Serving: validate request or online features; a training contract cannot protect a separate online pipeline.

For batch jobs, a failed check can preserve the previous known-good artifact. For online services, block malformed required fields but consider a safe fallback for optional features. State that policy explicitly.

Do not silently repair violations inside the gate. Converting cents to dollars might be wrong if the source sent yen or misplaced a decimal. Make intentional transformations explicit, versioned, and separately tested.

Version the agreement, not just the data

Contracts change with products. Adding an optional column may be compatible; renaming a field, changing its unit, removing a category, or changing UTC timestamps to local time may be breaking.

Publish the contract version with the data or artifact and record which version each consumer validated. A change should identify the old and new meaning, compatibility, migration window, approving owner, and compatibility tests.

A model version identifies the artifact that made a prediction. A contract version identifies the input assumptions that artifact was allowed to make. You need both during an incident. Connect contracts to data versioning and lineage so you can trace a failure to the source release, table snapshot, transformation, and model run.

Failure modes you will actually see

Checks pass, but model metrics fall. The rules may be too weak. A maximum catches the cents example but not every factor-of-two mistake. Add unit-bearing definitions, distribution comparisons, cross-field rules, and comparison with a known-good batch.

Nulls become zeros. A prediction shift may appear without a validation error. Preserve the distinction between missing and measured zero, check missingness before imputation, and contract the imputation step separately.

Training passes, but the endpoint behaves strangely. The online service may use a different transformation, stale lookup, unit, timestamp, or default. Compare feature definitions at both boundaries; an offline contract does not automatically cover serving.

The team bypasses the gate. Require an override reason and owner, and measure overrides. A gate nobody trusts becomes expensive decoration.

A contract cannot tell you whether the label is ethical, whether the model is fair, or whether the business chose the right target. It cannot prove that a source is truthful. It guarantees only the assumptions you wrote down and the checks you actually run.

That limitation is healthy. A contract is a seatbelt, not a map: it prevents predictable mistakes but does not tell you where the road should go. For a shared feature table, undocumented assumptions are already a contract. Make them visible, executable, versioned, and owned.

In one breath

A data contract is a producer–consumer agreement about a dataset’s shape, meaning, and operating expectations, enforced at data boundaries. Schema checks catch missing columns and wrong types; semantic checks catch wrong units, impossible values, invalid categories, and broken row relationships; freshness and volume checks catch late or incomplete data. Use blocking, quarantine, and warning policies deliberately. A contract stops known bad inputs early and routes failures to the right owner, but does not prove that data is truthful or that the model is good.

Practice

Return to the income batch. Its columns, types, nullability, countries, and IDs were valid. The failure came from a semantic rule: income exceeded the agreed dollar range.

Now imagine the values are 52,000 and 61,000, but the producer still sends cents and the documented maximum is 10,000,000. The range check passes. What evidence would you add: a unit declaration, distribution comparison, cross-field check, or all three? Decide which should block and which should quarantine based on confidence and the cost of delay.

Finally, ask who receives the alert. If the producer changed the unit, paging only the model team guarantees a slow investigation. The contract’s value is rejection with an owner and a reason.

A question to carry forward

A baseline gives you a number to beat, data-centric practice improves the examples, and a contract prevents predictable corruption from entering the system. But reproducibility also requires knowing which model was trained on which data snapshot, code commit, and hyperparameters. That is experiment tracking, where the model’s history stops living in somebody’s notebook.