Lineage and provenance

At 03:12 on a Tuesday, the fraud team notices something ugly: approved transactions are suddenly being charged back. The model has not changed. The deployment has not changed. The dashboard says everything is healthy.

An engineer finds the likely culprit: an upstream table started writing country codes with trailing spaces. A feature transformation now treats some German customers as unknown. The question arrives immediately:

Which predictions were affected?

A model registry can identify a registered or promoted model, and a data-versioning system can identify available snapshots. Neither alone records the complete runtime deployment or the data actually consumed by a run; those links must be recorded in lineage metadata.

That chain is lineage: a record of what produced what. Provenance is the evidence behind that record, such as content hashes, commits, configuration files, and timestamps.

The useful unit is not “the model.” It is one connected graph.

The graph behind a prediction

A lineage graph has nodes, the things that exist, and edges, the relationships between them. A raw table is a node. So are a feature table, training run, model file, deployment, and prediction request.

An edge records both dependency and operation: a training run produced a model; a run read specific features; a transformation ran specific code under specific configuration at a particular time.

For the fraud model, the graph looks like this:

This is more than a pipeline log. A pipeline log says what ran; lineage says what depended on what.

Data versioning gives datasets stable identities. Lineage connects those identities to transformations and consumers. A model registry tracks model versions and promotion stages; lineage connects each model back to its run and data, then forward to deployments and predictions.

Identity: why hashes beat timestamps

Suppose the fraud pipeline reads:

warehouse/transactions/latest.parquet

That path is a pointer, not an identity. Tomorrow it may contain different bytes. A timestamp is also weak: files with the same logical date can differ, clocks drift, and corrected files can retain old modification times.

A content hash is a fingerprint calculated from an object’s bytes. SHA-256 produces a 256-bit digest. Change one byte and the digest changes; identical bytes produce the same digest. A content-addressed identifier looks like:

sha256:8f4c...

Store the full digest in metadata; use shortened forms only for display.

The causal advantage is that identity comes from the artifact itself, not its location or the time it was touched. A run can say, “I read object with digest X,” rather than, “I read whatever latest meant on Friday.”

The hash identifies bytes, not meaning. Two Parquet files with the same rows in a different order have different hashes, as do files with different line endings. Decide whether a dataset identity means exact file bytes, a canonical representation of rows, or a manifest of immutable partition hashes, and record that policy.

A worked lineage chain

Consider a chargeback classifier retrained on 21 August.

The raw transaction snapshot contains 18,240,000 rows and has content hash sha256:raw-71c2 in this illustrative example. A cleaning job removes 222,558 malformed rows, leaving 18,017,442. The job runs from Git commit 4f91c2e.

The feature transformation produces 42 columns. Its customer_chargeback_rate_30d feature uses the previous 30 days of labelled transactions. The feature snapshot has identity sha256:feat-93a8.

Training run run-2026-08-21-1842 records the feature hash, commit, configuration hash, environment-lock hash, and output. Its configuration specifies maximum tree depth 6, learning rate 0.08, 500 estimators, random seed 17, and a validation split consisting of the final seven days rather than a random split. It produces model artifact sha256:model-c1d4, registered as fraud-v42.

That model is deployed as deploy-2026-08-22-a in a production image with its own digest. At 14:07:32 UTC, request req-88421 receives score 0.91 for transaction t-1001. The prediction record points to the deployment, model, feature values, request, output, and timestamp.

Walking backward from req-88421 reaches fraud-v42, then the training run, feature snapshot, raw snapshot, and transformation commit.

If the country-code repair changes the cleaned row count from 18,017,442 to 18,019,006, it produces a new feature hash. The old model remains linked to the old data; the graph keeps both histories distinct. That is the mechanism: a changed content hash creates a new node, so downstream artifacts can be compared against the corrected input instead of silently inheriting its identity.

The practical minimum

You do not need a graph database on day one. Require every training run to carry a complete, immutable lineage record.

A minimum lineage baseline includes:

1. Exact data snapshot hashes and relevant upstream parents, plus point-in-time and query semantics: extraction query, parameters, time zone, joins, filters, and feature windows.
2. The Git commit supplying transformation and training code.
3. A committed environment lock and its hash.
4. An immutable runtime or container image digest, with interpreter, operating-system, platform, and relevant hardware or accelerator details.
5. The run configuration as content and a hash, not only editable dashboard fields.
6. Deterministic-operation settings for the framework, numerical libraries, parallel execution, and accelerator kernels.

Also record the run ID, times, random seeds, evaluation data hash, output model hash, and launching person or service. MLflow can store this metadata, but a tracking tool is not the lineage policy. The pipeline must enforce the required fields.

Define “reproduce” before promising it. Bitwise reproducibility requires identical output bytes. A more realistic target may be numerical reproducibility within an agreed tolerance, such as a metric range and prediction threshold. Seeds do not guarantee identical results across kernels, hardware, operating systems, or library builds.

For an individual prediction, record a prediction ID, immutable model artifact digest, immutable serving image or preprocessing-code digest, feature schema or transformation version, deployment and request IDs, output and timestamps, and the actual feature vector or an immutable feature-snapshot reference. A mutable deployment ID is insufficient unless it resolves permanently to those digests. Do not recompute features later from the current feature table: new labels or repairs may have changed the result. If privacy rules prevent storing raw inputs, use a protected reference or hash plus a retention-controlled copy.

Coarse and fine-grained lineage

Coarse lineage records dataset- or table-level relationships: a run read transactions_snapshot_2026_08_21 and customer_features_snapshot_2026_08_21.

Fine-grained lineage records row-, column-, or field-level relationships: prediction req-88421 used fields from transaction t-1001 and 30 days of that customer’s events.

Coarse lineage is cheap because one edge can represent millions of rows. Fine lineage costs more because joins, filters, windows, aggregations, deletes, and sampling must preserve their relationships. One feature may depend on hundreds of source rows, multiplying a table into billions of edges.

Resolution	Question answered	Cost	Typical use
Path and timestamp	What did the job appear to read?	Low	Temporary debugging
Dataset or column hashes	Which artifacts or fields may be affected?	Low to moderate	Most production systems
Row-level lineage	Which records or decisions are affected?	High	Regulated or high-consequence decisions
Prediction snapshots	What exactly did this request see?	Moderate storage	Reproduction and investigations

The questions lineage must answer

What is the blast radius? Start at a bad source snapshot and traverse forward to feature snapshots, training runs, models, deployments, and predictions. Coarse lineage gives a safe upper bound: perhaps all 6,400 predictions from a deployment are potentially affected. Row-level lineage may narrow that to 214 predictions that used malformed rows.

Can we reproduce one prediction? For req-88421, retrieve the model, serving code or image, and exact feature values presented to it. A model file alone is not enough: current feature data may differ, and a Git commit may not pin native libraries or hardware behavior. Replay the request in a clean environment and compare it against an agreed tolerance.

Can we prove compliance? Lineage can show which approved dataset, code, configuration, model, and deployment led to a decision. It cannot prove that data was legally collected, access was authorised, or a deletion request was correctly enforced. Those require classifications, access logs, retention rules, approvals, and tests.

Choose resolution according to the cost of an imprecise answer. Use dataset-level lineage when retraining or reviewing a bounded group is acceptable; add column-level lineage for sensitive fields and schema changes; add row-level lineage when each decision has significant legal, financial, or safety consequences. Prediction snapshots often provide the useful middle ground.

The honest limitation

Lineage has carrying costs. Hashing a large dataset is usually manageable; retaining immutable snapshots, feature vectors, and prediction records is not free. Fine-grained lineage increases storage and pipeline complexity, can expose sensitive relationships, and becomes unreliable when transformations use opaque user-defined code.

There is also a social failure mode: nobody owns the graph. Six months later it contains final_final_v3, links to deleted storage, and a dashboard claiming “complete.” Lineage is only as reliable as the gates that prevent incomplete runs from becoming official artifacts.

Do not build row-level provenance because it sounds rigorous. Build it when the decision requires that resolution. For the fraud model, coarse lineage can identify every potentially affected deployment while prediction snapshots preserve high-value cases. That may resolve the incident in an afternoon; a row-level graph costing millions may add little.

The best lineage system is boring at the edges: immutable inputs, explicit parents, complete run metadata, and predictions tied to the model and immutable serving computation that produced them. The graph can become sophisticated later. The evidence cannot be invented after the incident.

What to remember

• Lineage connects raw data, transformations, features, training runs, model versions, deployments, and predictions.
• Content hashes identify exact bytes; paths and timestamps identify locations and moments that can change.
• A baseline records data, code, configuration, environment, runtime, data-creation semantics, and deterministic-operation identities.
• An individual prediction needs immutable model and serving-computation identities plus the actual inputs or an immutable reference.
• Coarse lineage identifies what may be affected; fine lineage identifies which rows or decisions, at higher cost.
• Lineage supports reproducibility and compliance but does not prove that the underlying data or process was correct or authorised.

Quick check

0/3

Q1

The current path of the feature tableThe exact feature values or immutable feature snapshot used by that requestThe dashboard screenshot showing the model metricThe timestamp when the deployment was created

Check answer

Q2

A hash makes the artifact smallerA hash records who approved the artifactA hash is derived from the artifact bytes, so a content change produces a different identityA hash guarantees the artifact is correct

Check answer

Q3

Keep only timestamps for the patient tableKeep dataset-level lineage only and delete prediction recordsKeep dataset lineage plus row-level relationships or retained prediction snapshots for decisionsRely on the model version because the model file did not change

Check answer