Data & model versioning
Version datasets and model artifacts so an old ML run can be rebuilt, inspected, and defended months later. Why a git commit is not enough, how content hashes work, and when to use DVC, lakeFS, or table snapshots.
What you'll learn
- Why a code commit alone cannot reproduce an ML run
- How content hashes, pointers, and object storage version large datasets
- How DVC, lakeFS, and table snapshots differ in practice
- What a reproducible run must pin beyond code and data
- Which versioning failures show up first in production
Before you start
At 3 a.m., a fraud model starts approving too many suspicious payments. You check the model’s MLflow run, find the exact git commit, and run the training command.
The new model is different.
The code is the same. The training table is not. Since the original run, 400,000 new transactions have been appended, old records have been corrected, and the feature query now sees today’s table. You have reproduced the instructions, not the experiment.
That is the gap data and model versioning closes.
A git commit identifies code. A data version identifies the bytes, rows, or table snapshot that code consumed. A model version identifies the resulting artifact. You need all three to answer a basic question:
Why is the model in production the model it is, and can we rebuild it?
Why git alone fails
Git pins source files, not a changing database table. A query against transactions can return different rows tomorrow even when its SQL and the code commit are unchanged. A fixed random seed cannot restore examples that were added, corrected, or removed.
Git is also a poor store for large datasets. A 5 GB Parquet file makes every clone heavier, and changing a few rows can create a large new blob without a useful line-by-line diff. The usual pattern is:
- Commit a small pointer to git. It records the identity of a large object.
- Store the dataset or model bytes in object storage.
- Use the pointer’s identity to retrieve the exact bytes later.
The mechanism: identity is not storage
A content hash is a digest calculated from a file or object. Change one byte and, with overwhelming practical probability, the digest changes:
7f2c...identifies one set of bytes.- A pointer containing
7f2c...identifies those same bytes. 91aa...identifies different bytes.
The hash does not contain the 5 GB dataset. It is a luggage tag, not a suitcase. A system therefore needs a content-addressed store, where objects are stored under names derived from their contents. Identical files can share storage; changed files receive new identities while old objects remain available.
This provides exact retrieval, deduplication, and an immutable identity even when a path such as train.parquet is reused. It still requires access control, backups, durability, and retention. Versioning is not a backup strategy: deleting the old object leaves a pointer to something unrecoverable.
Byte identity also does not capture meaning. Record schema, timezone, transformation code, and business definitions. A byte-perfect reconstruction of the wrong table is still the wrong training set.
A worked example: rebuilding risk-model-17
On January 15, the fraud pipeline creates a Parquet export:
- 10,000,000 transaction rows
- 38 columns
- 5.0 GB on disk
- 120,000 confirmed fraud labels
- data cutoff: January 14 at 23:59 UTC
It uses a 70/15/15 train, validation, and test split: 7,000,000, 1,500,000, and 1,500,000 rows. The pipeline records the data identity as 7f2c....
In March, the current export has 10,400,000 rows and 126,000 fraud labels. The same split proportions now produce 7,280,000 training rows, 1,560,000 validation rows, and 1,560,000 test rows. They are not the same examples, so a changed metric cannot be attributed cleanly to a code change.
To reproduce January:
- Check out the commit containing the matching
data/train.parquet.dvcpointer. - Retrieve the object identified by
7f2c.... - Check its row count, schema, and cutoff.
- Run with the same parameters and environment.
- Compare the model and metrics with the recorded run.
Assume 8c41e2d contains both the January code and pointer. If the pointer was committed separately, record that commit as pointer_commit and check it out too. Row count is only a sanity check; the content identity establishes byte-level equality.
Version the model as well. Store the 280 MB model.pt artifact and record its digest, b81a..., with the run:
code_commit: 8c41e2d
pointer_commit: 8c41e2d
data_version: 7f2c...
parameters: split_seed=42, max_depth=8, learning_rate=0.05
environment: sha256:9d10...
model_artifact: b81a...risk-model-17 is a useful name. The artifact digest and metadata establish what that name means.
DVC: pointers in git, bytes elsewhere
DVC, or Data Version Control, applies this pattern to files and directories. dvc add creates a small .dvc metadata file containing a content hash. Commit that metadata to git and push the large bytes to a DVC remote, commonly backed by S3, Google Cloud Storage, or Azure Blob Storage.
# First, configure a remote appropriate for your organisation.
dvc remote add -d storage s3://ml-data/prod
# Track the dataset and commit its pointer and remote configuration.
dvc add data/train.parquet
git add .dvc/config data/train.parquet.dvc data/.gitignore
git commit -m "version January fraud training data"
# Upload the bytes addressed by the pointer.
dvc push
# Rebuild the old workspace months later.
# 8c41e2d must contain the matching data/train.parquet.dvc pointer.
git checkout 8c41e2d
dvc pulldvc pull fetches missing objects and checks them out; a local cache can avoid the download. The remote URL belongs in .dvc/config, but credentials belong in .dvc/config.local, environment variables, or the cloud identity mechanism.
The pointer must be committed, and the referenced object must be pushed. A local dvc add without a git commit is invisible to collaborators; a committed pointer without dvc push names data nobody can retrieve.
DVC can also track model files. Other teams use MLflow or a model registry. The brand is less important than an immutable, retrievable artifact linked to its code and data.
Choosing the version boundary
| Data shape | Useful version boundary | Typical choice | Deciding question |
|---|---|---|---|
| Files or folders produced by a training job | The exact file tree | DVC | Can the training job consume a checked-out directory? |
| Many objects in a data lake | A branch or commit over an object-store namespace | lakeFS | Do teams need isolated changes and atomic promotion? |
| Managed analytical tables | A table snapshot or time-travel version | Delta Lake or Apache Iceberg | Can the query engine read an older snapshot? |
| A query over mutable warehouse tables | A retained export plus query metadata | Warehouse snapshot or object-store export | Will the source retain the same state six months from now? |
| A trained model or checkpoint | An immutable artifact plus run metadata | Artifact store, MLflow, or registry | Can you download the exact bytes and identify their producer? |
DVC fits file-shaped training inputs. lakeFS provides branches and commits over data lakes. Delta Lake and Apache Iceberg provide table snapshots and time travel, subject to retention and engine support. These are different version boundaries, not interchangeable commands.
SQL alone is insufficient for a mutable warehouse table. Record its snapshot or export the exact input, along with cutoff, schema, and transformation logic. If no stable snapshot exists, state that reproducibility is limited.
Reproducibility is a bundle of inputs
A trustworthy run record pins:
- Code: the git commit containing training and feature-transformation code, plus
pointer_commitif separate. - Data: the exact file, snapshot, or derived input. For derived data, retain upstream versions or the immutable output.
- Parameters: hyperparameters, split rule and seed, label window, feature flags, and decision threshold.
- Environment: dependency versions and system assumptions. A container image digest is stronger than a mutable tag such as
fraud-trainer:latest. - Artifact: the immutable model bytes and their digest.
There are two meanings of reproducible. Re-run reproducible means another engineer can reconstruct the inputs and obtain a result close enough to validate. Bit reproducible means the artifact is byte-for-byte identical.
A fixed seed does not guarantee bit reproduction: GPU kernels, parallel reductions, libraries, and compilers can change results. If exact bytes matter, record hardware and determinism settings and test the training stack. Otherwise, define an acceptable metric and artifact tolerance.
What breaks first
The old object is gone
Symptom: dvc pull fails or a lake snapshot cannot be opened.
Cause: cleanup, expired retention, or a local cache that was never pushed.
Fix: protect required prefixes, configure lifecycle rules deliberately, test restoration, and retain data for as long as dependent models must be reproducible.
The pointer or path is not reliable
Symptom: two engineers use the same commit but get different files, or s3://bucket/train/current changes checksum.
Cause: the pointer was not committed, the object was not pushed, or a moving name such as current or latest was treated as a version.
Fix: log the resolved content hash, object version ID, or snapshot ID; make training fail when the expected pointer is missing; and check that referenced objects exist.
The data matches but the model differs
Symptom: data hashes match but metrics or model bytes do not.
Cause: an unpinned parameter, dependency, preprocessing step, hardware setting, or nondeterministic operation.
Fix: compare the complete manifest, log resolved configuration and environment digest, capture relevant hardware, and define tolerances or enforce deterministic settings.
Labels can move too: backfills, confirmation windows, and business rules may change while the exported bytes remain fixed. Version label-generation code and record its policy and cutoff.
The cost you are choosing
History costs money. Twenty-four 5 GB snapshots can require up to 120 GB before compression, replication, and derived data. Deduplication reduces repeated bytes, but garbage collection can destroy the ability to reproduce old runs.
There is a privacy cost. Old snapshots may retain personal information after correction or deletion. Retention, deletion requests, encryption, access controls, and legal holds must be designed with versioning. Do not retain every intermediate file by default: keep the source snapshot, the consumed training set when necessary, the transformation recipe and upstream versions, and the model artifact.
A compact production manifest might look like this:
run_id: fraud-2026-01-15-0042
code_commit: 8c41e2d
pointer_commit: 8c41e2d
data_version: 7f2c...
data_cutoff: "2026-01-14T23:59:00Z"
parameters:
split_seed: 42
max_depth: 8
environment_digest: sha256:9d10...
model_artifact: b81a...This manifest joins the experiment tracker, data store, artifact store, and deployment system. It turns “the January model” into an addressable object.
Practice
Take the January fraud run: git commit 8c41e2d, data identity 7f2c..., and model artifact b81a....
What happens if the pointer is committed but dvc push never runs, or if the object is pushed but the pointer commit never reaches the shared repository? In the first case, history names data nobody can retrieve. In the second, data exists but no shared history identifies which run meant to use it.
The useful test is simple: can a new machine, with no warm local cache, retrieve the exact data and model from the manifest?
Quick check
A question to carry forward
You can now connect a model to the code, data, parameters, environment, and artifact that produced it. That is provenance: knowing where the model came from and being able to rebuild it.
But provenance is not operational control. Suppose forty reproducible models exist and someone must identify the current champion, the challenger, and approved rollback candidates. A model registry provides that shared state. The next question is not “can we find the old model?” but “which version is allowed to serve, and who decided?”
Practice this in an interview
All questionsDVC (and lakeFS) version raw datasets and model artifacts as immutable snapshots tied to Git commits, giving reproducibility and rollback. A feature store manages computed features for training and serving, its main job being to keep offline and online feature definitions in sync to prevent training-serving skew. They are complementary: DVC answers what data made this model, while a feature store answers how do I serve the same features consistently.
A git commit captures code, but an ML run also depends on the exact training data, hyperparameters, environment, and randomness, none of which live in Git. Datasets are too large for Git and change independently of code, so you need a data-versioning tool like DVC or lakeFS to pin a content hash of the data to the commit. Full reproducibility means versioning code, data, config, environment, and seeds together and linking them.
Full ML reproducibility requires locking three layers: the random seed across all frameworks, the software environment via pinned dependency manifests or container images, and the training data via content-addressed versioning. Missing any one layer means the same code can produce different models on different runs or machines.
A model registry is a central catalog for deployable model artifacts and their metadata, lineage, approvals, and immutable versions. Production systems promote a tested version through deployment controls, usually using a mutable alias such as champion while retaining the exact version for rollback and audit.