Reproducible environments and builds

At 09:14, the validation job passes on your laptop: accuracy is 91.8 percent.

At 09:22, CI runs the same commit and gets 91.1 percent. The deployment machine does not even get that far. Its image omitted cuDNN 9, or failed to make the cuDNN 9 user-space library discoverable by the dynamic loader. It prints:

ImportError: libcudnn.so.9: cannot open shared object file

Nothing in the model code changed. The laptop has Python 3.12.4, NumPy 1.26.4, and a newer NVIDIA driver. CI installed Python 3.12.5 and NumPy 2.0.2, while its host also had an older, unverified NVIDIA driver. The project said only numpy and torch, so it constrained neither those package versions nor the rest of the Python graph. A Python dependency specification cannot install or select the host NVIDIA driver; that older driver was a separate machine-level input. If it is too old for the framework’s CUDA runtime, the later symptom is a driver/runtime mismatch such as CUDA driver version is insufficient for CUDA runtime version, not normally a missing libcudnn.so.9 file.

That sentence describes ingredients. It does not describe the meal.

A reproducible environment is a recorded set of code, packages, native libraries, operating-system inputs, and hardware assumptions that can be rebuilt to produce the same intended result. A reproducible build creates that environment without silently taking whatever happens to be newest today.

The cure is not simply “use Docker.” Docker can preserve a badly chosen, moving environment. First pin the dependency graph, then handle the ML-specific native stack, then package the result.

A requirement is a wish; a lockfile is a decision

Your project has direct dependencies, the packages your code explicitly asks for. A pyproject.toml might say:

[project]
dependencies = [
  "numpy>=1.26,<3",
  "scikit-learn>=1.5,<2",
  "pandas>=2.2,<3"
]

This is a dependency specification. It says which versions are acceptable, not which versions will be installed.

Packages also depend on other packages. These transitive dependencies form the complete dependency graph. A resolver examines that graph and chooses versions satisfying all constraints. On 28 August 2026, it might choose NumPy 2.2.6; six weeks earlier, the same command might have chosen NumPy 2.0.2. Both satisfy numpy>=1.26,<3.

A lockfile records the resolver’s decisions: exact versions, often package hashes, and sometimes separate choices for operating systems or Python versions:

numpy==2.2.6
pandas==2.2.3
scikit-learn==1.6.1

The important difference is whether the file records a complete resolved graph rather than a few acceptable ranges.

Only the lockfile gives you a reproducible package selection, and it has a boundary: it reproduces the graph for the target Python version, platform, architecture, package indexes, and dependency markers it covers. A Linux lock is not automatically a Windows lock; a CPU lock is not automatically a GPU lock.

Why an unpinned sub-dependency can change a model

Suppose text-normalizer arrives through the listed package document-pipeline:

document-pipeline 4.2
└── text-normalizer >=1.4,<2
    └── regex >=2024.1

On the day you build, the resolver selects:

text-normalizer 1.4.1
regex 2024.5.15

Later, text-normalizer 1.5.0 changes punctuation-boundary handling. It still satisfies the range, so installation succeeds. But 37 of 1,000 documents now receive a different token sequence. A classifier that was 918 correct before is 912 correct after retraining. The model did not randomly become worse; its input changed.

The causal chain is:

1. A direct package permits a range.
2. A transitive release fits that range.
3. A fresh install selects it.
4. Preprocessing, numerical kernels, serialization, or defaults change.
5. The model sees different numbers or tokens.

The same mechanism applies to NumPy, SciPy, BLAS, or compiler-linked libraries. A lockfile prevents this movement by recording exact transitive versions and, with hashes, exact artifacts. The build must install from that file, not merely check it into Git.

The modern Python workflow

For a Python-first project, uv manages the project file, resolves dependencies, creates an environment, and writes uv.lock:

uv init
uv add "numpy>=1.26,<3" "scikit-learn>=1.5,<2"
uv lock
uv sync
uv run python -c "import numpy; print(numpy.__version__)"

Commit pyproject.toml and uv.lock, but not .venv. For CI or a release image:

uv sync --locked --no-dev
uv run python -c "import numpy; print(numpy.__version__)"

If your organisation uses pip, pip-tools provides the same separation:

pip-compile --generate-hashes requirements.in
pip-sync requirements.txt

pip-compile resolves the graph and writes exact versions; hashes verify the downloaded artifacts. Use separate generated files when platform or GPU wheels differ. pip-sync removes installed packages absent from the compiled file, unlike ordinary pip install.

Conda remains useful for compilers, OpenMP, image libraries, and other native packages. An environment.yml with ranges is still a specification; use a platform-aware solution such as conda-lock. If pip and Conda are mixed, decide which tool owns each package instead of installing competing native libraries.

The ML compatibility matrix

Four layers sit below Python:

• Python supplies the interpreter and ABI that compiled wheels must fit.
• The framework wheel is the CPU or CUDA build of PyTorch, TensorFlow, or another framework. It may bundle CUDA user-space libraries.
• The NVIDIA driver runs in the host operating system and talks to the GPU. Containers do not replace it.
• The GPU hardware determines available instruction sets and compute capabilities.

The CUDA toolkit provides compilers and developer libraries. The CUDA runtime provides libraries needed to execute compiled code. They are related but not identical.

A newer NVIDIA driver generally runs applications built against an older CUDA runtime; an older driver cannot generally run an application requiring a newer runtime. “CUDA 12.4” on a package page is not an instruction to install that toolkit everywhere. Compare the framework wheel with the host driver and current compatibility tables. NVIDIA’s documented minimum Linux driver versions include 520.61.05 for CUDA 11.8 and 525.60.13 for CUDA 12.0; later releases or features may require newer drivers.

On the execution machine, record:

nvidia-smi
python -c "import sys; print(sys.version)"
python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.is_available())"

nvidia-smi reports what the driver can support, not necessarily the CUDA version used by the Python framework. Choose the framework build from its official matrix, pin its version and package source, and test an actual GPU operation—not just an import.

Deterministic Docker builds

Docker provides a filesystem boundary and repeatable build recipe, but latest is not a version. Pin the base image by digest and pin the uv build tool separately:

ARG PYTHON_BASE
FROM ${PYTHON_BASE} AS dependencies

WORKDIR /app
COPY pyproject.toml uv.lock ./

ARG UV_VERSION
ARG UV_HASH
RUN pip install --no-cache-dir --require-hashes \
      "uv==${UV_VERSION}" \
      --hash="sha256:${UV_HASH}" \
    && uv sync --locked --no-dev --no-install-project

FROM ${PYTHON_BASE} AS runtime

WORKDIR /app
ENV PATH="/app/.venv/bin:$PATH" \
    PYTHONPATH="/app/src"
COPY --from=dependencies /app/.venv /app/.venv
COPY src ./src
CMD ["python", "-m", "my_package"]

The pipeline must provide PYTHON_BASE as a reviewed digest-pinned reference, plus a reviewed UV_VERSION and SHA-256 hash for the target platform. A lockfile does not pin the resolver or installer that reads it. Treat the installer, artifact, package index, lockfile, and base image as build inputs.

--no-install-project keeps source changes from invalidating the dependency layer, but it means the project is not installed there. This example uses PYTHONPATH=/app/src; alternatively, install the project in a later layer. Copying source without either step leaves code in the image that Python cannot import. The CMD must also be replaced with the real service command.

Copy metadata and the lockfile before application source, install dependencies, then copy changing source. This lets Docker reuse expensive framework layers after a one-line source edit. The multi-stage build leaves build tools and caches out of the runtime image; keep both stages on compatible base-image families. Use .dockerignore to exclude virtual environments, Git history, checkpoints, and test data. Store model and dataset versions as explicit artifacts instead of copying whatever is in the working directory. See data versioning and Docker for ML for the larger service pattern.

Which tool should own the job?

Choice	Best fit	What it freezes well	Main trap
uv with uv.lock	Python-first applications	Python packages, wheels, and transitive dependencies	Native libraries and host drivers remain outside it
pip-compile with hashes	Existing pip workflows	Exact, reviewable requirements and artifacts	Platform and GPU variants may need separate files
Conda with a lock solution	Python plus native libraries	Cross-platform native packages	An unconstrained environment.yml is not a lock
Docker plus pinned base digest	Deployment and CI parity	User-space filesystem and build steps	It cannot pin the host driver or hardware

These layers can work together: a GPU service might use uv.lock inside a digest-pinned Docker image while the deployment platform guarantees a driver branch.

Do not read uv.lock as pinning the Python patch version. Pin the interpreter separately with .python-version, an exact requires-python policy, or a digest-pinned base image, and test the resulting image.

Failure modes you will actually see

Clean CI installs different versions. The project had ranges but no enforced lock, or CI used an ordinary install. Commit the lock, use locked mode, and fail when it is stale.

A framework wheel has no matching distribution. The wheel does not exist for that Python version, platform, architecture, or index. Check supported wheel tags and choose a compatible combination; do not remove the pin.

A missing libcudnn.so.9 appears. The image lacks cuDNN 9 or the loader cannot find it. Check the framework wheel’s bundled libraries, image contents, and loader configuration. This is not automatically an old-driver problem; driver/runtime incompatibility usually reports a different error.

Import succeeds but the first GPU operation fails. Compare nvidia-smi, the framework-reported runtime, wheel variant, and GPU model. Validate a real tensor operation in CI or image testing.

The container cannot import my_package or rebuilds dependencies after every source edit. A src/ layout needs installation or PYTHONPATH, and the Dockerfile should copy metadata and install dependencies before copying source. Add the real CMD or ENTRYPOINT.

The honest limit: identical bits are not always possible

A lockfile and pinned image cannot guarantee bitwise-identical GPU output across all hardware. Floating-point addition is not associative; parallel scheduling, Tensor Cores, cuDNN algorithm selection, drivers, and atomic operations can introduce differences. A fixed seed controls only randomness exposed by the framework.

Deterministic algorithms and disabled benchmark selection can reduce variation, but may reduce performance, fail on unsupported operations, or still leave hardware differences. Treat determinism as measured, not assumed.

For most production systems, target behavioural reproducibility: predictions remain within an agreed tolerance and preserve decisions that matter. Record the Git commit, lockfile hash, image digest, Python and framework versions, CUDA runtime, driver branch, GPU model, seeds, model checksum, and data version with every artifact.

If exact output is legally or scientifically required, fix the hardware class and driver stack, use deterministic operations where supported, and preserve reference predictions. That guarantee is narrower and more expensive.

What to remember

• Specifications allow ranges; lockfiles record the resolved graph and artifacts.
• Transitive packages can change preprocessing, numerical behaviour, or defaults.
• A GPU environment is a matrix of Python, framework wheel, CUDA runtime, driver, and hardware.
• Pin image digests and build tools, enforce the lock, and put stable dependency layers before source.
• Reproduce behaviour across hardware unless identical bits are necessary and affordable.

Quick check

0/3

Q1

The project file, because ranges are more portableThe lockfile, because it records the resolver's exact graphWhichever file was edited most recentlyNeither file, because Docker chooses the version

Check answer

Q2

The model's learning rateThe Dockerfile's source-code copy orderThe host driver, framework wheel variant, and CUDA runtime compatibilityWhether the project has a README

Check answer

Q3

Delete the lockfile and reinstall latest packagesAssume the training data changed without checkingMeasure an agreed behavioural tolerance, record hardware and driver metadata, and use deterministic settings where requiredAdd latest to every dependency so both machines receive updates

Check answer