How do Docker and ONNX complement each other for packaging and deploying ML models portably?

How to think about it

The direct answer

Docker and ONNX solve different portability problems. Docker packages the user-space environment around the model; ONNX packages the model itself in a framework-independent format. Together, they let you export a model from PyTorch or TensorFlow once, then serve it from a small, repeatable container without installing the training framework.

Why two layers are better than one

Imagine a fraud model trained in PyTorch. It accepts 34 floating-point features and returns one logit. The training job uses PyTorch, CUDA libraries, pandas, a particular NumPy version, and several internal preprocessing functions.

A raw PyTorch checkpoint contains weights, but not a complete promise about how those weights should run. The serving machine still needs a compatible PyTorch installation, Python version, native libraries, preprocessing code, and sometimes CUDA support. A mismatch can cause an import failure, a slow CPU fallback, or a prediction that differs because the input was prepared differently.

Docker addresses that environment problem. A Docker image can contain:

• the Python interpreter and system libraries
• the inference server
• NumPy and other runtime dependencies
• the model file
• the exact serving code

An image is built into content-addressed layers and can be deployed by its digest, rather than trusting a mutable tag such as latest. That gives the operations team a repeatable user-space environment.

ONNX, short for Open Neural Network Exchange, addresses a different problem: framework dependence. An ONNX file describes a computation graph, its operators, tensor shapes, data types, and usually the learned weights. An ONNX Runtime process can execute that graph without importing PyTorch or TensorFlow.

Question	Docker	ONNX
What does it package?	The application environment	The model graph and weights
What does it remove?	Host dependency drift	Training-framework dependency
How does hardware enter?	Through the container and host runtime	Through the inference runtime provider
What it does not include	The host kernel and drivers	Your full preprocessing contract

ONNX Runtime uses an execution provider, meaning a backend that maps ONNX operations to particular hardware or libraries. Common examples include CPU, CUDA, TensorRT, OpenVINO, and Core ML providers, depending on the installed build and platform.

So the clean division is:

1. Export the trained model to ONNX.
2. Validate that the ONNX model matches the source model.
3. Put the ONNX file, ONNX Runtime, and serving code in a Docker image.
4. Select the execution provider appropriate for the deployment target.

Docker is the shipping box. ONNX is the portable description of what is inside the box that must be computed.

A concrete example

Suppose the PyTorch model has one input named features, with shape consisting of a batch dimension and 34 features. It returns an output named logit. A legacy-style PyTorch export could look like this:

import torch

model.eval()
example = torch.randn(1, 34, dtype=torch.float32)

torch.onnx.export(
    model,
    (example,),
    "fraud.onnx",
    input_names=["features"],
    output_names=["logit"],
    dynamic_axes={
        "features": {0: "batch_size"},
        "logit": {0: "batch_size"},
    },
    opset_version=17,
    dynamo=False,
)

The dynamic axis means the exported graph can accept a batch of one row or a batch of 64 rows while keeping the feature width fixed at 34. The opset is the version of ONNX’s operator definitions used by the export. It is not automatically the newest version you can find; it must be supported by the runtime you plan to deploy.

Export success is not validation. The exporter may produce a valid file whose numbers differ from the source model. I would check both the ONNX file and its predictions:

import numpy as np
import onnx
import onnxruntime as ort

onnx_model = onnx.load("fraud.onnx")
onnx.checker.check_model(onnx_model)

with torch.inference_mode():
    reference = model(example).detach().cpu().numpy()

session = ort.InferenceSession(
    "fraud.onnx",
    providers=["CPUExecutionProvider"],
)
actual = session.run(
    ["logit"],
    {"features": example.cpu().numpy()},
)[0]

np.testing.assert_allclose(
    reference,
    actual,
    rtol=1e-4,
    atol=1e-5,
)

Those tolerances are examples, not a universal law. For a fraud model, I would also compare probabilities, classification decisions, and business metrics over a representative held-out set, perhaps 1,000 rows containing rare and ordinary fraud cases. A random tensor mostly tests that the shapes line up. It does not test a branch that only activates for a missing value or an unusually large transaction.

The export job needs PyTorch. The serving image does not:

# requirements.txt
onnxruntime
numpy
fastapi
uvicorn

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY fraud.onnx serve.py ./

USER 10001:10001
EXPOSE 8080

CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8080"]

The CPU package is intentional here. A matching serving function might be:

import numpy as np
import onnxruntime as ort

from fastapi import FastAPI, HTTPException

app = FastAPI()

session = ort.InferenceSession(
    "fraud.onnx",
    providers=["CPUExecutionProvider"],
)

@app.post("/predict")
def predict(features: list[float]):
    if len(features) != 34:
        raise HTTPException(
            status_code=422,
            detail="Expected exactly 34 features",
        )

    values = np.asarray([features], dtype=np.float32)
    output = session.run(
        ["logit"],
        {"features": values},
    )[0]

    return {"logit": float(output[0, 0])}

In a real build, I would pin the Python dependencies, use a base-image digest, record the ONNX Runtime version, and generate a software bill of materials. An unpinned pip install inside Docker is still dependency drift wearing a container costume.

For GPU inference, the image needs a GPU-enabled ONNX Runtime package and compatible CUDA and cuDNN libraries. The host also needs a compatible NVIDIA driver and container runtime integration. Listing CUDAExecutionProvider in Python does not install any of those pieces.

The senior-level nuance

Neither Docker nor ONNX means “identical everywhere” in the absolute sense.

A container shares the host kernel, and GPU containers depend on host drivers. Different CPUs can use different instruction sets. Different execution providers can use different kernels or reduced-precision arithmetic. The same float32 model can therefore differ by a small amount across CPU, CUDA, and TensorRT execution.

That is usually manageable. It is not a reason to give up. It is a reason to define an acceptable numerical tolerance and test the actual deployment target.

The other major trap is preprocessing. ONNX generally contains the model computation, not the entire data contract around it. If training standardized feature 7 using one mean and standard deviation, the server must use the same values. If training ordered the 34 columns as age, amount, country_code, the request builder must preserve that order. A perfectly exported model fed the wrong column order is still a broken production system.

Common misconception: ONNX is not a complete serving system. It does not automatically provide an HTTP endpoint, request validation, feature transformation, authentication, monitoring, or a compatible hardware backend. Docker is not a model optimizer either. It makes packaging and deployment more consistent; ONNX Runtime or another backend may improve inference efficiency, but performance must be measured.

A practical failure mode appears when someone copies the provider list from a GPU example into a CPU image. The container starts, its provider list contains only CPUExecutionProvider, and GPU utilization remains at zero. Latency then jumps from the expected GPU level to several hundred milliseconds per request. The cause is often the onnxruntime CPU package, a missing container GPU integration, or incompatible CUDA libraries. Check session.get_providers() during startup and fail readiness if GPU execution is a hard requirement. Do not quietly accept a CPU fallback for a latency-critical service.

ONNX also does not support every operator equally across every exporter, opset, and runtime. Custom PyTorch operators, dynamic control flow, and newer attention kernels are common trouble spots. The first symptom may be an export exception such as an unsupported operator error. A more dangerous symptom is a model that exports but loses accuracy on a particular input family.

When conversion is not reliable, native PyTorch or TensorFlow serving may be the better choice. If the target hardware is fixed and maximum throughput matters more than portability, a hardware-specific compiled engine such as a TensorRT engine may outperform a general ONNX Runtime deployment. In that design, ONNX can still be the interchange artifact used before compiling for the target GPU.

The production pattern I would use

The model registry should store the ONNX artifact by content digest, along with its opset, input schema, output meaning, preprocessing version, and validation results. The container image should identify the serving code and runtime version. A deployment should therefore refer to both a specific image digest and a specific model digest.

For a tightly coupled release, bake the model into the image. The image then represents one atomic model server: code, runtime, and weights travel together. For independent model releases, let the container fetch a model by digest at startup, verify the digest, and report ready only after the model and preprocessing metadata have loaded successfully.

The export and comparison test belongs in continuous integration. The deployed service needs health checks, provider checks, latency metrics, input-distribution monitoring, and a rollback path. That is where the portability story becomes operational rather than merely photographic: the same artifact can be moved, inspected, and rolled back.

What they’ll ask next

Can I use ONNX without Docker?
Yes. Install ONNX Runtime directly on the host and run the model there. Docker is useful when you also want to control Python, system libraries, server code, and deployment configuration.

Does Docker guarantee identical predictions on every machine?
No. It controls much of user space, but not the host kernel, GPU driver, CPU instruction set, execution provider, or floating-point behavior. Validate numerical and business-level equivalence on each supported target.

What do you do when a model uses an unsupported ONNX operator?
First check the exporter and target runtime versions, then simplify or rewrite the model if practical. Otherwise use a custom runtime operator or keep the model on native framework serving. I would not force an ONNX conversion that changes production accuracy just to make the architecture look tidy.

Say this in the interview

“Docker makes the serving environment reproducible, while ONNX removes dependence on the training framework; together they let us ship a validated model artifact in a small container and choose the right inference backend for each deployment target.”