Serving a model with FastAPI

The whole Tooling chapter built toward one thing: press the button and out comes a trained, versioned, tested, blessed model sitting in the registry. And there it sits. We closed that chapter on the gap it leaves — a model in a registry answers no requests and earns nothing until some process loads it and replies, in milliseconds, over the network, under real traffic. This lesson is that process, and it opens with the exact moment it goes wrong.

It’s 2 AM. The mobile team just pushed an iOS update that doubled request volume on your churn-prediction endpoint. Latency p99 jumped from 40 ms to 1.8 s. You SSH into the box and htop shows every gunicorn worker pegged at 100% CPU. Then you read the code and find it: every request reloads the 380 MB joblib file from disk. Someone wrote model = joblib.load(...) inside the handler.

The shortest path from “I have a model in a .joblib file” to “other services can call it over the network” is FastAPI. It’s three things at once: a web framework, a request-validation library (Pydantic), and an OpenAPI documentation generator. The naive version takes 20 lines. The version you’d actually ship takes about 60. The extra 40 lines are where the 2-AM-incidents live.

Monolith or microservice?

Before any code, the architecture question hiding inside “other services can call it over the network”: should the model even be a separate service?

In a monolith (one application, one deployable unit), the model lives inside your main app — the same process imports it and calls model.predict() directly. One codebase, one deploy, one thing to run and monitor. It’s the right call when a single small team owns everything and the model is light.

In a microservice setup, the model is its own small program behind an HTTP API — exactly what this lesson builds. Your app calls it over the network. Now the model can run on its own GPU box, scale on its own, ship on its own release schedule, be written in any language, and be shared by three other teams — paid for with one network hop and one more service to operate.

FastAPI is how you build that service. Here’s what a single request looks like once it does — the one-time model load on startup, then the per-request path:

The naive version (don’t ship this)

# app/main.py — minimal but wrong
from fastapi import FastAPI
import joblib

app = FastAPI()
model = joblib.load("models/model.joblib")    # loaded at import time

@app.post("/predict")
def predict(payload: dict):
    return {"prediction": int(model.predict([payload["features"]])[0])}

What’s wrong with that? Four things, all of which bite you in production:

1. No request validation. payload["features"] raises a 500 if the client forgets the key. There’s no schema, no helpful error message, no auto-generated docs.
2. Model loaded at import time. Works, but in some deployment modes (e.g. testing, Lambda) imports happen at unexpected times. The startup hook is the right place.
3. No batch endpoint. Every request triggers a separate model.predict call. sklearn and XGBoost both vectorise well; doing 100 single-row predictions costs ~100x more than one batched call.
4. No health endpoint. Kubernetes / load balancers need a path that returns 200 only when the service is ready. Without it, traffic hits pods that are still initialising.

Let’s fix all four.

The version you’d actually ship

# app/main.py — production-grade in ~60 lines
from contextlib import asynccontextmanager
from typing import List
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import joblib

# 1. Request schemas — Pydantic is non-optional.
class PredictRequest(BaseModel):
    features: List[float] = Field(..., min_length=8, max_length=8)

class PredictBatchRequest(BaseModel):
    rows: List[List[float]] = Field(..., min_length=1, max_length=1000)

class PredictResponse(BaseModel):
    prediction: int
    proba: float

# 2. Lifespan — load the model once, on startup. Replaces the old @on_event.
@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.model = joblib.load("models/model.joblib")
    yield
    # (nothing to clean up here, but this is where you'd close DB pools etc.)

app = FastAPI(title="Churn predictor", version="0.1.0", lifespan=lifespan)

# 3. Single-row endpoint
@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
    model = app.state.model
    proba = model.predict_proba([req.features])[0]
    return PredictResponse(prediction=int(proba.argmax()), proba=float(proba.max()))

# 4. Batch endpoint — much cheaper per-row at scale
@app.post("/predict/batch")
def predict_batch(req: PredictBatchRequest):
    model = app.state.model
    probas = model.predict_proba(req.rows)
    return [
        {"prediction": int(p.argmax()), "proba": float(p.max())}
        for p in probas
    ]

# 5. Health endpoints — liveness vs readiness
@app.get("/healthz")     # liveness: am I running at all?
def liveness():
    return {"status": "ok"}

@app.get("/readyz")      # readiness: am I ready to serve traffic?
def readiness():
    if not hasattr(app.state, "model") or app.state.model is None:
        raise HTTPException(status_code=503, detail="model not loaded")
    return {"status": "ready"}

That’s the whole production-ready service. Now let’s pick it apart.

Pydantic — your free request validator

Pydantic models are your API schema. FastAPI uses them for:

• Parsing and validating incoming JSON (wrong type → 422 with a helpful error message, not a 500).
• Generating OpenAPI / Swagger docs at /docs.
• Typed access in your handler (req.features is List[float], not Any).

Two patterns that punch above their weight:

• Constraints in the schema, not the handler. Field(min_length=8, max_length=8) enforces feature-vector length at the framework boundary. Junk requests are rejected before your model sees them.
• Separate response models. response_model=PredictResponse means FastAPI strips unexpected fields from your response (security) and documents the contract (clients can generate types from the OpenAPI spec).

Lifespan — the modern startup hook

You used to write @app.on_event("startup") to load the model. That’s deprecated. The modern API is lifespan, an async context manager. Code before yield runs at startup; code after runs at shutdown. Why is startup the right place? Because the lifespan runs exactly once per process, so shared state (the model, a DB pool, a cache) is initialised before any request arrives and freed cleanly on shutdown — no matter how many workers or test clients import the module.

Things that belong in lifespan:

• Loading the model (or fetching from MLflow / a registry).
• Connecting to a feature store or vector DB.
• Warming a cache.
• Setting any global state on app.state.

Things that don’t belong there:

• Per-request work (use a dependency or middleware).
• Anything that can fail in a way you don’t want to crash startup with unless you actually want startup to crash on it.

If lifespan raises, the app fails to start — which is correct behaviour for “the model file is missing” but bad behaviour for “an optional cache is down.” Be deliberate.

Why a batch endpoint matters

Imagine you’re scoring users in a nightly job: one million rows. With only a /predict endpoint, the consumer makes a million HTTP requests, each doing one sklearn predict_proba on one row. The model runs a million times. Throughput is dismal.

With /predict/batch, the consumer batches into chunks of (say) 500 rows. Now predict_proba runs on a 500-row matrix — and sklearn’s vectorised linear algebra makes it ~10–50x faster than the equivalent 500 single-row calls. You also pay HTTP overhead once per 500 rows instead of once per row.

The trade-off: latency. A batch endpoint typically isn’t on the critical path for a real-time user-facing request. The pattern is both endpoints: /predict for low-latency single calls, /predict/batch for offline scoring jobs.

# Simulating the throughput math
import time, numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=5000, n_features=20, random_state=0)
model = RandomForestClassifier(n_estimators=50).fit(X, y)

X_score = np.random.rand(2000, 20)

# 2000 single-row calls
t0 = time.time()
for row in X_score:
    model.predict_proba(row.reshape(1, -1))
single = time.time() - t0

# One batched call
t0 = time.time()
model.predict_proba(X_score)
batched = time.time() - t0

print(f"2000 single rows: {single*1000:.1f} ms")
print(f"1 batched call : {batched*1000:.1f} ms")
print(f"speedup        : {single/batched:.1f}x")

You can run that locally — typical speedups are 10–100x for tree ensembles, 5–20x for linear models. Worth a batch endpoint.

Health endpoints — liveness vs readiness

There are two distinct questions a load balancer or Kubernetes wants to ask:

The distinction matters. If you map both to the same endpoint, K8s will restart your pod the moment the model takes too long to load — instead of just waiting and routing traffic to other ready pods. Make /readyz check that the model exists in app.state. Make /healthz always return 200 unless your process is genuinely sick.

A runnable inference-loop sanity check

You can’t run uvicorn in the browser, but you can verify the logic that lives inside your FastAPI handler — i.e. the model loads and predicts on a sample row.

# Simulates the FastAPI predict handler without the HTTP server.
import joblib, tempfile, os
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# 1) train + persist (this is your CI artifact)
X, y = make_classification(n_samples=500, n_features=8, random_state=0)
model = RandomForestClassifier(n_estimators=50, random_state=0).fit(X, y)
with tempfile.TemporaryDirectory() as tmp:
    path = os.path.join(tmp, "model.joblib")
    joblib.dump(model, path)

    # 2) startup (the lifespan event in real FastAPI)
    loaded = joblib.load(path)

    # 3) single-row predict — same logic as your FastAPI handler
    features = [0.1, -0.2, 0.5, 1.1, -0.3, 0.4, 0.2, -0.7]
    proba = loaded.predict_proba([features])[0]
    print({"prediction": int(proba.argmax()), "proba": float(proba.max())})

    # 4) batch predict — same logic as /predict/batch (seeded for reproducibility)
    rng = np.random.default_rng(0)
    rows = rng.random((5, 8)).tolist()
    probas = loaded.predict_proba(rows)
    for p in probas:
        print({"prediction": int(p.argmax()), "proba": round(float(p.max()), 3)})

{'prediction': 1, 'proba': 0.74}
{'prediction': 0, 'proba': 0.68}
{'prediction': 1, 'proba': 0.54}
{'prediction': 0, 'proba': 0.72}
{'prediction': 1, 'proba': 0.54}
{'prediction': 0, 'proba': 0.5}

The first line is the single-row handler: a confident prediction: 1 at proba: 0.74. The five lines below are the batch path, the exact same model call run over a 5-row matrix instead of one row at a time — and that sameness is the whole point. The logic inside /predict and /predict/batch is identical; only the shape of the input changes. Once this passes outside any server, wrapping it in the FastAPI handlers above is mechanical — the hard part was never the HTTP, it was loading the model once and predicting on the right shape.

Run it locally

# Install
uv add fastapi uvicorn[standard] scikit-learn joblib pydantic

# Run
uvicorn app.main:app --reload --port 8000

# Hit it
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": [0.1, -0.2, 0.5, 1.1, -0.3, 0.4, 0.2, -0.7]}'

# Auto-generated interactive docs
open http://localhost:8000/docs

In one breath

FastAPI turns a .joblib file into a production REST service, and the gap between the 20-line naive version and the 60-line shippable one is four things juniors miss: Pydantic request schemas that reject junk at the framework boundary (422, not 500), the lifespan context manager that loads the model once on startup and attaches it to app.state (never per-request, the classic 2-AM incident), a /predict/batch endpoint that scores a matrix 10–100× faster than one-row-at-a-time, and split /healthz//readyz health checks so slow startup isn’t mistaken for a sick pod.

Practice

Before the quiz, return to the 2-AM story that opened the lesson — every request reloading a 380 MB model from disk. Name the exact one-line fix and say where that line moves to. Then reason about health checks: if you wire both liveness and readiness to the same endpoint that only returns 200 once the model is loaded, what failure mode do you create on a pod whose model takes 30 seconds to load, and why is the two-endpoint split the cure?

A question to carry forward

Look back at the two things that made our service fast: we loaded the model once, and we wrote a separate batch endpoint so callers could amortise the per-row cost. Both are real wins — and both are manual. The caller has to decide to batch, has to hit the right endpoint, has to chunk into 500s. The server just does what it’s told, one-model-per-process, scaled by hand with more uvicorn workers.

But the expensive truth of production serving is that the requests arriving one-at-a-time on /predict are exactly the ones that would run 50× cheaper if the server quietly grouped the ones landing within the same few milliseconds and ran them as one matrix — without the caller knowing or a second endpoint existing. So the question to carry forward is: what if the serving layer did the batching for you, hosted several models at once, and scaled itself on GPU load? That is BentoML and Ray Serve, purpose-built model servers, and it is the next lesson.