BentoML, Ray Serve, and friends — when FastAPI isn't enough

The last lesson left us with a service that worked but batched by hand — a separate endpoint, the caller deciding to chunk into 500s, one model per process scaled with more uvicorn workers. We ended on the expensive truth it exposed: the single requests arriving on /predict are exactly the ones that would run 50× cheaper if the server quietly grouped the ones landing within a few milliseconds and ran them as one matrix — without the caller knowing. We asked what it would take for the serving layer to do that batching for you. This lesson is the answer, and the honest boundaries of when you actually need it.

A FastAPI service running a sklearn model behind nginx is a totally respectable production setup. Plenty of companies have shipped exactly that and made millions. So the question isn’t “is FastAPI good enough?” — it usually is. The question is “what specific problem am I about to hand-roll, and is there a library that already solves it?”

There are five problems that FastAPI doesn’t solve on its own. Once two or more of them are on your roadmap, look at a dedicated serving framework.

When FastAPI starts to hurt

Each of these is “doable” in FastAPI. None of them is trivial. If your service does inference at sub-100ms latency, has multiple models, or needs GPU efficiency, you’re going to write a small version of one of these frameworks. Better to use the real one.

BentoML — the service abstraction

BentoML’s pitch: a service is a class. Methods are endpoints. Decorators wire up batching, async, and model loading.

# service.py — BentoML 2.x API
import bentoml
import numpy as np

# 1. Decorate a class to declare it as a BentoML service.
@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class ChurnService:
    # 2. Load the model once at class instantiation (startup), not per request.
    def __init__(self):
        self.model = bentoml.sklearn.load_model("churn_classifier:latest")

    # 3. Endpoint — type annotations drive validation and the OpenAPI schema.
    @bentoml.api
    def predict(self, input_arr: np.ndarray) -> np.ndarray:
        return self.model.predict_proba(input_arr)

That’s the minimum. Three things to notice:

• @bentoml.service is the key abstraction. BentoML can run multiple replicas of this class, hand them GPUs, and batch incoming requests automatically — you describe the model, not the HTTP plumbing.
• Adaptive batching for free. Add @bentoml.api(batchable=True) and BentoML collects concurrent requests, batches them, and fans the responses back out. You don’t write a queue. You don’t tune it manually. It adapts to live traffic.
• The bento itself. bentoml build packages the code + the model artifact + the dependency lockfile into a “bento” — a versioned, hashable bundle. bentoml containerize turns that into a Docker image.

The mental shift from FastAPI is: in FastAPI you wrote the HTTP layer and treated the model as a Python object on app.state. In BentoML you describe the model as a service class and let the framework handle the HTTP, batching, and scaling.

Ray Serve — when the whole pipeline is a graph

Ray Serve sits on top of Ray (the distributed-compute framework). Its sweet spot is multi-step inference pipelines: a preprocessing service feeds an embedding service which feeds a ranker which feeds a business-rules layer. Each step is its own deployment, scaled independently, and connected with method calls that may cross machines.

# Ray Serve sketch
from ray import serve

@serve.deployment(num_replicas=4)
class Featurizer:
    def __call__(self, raw):
        return preprocess(raw)

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Ranker:
    def __init__(self):
        self.model = load_model()
    def __call__(self, features):
        return self.model.predict(features)

@serve.deployment
class Pipeline:
    def __init__(self, featurizer, ranker):
        self.featurizer = featurizer
        self.ranker = ranker
    async def __call__(self, raw):
        features = await self.featurizer.remote(raw)
        return await self.ranker.remote(features)

# Compose
app = Pipeline.bind(Featurizer.bind(), Ranker.bind())

Ray Serve’s killer feature is heterogeneous replication: the featurizer needs CPUs and lots of replicas; the GPU ranker is expensive and you want exactly two of them. Hand-rolling that in FastAPI means K8s manifests + service-to-service calls + your own batching. In Ray Serve it’s the deployment decorator.

If you’re not running multi-stage inference, Ray Serve is overkill.

Other serious options

The pattern across all of them: they pick a different axis of pain to solve. Triton solves “GPUs are expensive, batch them properly.” KServe solves “I want this to look like any other K8s workload.” Modal solves “I never want to touch K8s.” Pick the axis that hurts you most.

The honest “should I migrate?” framework

You’re on FastAPI. Migrate when two or more of these are true:

1. Tail latency (p99) matters and you’re not micro-batching.
2. You have more than ~3 models in one service.
3. You have GPUs and you can’t keep them busy.
4. Your traffic is bursty (10x peaks) and you autoscale on CPU only.
5. You’re rewriting the same “load model from registry by name” code into every service.

If only one is true, you can probably keep FastAPI + write the one thing yourself. If three or more are true, you’ve already half-written a serving framework — switch to one that’s been hardened by other teams.

A runnable smoke test — same model, different wrappers

You can’t run BentoML’s HTTP server in the browser, but you can verify the model wrapper logic — turning a single inference into a batched one — works against the same sklearn artifact. This is the contract between your model code and any serving framework.

import time
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# 1) Train the same way any framework would consume.
X, y = make_classification(n_samples=2000, n_features=8, random_state=0)
model = RandomForestClassifier(n_estimators=80, random_state=0).fit(X, y)

# 2) A tiny 'runner' abstraction: take a list of single requests, batch them.
class BatchedRunner:
    """The pattern BentoML / Ray Serve apply automatically."""
    def __init__(self, model):
        self.model = model
    def predict_batch(self, rows):
        return self.model.predict(np.array(rows)).tolist()

requests = np.random.default_rng(0).random((200, 8)).tolist()   # 200 incoming requests
runner = BatchedRunner(model)

# Single-request mode (what hand-rolled FastAPI does row-by-row)
t0 = time.time()
out_single = [int(model.predict(np.array(r).reshape(1, -1))[0]) for r in requests]
single = (time.time() - t0) * 1000

# Batched mode (what BentoML/Ray Serve auto-batching does for you)
t0 = time.time()
out_batched = runner.predict_batch(requests)
batched = (time.time() - t0) * 1000

print("predictions identical:", out_single == out_batched)
print(f"single-request: {single:6.1f} ms")
print(f"batched      : {batched:6.1f} ms")
print(f"speedup      : {single/batched:.1f}x")

predictions identical: True
single-request:  289.7 ms
batched      :    1.7 ms
speedup      : 172.5x

Two numbers, two different kinds of truth. predictions identical: True is the guaranteed part — the same model on the same inputs gives the same answers whether you feed it one row at a time or all 200 at once, so a serving framework can batch behind your back with zero change in correctness. The timings are one machine’s numbers and will differ on yours — but the magnitude, a roughly hundred-fold gap between row-by-row and batched on a 200-row tree ensemble, is real and repeatable, because it is an algorithmic win (vectorised matrix math plus amortised per-call overhead), not a fluke. That gap is the entire economic case for a serving framework: it applies this batched code path automatically across concurrent HTTP requests, without you ever writing the queue.

In one breath

FastAPI is a respectable production setup, so you graduate to a dedicated serving framework not for prestige but when two or more specific pains appear — adaptive micro-batching, several models in one process, version routing, model-load-aware autoscaling, GPU scheduling — with adaptive batching the single feature most worth migrating for (a 5–20× GPU throughput win that’s painful to hand-roll); BentoML makes a service a class and batches for free, Ray Serve shines for multi-stage pipelines with heterogeneous replicas, and Triton/KServe/Modal/vLLM each solve a different axis of pain — so pick the one matching the axis that actually hurts, and stay on FastAPI if only one trigger is true.

Practice

Before the quiz, apply the migration framework to two teams. Team A: one sklearn model, no GPU, ~10 requests per second. Team B: a GPU transformer with bursty 10× traffic peaks, three model versions live at once, and a p99 SLA. For each, say whether they should stay on FastAPI or migrate, and which one trigger decides it. Then the batching insight: the demo showed batched inference is identical in output but ~100× faster — so why can’t you just always call the batch endpoint, and what does the framework do that a hand-written batch endpoint cannot?

A question to carry forward

Notice the unexamined assumption running under this whole lesson — and the last one. Every framework here, every micro-batching queue and GPU scheduler, exists to make a request-time prediction faster: a user is waiting, milliseconds matter, batch the bursts. We have spent two lessons optimizing the live path without once asking whether the prediction needs to be live at all.

And for most predictions, it doesn’t. A churn score, a lead ranking, a recommendation list — none of these change in the instant a user loads the page; they could have been computed at 3 a.m. over the whole table and simply looked up. So the question to carry forward steps back from “how do I serve this request fast?” to the cheaper, prior one: does this prediction need to happen in real time at all, or can it be precomputed in a batch job and served from a lookup? That choice — batch versus real-time inference — is one of the highest-leverage cost decisions in MLOps, and it is the next lesson.