LLM serving ops

An LLM endpoint is still a production service: it needs autoscaling, capacity planning, load testing, monitoring, and resilience like anything else. But several LLM-specific facts break the standard SRE playbook. Weights are gigabytes, so a fresh replica takes minutes to become ready. The hardware is scarce and expensive, so you can’t just over-provision. And capacity is measured in tokens and concurrent streams, not requests-per-second. Serving ops for LLMs is the normal discipline plus those twists.

Cold starts break autoscaling

A normal web service scales by adding stateless replicas in seconds. An LLM replica must pull a multi-GB image, download or load multi-GB weights onto the GPU, and warm its caches — often 2–5 minutes — and that’s after a GPU is even available, which during a cluster crunch may itself take minutes. So when traffic spikes, capacity cannot follow instantly, and the gap is dropped or badly-queued requests:

The mitigations: keep a warm pool of ready replicas as headroom, scale predictively (on schedules and leading indicators, not just current load), speed cold starts (cached images, fast weight loading, snapshotting), and be very careful with scale-to-zero — saving money idle can mean a multi-minute stall on the next request.

And scale on the right signal. CPU utilization is meaningless for a GPU service. Scale on queue depth, GPU utilization, in-flight tokens, or — best — goodput against your SLO.

Capacity planning with Little’s Law

How many GPUs do you actually need? Little’s Law answers it: the average number of requests in flight is the arrival rate times the average time each spends in the system, L = λ × W. Convert that concurrency into GPUs using how many SLO-meeting streams one GPU sustains:

arrival_rate = 50      # requests/sec  (lambda)
avg_latency  = 8.0     # seconds in system per request (W) = TTFT + decode time
per_gpu      = 32      # concurrent streams one GPU sustains within SLO (from goodput)

concurrency = arrival_rate * avg_latency          # Little's Law: L = lambda * W
gpus = -(-int(concurrency) // per_gpu)            # ceil division

print(f"in-flight requests (L = lambda*W): {concurrency:.0f}")
print(f"GPUs at {per_gpu} SLO-slots each : {gpus}  (before spike/cold-start headroom)")

in-flight requests (L = lambda*W): 400
GPUs at 32 SLO-slots each : 13  (before spike/cold-start headroom)

Fifty requests a second, each living eight seconds, means 400 concurrent in flight — 13 GPUs at 32 SLO-slots each, plus headroom for spikes and cold starts. The per-GPU number comes straight from your goodput measurement, which is why you measure it first.

Two paths: online, batch, and edge

Not all traffic is interactive. Split the workload:

• Online — latency-sensitive, user-facing. Optimize TTFT/ITL/goodput, keep warm capacity, stream responses.
• Batch — bulk, non-urgent work (nightly summarization, dataset labeling). Use the providers’ async batch APIs (often ~50% cheaper) or your own queue: throughput matters, latency doesn’t, so you can pack big batches and run off-peak.
• Edge — push a small model close to the user for latency or privacy, accepting lower quality. A tiered design runs the edge model for easy/sensitive cases and falls back to a hosted large model for the rest.

Test it, and break it on purpose

• Load test at token granularity. RPS alone lies — an LLM’s cost depends on prompt and output lengths. Drive load with realistic prompt/length distributions and measure TTFT, ITL, and goodput at the p95/p99 tail under load, not just averages at idle.
• Chaos and graceful degradation. Things will fail: a provider has an outage, a GPU dies, a region goes dark. Design the failure modes deliberately — fail over via the AI gateway, trip circuit breakers on a sick backend, and degrade gracefully: downgrade to a smaller model, shed or queue low-priority load, and return a useful partial answer rather than a hard error.

In one breath

• An LLM endpoint is a normal service plus hard twists: GB weights → minutes-long cold starts, scarce GPUs, and capacity measured in tokens/streams, not RPS.
• Cold starts break reactive autoscaling — capacity can’t follow a spike — so keep warm headroom, scale predictively, speed weight loading, and scale on queue depth / GPU-util / goodput, never CPU.
• Capacity planning uses Little’s Law (L = λ·W): 50 req/s × 8s = 400 in flight ≈ 13 GPUs at 32 SLO-slots each, plus headroom.
• Split traffic into online (latency, warm), batch (async APIs, ~50% cheaper), and edge (small local models); load-test at token granularity with realistic length distributions.
• Design failure on purpose — gateway fallback, circuit breakers, graceful degradation — and dashboard tokens/cost, TTFT/ITL/goodput, queue/GPU, and quality drift.

Quick check

Next

Serving ops leans on the AI gateway for fallback and governance, inference metrics for the goodput you scale on, and circuit breakers for resilience. For the serving-architecture win underneath it all, revisit disaggregated serving.