LLM serving ops

At 3 a.m., your support copilot gets featured in a product launch. Traffic jumps from 20 requests per second to 50. The autoscaler notices. It asks for four more GPU replicas.

The replicas are still downloading weights eight minutes later.

Meanwhile, users see a spinner, the queue grows, and the on-call engineer discovers that CPU utilization is a comfortable 18 percent. The dashboard looks healthy in the way a parked ambulance looks healthy.

An LLM endpoint needs the same disciplines as any production service: capacity planning, monitoring, load testing, autoscaling, and failure handling. But the unit of work is different. A request consumes input tokens, output tokens, GPU memory, and time. A replica may require tens or hundreds of gigabytes of weights. GPUs are expensive and sometimes unavailable on demand.

That combination changes the operating model. You cannot treat an LLM server as a stateless HTTP process that appears in two seconds.

Cold starts break reactive autoscaling

A replica is one running copy of the model server. A normal web replica can pull a small container, start, and pass readiness in seconds. An LLM replica must:

1. Find an appropriate GPU, sometimes in a scarce cluster.
2. Pull its container and model files, potentially many gigabytes.
3. Load the weights into GPU memory.
4. Initialize CUDA libraries, kernels, memory pools, and the inference runtime.
5. Run a readiness or warm-up request.

Timing depends on the model, hardware, image cache, storage, and serving stack. A warm image on an available GPU may start quickly, but a fresh replica taking two to five minutes is common. Provisioning the GPU first can take longer.

The problem is feedback lag. A reactive autoscaler sees a large queue, requests capacity, and then waits for that capacity to exist. During that interval, demand and service capacity are different curves:

Use a warm pool of ready replicas for latency-sensitive traffic, and scale predictively from schedules, reservations, campaigns, or leading traffic indicators. Make startup faster with cached images, fast model storage, efficient weight loading, and readiness checks that prove the model is usable. Scale down slowly and drain replicas before removing them; killing active streams turns a capacity problem into an error spike.

Scale-to-zero is a cost-versus-latency choice. It suits occasional, tolerant jobs, but is a poor default for chat with a two-second TTFT objective. Keep strict-SLO models warm and route long-tail or low-priority models to a shared pool, batch queue, or slower fallback. Ask which users are allowed to pay restart latency.

Choose a scaling signal that describes the work

CPU utilization is not a reliable measure of GPU inference capacity. CPU may be idle while the GPU is full, or busy tokenizing while the GPU waits. It matters for tokenizers, routers, and network paths, but should not be the only GPU scaling signal.

• Queue depth shows how much work is waiting.
• Queue age shows whether the oldest request is nearing its deadline.
• GPU utilization shows device activity, not whether the user SLO is met.
• In-flight sequences or tokens capture scheduling and memory pressure better than request count when request sizes vary.
• Goodput measures useful work completed inside latency targets. Use it as an SLO-aware outcome metric, alongside leading controls such as queue age, admitted token rate, active sequences, and KV-cache pressure.

A practical autoscaler uses several signals: queue age for rapid scale-up, smoothed token or GPU pressure to maintain capacity, and scheduled minimums for known peaks. Scale down over a longer quiet period than scale up to avoid thrashing. Use separate targets for models and traffic classes; a seven-billion-parameter and seventy-billion-parameter model are not interchangeable utilization numbers.

The control loop is:

1. Measure arrivals, queue age, in-flight tokens, latency, and goodput.
2. Compare them with targets for each model and traffic class.
3. Add warm capacity before queue age violates the SLO.
4. Mark replicas ready only after weights and runtime initialization finish.
5. Stop sending work before scaling a replica down.
6. Recheck cost, quality, and errors after policy changes.

Capacity planning with Little’s Law

Autoscaling reacts to the present. Capacity planning asks whether the fleet can handle the workload at all.

Little’s Law says that the average number of items in a stable system equals arrival rate multiplied by the average time each item spends in that system:

L = λ × W

L is average in-flight requests, λ is completed-arrival rate, and W is average time in the system. Every request occupies one place for its duration, so arrivals per second multiplied by seconds occupied gives requests occupied.

Consider the support copilot:

• Arrival rate: 50 requests per second.
• Average time in the system: 8 seconds, including queue time, prefill time to first token, and post-first-token decoding. TTFT, or time from arrival to first token, includes queue time here; it is not an extra term.
• Average total system concurrency: 50 × 8 = 400 in-flight requests, including queued requests.
• A separate queue-free benchmark reports 32 active, SLO-meeting service slots per GPU.

The 400 is total system concurrency, not automatically active GPU concurrency. Queued requests increase L but do not consume active service slots. Dividing 400 by 32 is valid only when 400 is independently established as the target active concurrency and 32 was measured for the same workload, hardware, and latency target. If the eight-second observation came from a fleet already queueing, the calculation is circular: queueing increases W, which increases L.

Under that condition, the first-order GPU count is ceil(target_active_concurrency / measured_active_slots_per_gpu), or ceil(400 / 32) = 13, before headroom:

import math

arrival_rate = 50          # requests per second
avg_time = 8.0             # seconds in system: queue + prefill + decode
system_in_flight = arrival_rate * avg_time

in_flight = 400.0          # independently measured target active concurrency
per_gpu = 32                # active SLO-meeting slots per GPU, same benchmark

gpus = math.ceil(in_flight / per_gpu)

print(f"total system concurrency (L = lambda * W): {system_in_flight:.0f}")
print(f"target active concurrency: {in_flight:.0f}")
print(f"GPUs at {per_gpu} active SLO-slots each: {gpus}")

total system concurrency (L = lambda * W): 400
target active concurrency: 400
GPUs at 32 active SLO-slots each: 13

Thirteen GPUs is a sanity check, not a launch guarantee. The number 32 must come from a benchmark using your model, hardware, context and output lengths, batching policy, and latency target. If active concurrency is not independently known, size from sustainable per-GPU request and token throughput, then validate with a token-distributed load test.

Capacity is token-shaped. Two workloads with the same requests per second can have very different prefill work, decode time, and KV-cache pressure when their prompt and output lengths differ.

Little’s Law has important boundaries:

• It describes averages over a stable interval, not sudden bursts or worst cases.
• It counts time in the whole system; using only decode time undercounts concurrency.
• It gives request concurrency, not GPU memory capacity. Active sequences consume KV-cache memory, and long contexts can cause OOM before arithmetic says the GPU is busy.
• Measured service capacity may fall sharply near memory or latency limits.

Use it to catch impossible plans, then validate with a token-aware load test. Add headroom for bursts, failed GPUs, deployments, and the cold-start interval.

Separate online, batch, and edge paths

A chat request and a nightly labeling job should not compete for one queue.

Online serving keeps capacity available because users experience queueing directly. Streaming makes waiting visible but does not make generation faster. Track TTFT, inter-token latency, completion latency, and abandonment.

Batch serving trades immediate response for efficiency. A durable asynchronous queue can collect, retry, and schedule work when capacity is cheap. Give batch work a deadline and failure policy; “run eventually” is not an SLO. Store inputs and idempotency keys so retries do not duplicate labels or charges, and protect the warm online reserve.

Edge serving puts a smaller model on a device or nearby server. It can reduce network latency and keep data local, but brings smaller contexts, weaker capability, update logistics, and device-specific failures. A tiered design can send difficult cases to a hosted model only when policy permits that fallback.

The path is a product decision. A password-reset request should not wait behind 100,000 offline classifications because they share an endpoint.

Load-test the tokens, then break the service

Requests per second alone is a misleading load-test input. Each request has input and output lengths; input also affects context memory. A test with short prompts and 20-token outputs can make a badly provisioned service look excellent.

Build a workload distribution from production or a reviewed proxy:

• input-token buckets for short chat, retrieved context, and long documents;
• output-token buckets, because short and long generations occupy the decoder differently;
• concurrency and arrival patterns, including bursts;
• cache hits and misses when prompts or prefixes are cached;
• cancellations, retries, rate limits, and provider errors.

Measure p50, p95, and p99 TTFT, inter-token latency, completion time, queue age, tokens per second, goodput, GPU memory, utilization, and cost per successful request while the system is busy. An idle p99 is decorative.

Exercise realistic failures: isolate a GPU, make a backend time out, exhaust the queue, deploy with active streams, lose a provider or region, submit an unusually long context, and replay a burst during a cold start.

Each needs an explicit response. An AI gateway can route to a fallback model or provider. A circuit breaker stops sending requests to a timing-out backend so it can recover. Graceful degradation may use a smaller model, cached answer, queued low-priority job, or concise partial result. It must not silently turn a high-stakes answer into an unreviewed guess.

Failure modes you can spot from the first symptom

The queue grows while GPUs are busy. Rising TTFT and timeouts indicate more work than measured SLO capacity, often because outputs are longer than assumed. Check queue age and token rates, cap or route long generations, shed low-priority work, and add capacity if the load is real.

TTFT rises while GPU utilization is moderate. CPU tokenization, network transfer, long prefill, synchronization, or batching may be waiting. Split latency into queue, prefill, and decode before adding GPUs.

Workers die with out-of-memory errors at high concurrency. Active contexts and simultaneous sequences consume KV-cache memory. Lower context or concurrency limits, reserve runtime memory, or route long contexts separately. A request-count queue limit that ignores token length is not a real memory limit.

Replica counts flap. Noisy metrics, aggressive scale-down, or delayed GPU provisioning can make the controller chase its past decisions. Smooth signals, separate scale-up and scale-down windows, keep a warm minimum, and count pending capacity.

Infrastructure is green but quality degrades. A weaker fallback, truncated prompt, or changed route can preserve latency and error metrics while harming answers. Track model and route labels with evaluations, refusal rates, tool success, user feedback, and sampled review.

Quick check

Next

Serving ops leans on the AI gateway for fallback and governance, inference metrics for goodput, and circuit breakers for resilience. For the architecture underneath online capacity, revisit disaggregated serving.