An inference server has high GPU utilization but poor p99 latency for short requests. How would continuous batching, sequence scheduling, prompt length, output length, and KV-cache memory explain the behavior, and which scheduler changes would you try first?
High GPU utilization can coexist with poor p99 because long prefills, long decodes, and KV-cache pressure keep the GPU busy while short requests wait. I would first add token-aware admission and batch budgets, prioritize latency-sensitive decode work with aging, chunk long prefills, and reserve KV-cache headroom.
How to think about it
High GPU utilization with poor short-request p99 usually means the GPU is busy doing useful work while requests wait behind long prefills, long decodes, or KV-cache capacity limits; utilization is not a latency SLO. I would first use token-aware continuous batching with bounded batch budgets, protect short or deadline-sensitive requests, and prevent KV pressure from causing preemption or swapping.
Why utilization can look healthy while p99 is awful
The GPU reports how busy its execution units or memory system are. It does not report how long a request sat in a queue.
p99 latency is the latency below which 99 percent of requests finish. The remaining 1 percent are the tail, and that tail is usually where production complaints live. A service can have a 40 ms median and a 1.8 second p99. The GPU can be at 95 percent utilization in both cases.
The reason is queueing. As the system approaches its throughput limit, a small burst of work creates a disproportionately large queue. The GPU stays busy draining old work, while a newly arrived short request waits for its turn. A GPU at 98 percent is not a customer-service metric.
For an LLM server, two kinds of work matter:
- Prefill processes the input prompt and builds the first KV cache entries. A 4,000-token prompt is a large burst of computation.
- Decode generates output one token at a time. Each active sequence usually contributes one next-token position per decoding iteration.
A sequence is one request currently being generated. A sequence scheduler decides which active sequences receive GPU work in each iteration, which new requests enter, and how many prompt or output tokens fit in the next batch.
With static batching, a batch is assembled and often advances as a group. Continuous batching, also called iteration-level batching, makes a new scheduling decision at each generation iteration. When one request finishes, another can take its place without waiting for every request in the old batch to finish.
That removes a major source of wasted capacity. It does not make long requests cheap.
A long output still occupies a sequence slot for hundreds or thousands of decode iterations. A long prefill can still consume most of an iteration’s token budget and delay latency-sensitive decode work. Continuous batching improves the average use of the GPU, but without QoS rules it can improve throughput while making the tail worse.
Prompt length, output length, and KV cache
Prompt and output length hurt latency in different ways.
A longer prompt mainly hurts time to first token because the server must process those input tokens before generation can begin. It also allocates more KV cache.
A longer output mainly hurts time to last token because generation is sequential. The request must receive another decode iteration for every generated token. Its KV cache also grows as those tokens are produced.
The KV cache stores the attention keys and values for tokens that have already been processed. Reusing them avoids recomputing the entire history on every decode step. The price is memory.
For a simplified model with 32 layers, 8 KV heads, head dimension 128, and bfloat16 values, the KV memory per token is approximately:
2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes
The first factor of 2 is for keys and values. That is 128 KiB per token, before allocator and metadata overhead. A 4,096-token sequence therefore needs about 512 MiB of KV cache. Thirty-two such sequences need about 16 GiB.
Real models vary. Multi-query and grouped-query attention reduce KV heads, while different precision and allocator choices change the result. The important point is that cache usage grows with total sequence length, not merely with the number of requests.
When KV memory is nearly full, the scheduler has fewer choices. It may delay admission of a short request, preempt an active sequence, recompute its cache later, or move cache blocks between GPU and host memory, depending on the server. Those actions can produce a large latency spike even if the GPU remains busy.
A concrete scenario
Suppose an interactive endpoint receives requests with 128 input tokens and an expected 32 output tokens. When the server is quiet, the trace shows about 40 ms end-to-end latency.
Now 32 batch-oriented requests arrive. Each has a 3,072-token prompt and may generate 1,024 tokens. The scheduler admits them because the GPU is capable of running 32 active sequences. Their initial prefills consume substantial compute, and their decodes continue for up to 1,024 iterations.
A short request arrives while all 32 sequence slots are occupied. Continuous batching helps only if the scheduler has room in its active-sequence and token budgets. If it has a hard limit of 32 active sequences, the short request waits for a long request to finish. That could mean hundreds of decode iterations.
Even if a slot is available, the long prefills can delay it. A scheduler that admits a 3,072-token prefill as one large unit may run that work before scheduling the short request’s first token. The GPU is highly utilized. The short request’s p99 is still poor.
Now consider the cache. Using the illustrative model above, each long request can reach 4,096 total tokens and consume roughly 512 MiB. Thirty-two of them consume roughly 16 GiB. If only 16 GiB remains after model weights and workspaces, there is no practical headroom for the new 160-token request, whose cache alone is about 20 MiB before overhead. The scheduler must wait, evict, preempt, or recompute.
A useful production trace would separate these times:
| Measurement | What it reveals |
|---|---|
| Queue wait | Whether the request is waiting for admission or a scheduling turn |
| Prefill time and prompt tokens | Whether long inputs are blocking first-token latency |
| Decode time and output tokens | Whether long generations occupy capacity |
| KV occupancy and allocation failures | Whether memory limits are shaping scheduling |
| Inter-token latency | Whether active generations are being starved between tokens |
Scheduler changes I would try first
First, I would make the scheduler token-aware, rather than limiting only the number of requests. Set a maximum number of tokens processed per iteration and a maximum number of running sequences. Also reserve KV-cache headroom for arrivals. A batch of eight 4,000-token prompts is not equivalent to a batch of eight 100-token prompts.
Second, I would give interactive decode work a latency-aware priority over large prefills, while using aging so that batch jobs cannot starve forever. A practical policy might admit only a bounded number of prefill tokens per iteration, then schedule pending decode tokens and newly arrived short requests. The exact controls vary by server, but the principle is stable: do not let one giant prefill monopolize an iteration.
Third, I would use chunked prefill, which splits a long prompt into smaller pieces that can be interleaved with decode work. This usually improves time to first token and inter-token latency for other requests because the scheduler regains control more often. It can cost some throughput through extra scheduling and kernel overhead.
Fourth, I would add workload separation if the SLO matters. Interactive requests and unconstrained batch generation often deserve separate queues, GPU pools, or at least separate capacity reservations. A single shared queue is simpler and cheaper, but it makes a 3 a.m. interactive request compete with a document summarization job that wants 8,000 output tokens.
I would also enforce sensible prompt and output limits. Reducing a maximum output from 4,096 to 1,024 tokens does not improve every request’s latency, but it bounds how long a sequence can occupy capacity and how large its cache can grow. Prefix caching can reduce repeated prompt work when requests share an exact prefix, but it should be treated as an optimization, not as permission to ignore worst-case cache demand.
The trade-off and the common wrong answer
The tempting answer is “lower the batch size.” That may improve p99 by reducing contention, but it can waste GPU capacity and lower throughput. The better control is usually a token budget and a scheduling policy that distinguishes prefill from decode.
Likewise, shortest-job-first can make short requests fast while starving long ones. Use aging, weighted fairness, or deadlines. Protect the interactive tail without pretending batch work is free.
One common failure mode is visible in the metrics: GPU utilization remains above 90 percent, KV occupancy sits near its limit, and p99 suddenly jumps while median latency barely moves. If traces show preemptions, cache transfers, or recomputation, the problem is not insufficient GPU utilization. It is cache pressure and admission policy.
What they’ll ask next
How would you tell whether prefill or decode is the problem?
Compare time to first token with inter-token latency and group traces by prompt and output length. High first-token latency points toward queueing or prefill contention. High inter-token latency points toward decode saturation, scheduling interference, or cache pressure.
Would you always prioritize short requests?
No. I would prioritize latency-sensitive traffic with an explicit policy and aging. Unbounded short-job priority can starve long requests and create an unfair system.
Why not add more GPU replicas?
More replicas reduce contention, but they cost money and may not fix a single overloaded queue or a poor batching policy. I would first confirm whether queue wait, token budget, or KV capacity is the bottleneck, then scale out if the required SLO still cannot fit within one worker’s safe operating range.
One line to say in the room
“High utilization tells me the GPU is busy, not that requests are being served fairly, so I would bound token and KV-cache occupancy, schedule decode-sensitive work with aging, and use chunked prefill before reaching for a smaller batch.”