Continuous batching and the inference scheduler

A support chatbot receives eight requests at 3 a.m. Seven users want a short answer. Each response takes about 40 tokens. One user asks for a detailed report and gets 900 tokens.

A static batch starts all eight requests together and keeps them together. After 40 decoding steps, seven responses are finished. Their execution lanes sit empty while the eighth request generates another 860 tokens.

The GPU is still working. The useful work has simply collapsed from eight sequences to one.

This is especially wasteful for language models because generation is both variable-length and autoregressive: each new token depends on the token generated immediately before it. A batch cannot know in advance which request will finish first, and a conventional batch usually waits for the slowest member before it is retired.

The fix is continuous batching, also called in-flight batching. A scheduler treats each active request as an independent sequence. Whenever one finishes, the scheduler admits another queued request into that newly available lane instead of waiting for the whole batch to finish.

That small scheduling change is one of the largest throughput levers in LLM serving.

Static batching leaves slots behind

A slot is one active sequence position in a model invocation. An eight-slot batch can generate one next token for up to eight requests per decoding step.

Suppose our eight responses have these lengths:

• One response: 900 tokens
• Seven responses: 40 tokens each

The static batch runs for 900 steps because the longest response needs 900 next-token predictions. During the first 40 steps, all eight slots produce useful tokens. During the next 860 steps, only one slot does.

Here is the shape of that waste. The diagram is schematic rather than a hardware trace.

The arithmetic is blunt:

• Total available slot-time: 8 * 900 = 7,200 slot-token opportunities
• Useful output tokens: 900 + 7 * 40 = 1,180
• Ideal slot utilisation: 1,180 / 7,200 = 16.4%

This does not mean the GPU is 16.4 percent busy. Kernels, memory traffic, and attention work still run. It means only 16.4 percent of the available sequence positions produce useful output in this simplified calculation.

Continuous batching changes the unit of scheduling. The scheduler runs a decode iteration, samples the next token for active sequences, removes sequences that reach a stop condition, and fills free positions from the waiting queue. “The moment” means the next safe scheduler iteration, not an interruption halfway through a running GPU kernel.

With enough queued work, the seven free slots are refilled at step 40. Across the same 900-step window, eight slots can remain active:

• Useful output in the window: about 8 * 900 = 7,200 tokens
• Ideal slot utilisation: 100 percent
• Ideal useful throughput: 8 tokens per decode step instead of 1,180 / 900, or about 1.31

The idealised throughput ratio is about 6.1 times. Real hardware will show less because changing batches add scheduling overhead, and decode throughput depends on memory bandwidth, kernel efficiency, and the model. The wasted-capacity problem remains real.

Prefill and decode are different jobs

A request starts with prefill, the pass that reads its prompt and builds reusable attention state. A 2,000-token prompt can be processed in parallel across its positions, subject to the attention implementation.

Then comes decode, which generates one new token at a time. Each step reads the prior state, predicts a token, appends its key and value to the KV cache, and repeats.

The phases have different hardware profiles:

• Prefill has many input tokens and tends to be compute-heavy.
• Decode has one new token per active sequence and tends to be memory-bandwidth-heavy because it reads the growing KV cache.

A scheduler must decide how much prefill to run beside decode. If seven users are streaming answers when a new user arrives with a 4,000-token document, processing that entire prompt in one operation can make the existing streams pause. Refusing to mix prefill and decode protects those streams but delays the new request’s time to first token (TTFT).

Chunked prefill divides the prompt into smaller pieces and interleaves them with decode iterations. A larger chunk finishes prefill sooner and may use the GPU more efficiently. A smaller chunk gives active streams more frequent turns and limits interruption. The right size depends on prompt lengths, output lengths, hardware, and the latency target.

A useful scheduler tracks a token budget: the maximum prompt and decode tokens it will process in one iteration. It may also cap active sequences. These constraints differ: eight short requests and eight 16,000-token requests have the same sequence count but very different compute and memory demands.

The scheduler is also a memory allocator

Model weights are only one resident object. Each active request also owns a growing KV cache containing attention keys and values for its prompt and generated tokens. The approximate cache memory per token is:

2 * number of layers * KV heads * head dimension * bytes per value

The first factor of two is for keys and values.

For a model with 32 layers, 8 KV heads, head dimension 128, and fp16 cache values:

2 * 32 * 8 * 128 * 2 = 131,072 bytes

That is 128 KiB per token, ignoring metadata and workspace. One 8,192-token sequence needs roughly 1 GiB of KV cache; ten need roughly 10 GiB. Grouped-query attention, cache quantisation, and prompt limits can therefore affect concurrency more than a simple maximum-batch setting.

Many engines use a paged KV cache, divided into fixed-size blocks assigned as a sequence grows. Paging avoids reserving one contiguous region per request and reduces fragmentation. It does not create more memory.

Admission is closer to:

Do I have enough free KV blocks and activation workspace for the prompt plus the request’s declared maximum output, or do I have an explicit preemption or offload policy?

This is conservative: it reserves for the prompt and declared maximum rather than the expected output. Reserving only for expected length may keep more requests resident when they finish early, but a request that reaches its maximum can exhaust KV capacity and trigger preemption, recomputation, swapping, or an out-of-memory failure. If that trade-off is intentional, the policy must be explicit.

The per-iteration token budget is separate. It limits work in one iteration; it does not reserve the growing KV memory required to finish admitted requests.

When the free KV pool runs out, the scheduler can leave a request queued, preempt an active request, recompute its state later by rerunning its prompt and prior tokens, or swap cache state to host memory. Recomputation spends GPU compute; swapping spends transfer bandwidth and time. Repeated eviction under heavy load can turn decoding into a cycle of rebuilding state.

Throughput has a latency price

Increasing the maximum active batch usually helps at first. More sequences amortise fixed launch overhead and keep the GPU supplied with work. Then the curve bends: a larger decode batch reads more KV data, increases iteration time, and can raise the gap between successive tokens, called inter-token latency (ITL). A large prefill in the same iteration can increase it sharply.

The useful operating point is not “the largest batch that fits.” It is the point where the service-level objective is met while throughput is high enough to use the hardware efficiently. Streaming assistants may value low ITL; offline summarisation may accept higher TTFT for better total tokens per second.

As arrival rate approaches service capacity, queueing delay grows quickly. Small bursts wait behind existing work, so p99 latency can worsen while the median remains stable. p99 latency is the value below which 99 percent of requests finish or reach the measured event.

Use the symptom to identify the bottleneck:

• A p99 TTFT spike suggests ingress queueing, prefill contention, or admission delay.
• A p99 ITL spike suggests oversized decode batches, long prefill chunks, or preemption during streaming.
• Stable TTFT with a high end-to-end tail may simply indicate unusually long outputs.
• High aggregate throughput with bad p99 means the average request is productive while some users wait.

See inference metrics for metric definitions and boundaries.

A practical scheduler pattern

A production path usually has an ingress queue, an active-sequence table, a KV-block allocator, and a loop that schedules prefill and decode.

Each request enters with a prompt, output limit, priority, and service class. The scheduler admits it only when KV blocks and token budget are available, or applies a defined overload policy. Each iteration then:

1. Selects decode sequences eligible for another token.
2. Selects prefill work within the token budget and latency policy.
3. Runs the model.
4. Appends generated tokens and KV blocks.
5. Retires stopped sequences and releases their blocks.
6. Admits waiting requests into freed capacity.

Track active sequences, queued requests, KV occupancy, free blocks, prefill tokens per iteration, decode batch size, preemptions, recomputed tokens, TTFT, ITL, and output tokens per second. “The GPU is busy” is not enough to explain a latency problem.

Choosing the right approach

Approach	Best fit	Main cost or trap
Static batching	Offline jobs with similar, known output lengths	Finished sequences hold slots until the longest job ends
Continuous batching	Online traffic with variable lengths and enough backlog	Scheduler complexity, fairness work, and KV pressure
Little or no batching	Very low traffic or strict per-request isolation	Poor GPU utilisation
Disaggregated prefill and decode	Prompt-heavy traffic interfering with long-lived streams	KV transfer, networking, routing, and operational complexity

Disaggregated serving separates prefill and decode onto different worker groups; each group can still use continuous batching. It is useful when the phases need different scaling or hardware, as described in disaggregated serving.

Continuous batching is not free. With sparse traffic there may be no queued request to admit, and static batching may have lower overhead. Online systems need admission control, memory accounting, fairness, and overload handling.

Failure modes you can recognise

First symptom	Likely cause	Fix to investigate
Throughput drops after short responses finish	Static batching or no refill at iteration boundaries	Enable in-flight replacement and verify queued admission
Out-of-memory with few active requests	Long contexts or outputs exhausted KV blocks	Use token-based admission and inspect KV occupancy
p99 TTFT jumps while average throughput stays high	Saturated prefill or admission queue	Cap or chunk prefill, rate-limit, or scale replicas
Streaming pauses when long prompts arrive	Unchunked prefill monopolises an iteration	Interleave decode with chunked prefill or separate workers
Preemption and recomputation climb	More live token state was admitted than memory can sustain	Lower concurrency and reserve KV headroom

Raising the batch limit is not a universal fix: it can worsen p99 when the queue is saturated and cannot solve exhausted KV memory.

Quick check

0/3

Q1

8.0%16.4%40.0%87.5%

Check answer

Q2

Prefill deletes their generated tokens from the KV cachePrefill prevents the model from using attentionPrefill can consume enough compute in an iteration that decode work waits behind itPrefill always uses a separate GPU and adds network latency

Check answer

Q3

Raise the sequence cap to 32Admit requests based on available KV blocks and token growth, then reduce or manage context demandDisable continuous batching so every request gets a private batchIncrease the output sampling temperature

Check answer