You enabled speculative decoding but throughput got worse for some requests. How would you reason about draft-model quality, acceptance rate, verification cost, memory bandwidth, sequence length, and batching before deciding whether to keep or remove it?

How to think about it

I would not keep speculative decoding just because it sounds clever, or remove it because a few requests slowed down. I would compare target-only and speculative decoding on identical traffic, split by context length, batch size, and request type, then keep it only where accepted tokens per verification round pay for draft inference, target verification, synchronization, and extra memory traffic.

The key question is not “What is the draft model’s perplexity?” It is “How many output tokens do I get for each expensive target-model step, and what did that step cost?”

Why speculative decoding can help

Normally, the large target model generates one token at a time. Each token requires another decoder forward pass, and the model often has to stream its weights from GPU memory again. At small batch sizes, this process is frequently limited by memory bandwidth rather than raw arithmetic.

Speculative decoding adds a smaller draft model. The draft generates, say, five tokens autoregressively. The target then examines those five proposed tokens in one verification pass. Because the target can process the proposed positions together, its matrix operations may use the GPU more efficiently.

If the target accepts most of the draft, one target verification pass can produce several output tokens. That is the win.

Draft quality matters because acceptance is conditional. A draft token must agree with what the target would produce given the exact preceding context. A small model that performs well on a general benchmark can still be a poor draft for SQL, legal text, code, or a particular language. The target does not care that the draft has a respectable average score. It cares whether the next proposed token matches its own distribution at this position.

For exact sampling, speculative decoding can use rejection sampling: a proposed token from draft distribution q is accepted according to its probability under the target distribution p, with a residual correction when it is rejected. Greedy implementations commonly accept a proposed token when it matches the target’s selected token. Either way, the implementation must preserve the intended output distribution or decoding policy. A fast system that quietly changes sampling behavior is not a free speedup.

Acceptance rate is necessary, but not sufficient

Define token acceptance rate as accepted proposed tokens divided by proposed tokens. Track it, but do not stop there.

The target usually accepts a contiguous prefix of the draft proposal. If token three is rejected, tokens four and five cannot simply be kept, because they were drafted using the wrong history. A correction token may then be emitted by the target.

That makes the average output per round lower than a naive calculation suggests. Suppose the draft proposes k = 8 tokens and each position has an independent acceptance probability of 0.70. The expected accepted prefix is approximately:

0.70 + 0.70^2 + ... + 0.70^8 = 2.20 tokens

A correction token may bring the average output to roughly 3.20 tokens per round. The independence assumption is only an approximation; real token decisions are correlated. The point is that a “70 percent acceptance rate” does not mean five or six useful tokens per target pass.

I would therefore record at least four counters:

• proposed tokens
• accepted tokens
• accepted prefix length per round
• output tokens per draft-plus-verification round

The last number is the one throughput actually feels.

Draft quality also changes with speculation length. With k = 2, a mediocre draft may still deliver a useful two-token burst. With k = 8, errors near the beginning waste the rest of the proposal. Acceptance should be measured by position: position one, position two, and so on. A sharp fall after position three is evidence for a smaller k, not necessarily for deleting speculation entirely.

The arithmetic of the decision

A useful mental model is:

speculative throughput ≈ output tokens per round / (draft time + target verification time + overhead)

Compare that with target-only throughput measured under the same batch and context conditions:

target-only throughput ≈ 1 / target time per token

Consider two illustrative profiler traces for a draft length of five. These are example measurements, not universal benchmarks.

Request class	Target-only time	Draft time	Verify time	Mean output per round	Result
2,000-token context, batch 1	1.00 ms/token	0.50 ms	0.65 ms	4.2 tokens	3.5 tokens/ms
32,000-token context, batch 1	1.00 ms/token	0.90 ms	1.20 ms	1.4 tokens	0.67 tokens/ms

The first case wins because 4.2 / 1.15 is about 3.65 tokens per millisecond, before small scheduling overheads. The second loses because 1.4 / 2.10 is about 0.67 tokens per millisecond. The draft is doing extra work, and the target still verifies almost a whole token’s worth of useful output per round.

This is why a single global acceptance-rate number is dangerous. The first request may have an 85 percent token acceptance rate and the second 25 percent. Even with the same rate, their verification costs can differ substantially.

Verification cost and sequence length

Verification is not free. The target processes several proposed positions together, but it still computes their hidden states, applies attention, writes key-value cache entries, and performs the final token selection. The target’s causal attention must also account for the existing context.

Long contexts make this more painful. Each proposed token may need to attend over a large key-value cache. The target can gain from better matrix utilization during verification, but attention still creates memory traffic proportional to the amount of context being consulted. The draft model has its own cache work as well.

Short generations have the opposite problem. If a request asks for 12 output tokens, a few draft passes, kernel launches, and synchronization points can dominate the entire request. There may not be enough generation left to recover the setup cost.

I would bucket results by prompt length and generated length. A policy might allow speculation for medium and long generations with short or moderate contexts, reduce k for long contexts, and disable it for very short completions. The thresholds should come from measurements, not folklore.

Memory bandwidth and model placement

At low batch size, target-only decoding often spends much of its time moving model weights and cache data rather than filling tensor cores. Speculation can help because one target invocation processes multiple positions, allowing more reuse of loaded weights and better arithmetic intensity.

But it adds traffic too. The draft model reads its own weights and maintains its own cache. The target reads and writes cache entries for proposed positions. If both models share one GPU, the draft can evict useful data, consume memory bandwidth, or reduce the batch size that fits in memory. If the draft runs on another device, data transfer and synchronization can erase the gain.

I would inspect hardware counters and timelines, not just GPU utilization. A GPU sitting at 60 percent utilization does not automatically mean speculation is useful. It may be stalled on memory or waiting between alternating draft and target kernels. Conversely, a target already running near its efficient compute and bandwidth limit may gain little from speculation while the draft competes for the same resources.

Batching changes the answer

Speculation often shines most clearly at batch one, where the target is underutilized and latency matters. High-throughput serving is different.

With a large batch, the target model may already process enough requests together to use the GPU efficiently. Adding a draft pass can then be pure overhead. Variable acceptance also makes batches ragged: one request may accept five tokens, another may accept one, while the scheduler must keep their work coordinated. Padding, synchronization, and continuous-batching decisions can waste the theoretical gain.

I would compare both per-request latency and aggregate service throughput. A configuration that improves one request from 40 milliseconds per generated token to 25 milliseconds but reduces total server throughput by 15 percent is not an automatic victory. For a user-facing assistant, p95 inter-token latency may matter more. For offline generation, aggregate tokens per second and cost per million tokens usually matter more.

The senior decision

I would first establish a target-only baseline, then run an A/B comparison with the same prompts, decoding parameters, hardware, scheduler, and traffic shape. I would segment by draft acceptance, context length, output length, batch occupancy, and queue delay.

The production choice is usually a gate, not a religious conversion. Keep speculation for request classes where it wins; lower the draft length where acceptance collapses; and disable it when the target is already saturated, the context is very long, or the output is too short to amortize the overhead.

A common failure mode appears first as worse p95 latency rather than an obvious average-throughput collapse. The trace shows alternating draft and target kernels with idle gaps, while the target produces only about one output token per round. That points to low acceptance or synchronization overhead. If acceptance is healthy but aggregate throughput still falls, inspect memory bandwidth, model placement, and batch fragmentation next.

What they’ll ask next

“Is a high acceptance rate enough to prove speculative decoding is helping?”
No. A high rate can still lose if the draft is slow, verification is expensive, or batching was already efficient. Measure output tokens per round divided by total wall-clock time.

“Would you increase the draft length to get more tokens per verification?”
Only while the marginal proposed token pays for its draft cost and does not cause earlier-prefix failures. Measure acceptance by position. If positions four through eight are rarely accepted, a shorter proposal is probably better.

“When would you remove it globally?”
If no meaningful workload segment beats target-only decoding after accounting for latency, throughput, memory use, and output correctness. Otherwise, route selectively rather than throwing away a useful optimization for the requests it actually helps.

One line to say in the room

“I would keep speculative decoding only where measured accepted tokens per round exceed the full cost of drafting, verification, memory traffic, and batching overhead—not where acceptance rate merely looks impressive.”