Your service meets average latency targets but users complain about sluggish streaming. Which TTFT, inter-token latency, end-to-end latency, throughput, utilization, and goodput measurements would you collect, and how would you tie them to separate interactive and batch SLOs?

How to think about it

I would measure TTFT, inter-token latency, end-to-end latency, token and request throughput, GPU and queue utilization, and SLO-qualified goodput as distributions split by workload, model, prompt length, output length, and priority. I would give interactive traffic tight p95 and p99 streaming SLOs, while batch traffic gets deadline, throughput, completion, and validity SLOs; one blended average would hide the problem.

Why the average is lying

Imagine a support copilot. A user submits a 700-token question and waits for the answer to appear. The first token arrives after 400 milliseconds, then the screen updates every 80 milliseconds. That feels responsive.

Now change the same request to a two-second wait for the first token, followed by bursts of three tokens and a 600-millisecond pause. The total answer may finish in roughly the same time. The user still calls it slow. Human perception notices silence and irregular pauses more readily than a dashboard notices a respectable mean.

An LLM server has two important phases:

• Prefill processes the input prompt. Its work largely determines time to first token.
• Decode generates output tokens one step at a time. Its scheduling and communication largely determine the gaps between tokens.

Continuous batching lets a server combine work from many requests. That usually raises total throughput because the GPU stays busy. It can also make one interactive request wait behind a large prefill or a crowded decode batch. This is the central trade-off: maximizing work per GPU cycle can worsen the latency tail for the person waiting at the keyboard.

The measurements I would collect

Measurement	Definition and useful breakdown
TTFT	Time to first token: request start to the first token or first visible stream chunk. Split it into queue wait, prefill time, and network or proxy delay.
ITL	Inter-token latency: time between consecutive generated tokens, or between streamed chunks when token timestamps are unavailable. Report p50, p95, p99, and the fraction of gaps above a noticeable threshold.
End-to-end latency	Request start to the final token, completion event, or disconnect. Record both server-observed and client-observed values.
Throughput	Requests per second, input tokens per second, output tokens per second, and completed jobs per second. These answer different questions.
Utilization	GPU compute activity, memory bandwidth, memory use, KV-cache occupancy, active batch size, scheduler queue depth, and worker concurrency.
Goodput	Useful completed work that satisfies its contract per unit time: for example, valid requests meeting their latency SLO, or valid output tokens produced within the batch deadline.

Every request should carry a trace ID with timestamps for client send, gateway receipt, scheduler admission, prefill start and end, first token generated, each stream flush, final token, completion, cancellation, and error. A production metric should not attach an unbounded trace ID as a label; keep request-level detail in traces or logs and use bounded dimensions such as model, region, route, token-length bucket, and traffic class in metrics.

For TTFT, the distinction between “first token generated” and “first token received by the browser” matters. If the server generated the token at 500 milliseconds but a proxy buffered it until 1.2 seconds, the model is not the only suspect. Measure both ends.

ITL needs similar care. If the server sends chunks containing several tokens, a client can measure chunk gaps but cannot honestly claim exact token-level ITL. Instrument token or decode-step timestamps on the server, and separately measure visible chunk gaps. A user experiences the latter.

Use histograms rather than averages. At minimum, inspect p50, p95, and p99, plus a histogram of ITL. Also record output-token count. A 30-token answer and a 1,000-token answer should not be treated as equivalent latency observations.

A concrete diagnosis

Suppose the copilot has 300 concurrent users. The dashboard reports average end-to-end latency of 13.1 seconds against an average target of 15 seconds, so the service appears healthy.

The detailed data says:

• TTFT: p50 420 milliseconds, p95 2.4 seconds, p99 4.8 seconds.
• ITL: p50 78 milliseconds, p95 310 milliseconds, p99 640 milliseconds.
• Output length: median 180 tokens, p95 420 tokens.
• End-to-end latency: p50 9.4 seconds, p95 29 seconds, p99 43 seconds.
• Output throughput: 2,100 tokens per second across the fleet.
• GPU compute activity: 92 percent on average.
• Scheduler queue depth: usually 4, but above 40 during prompt-heavy bursts.
• KV-cache occupancy: 88 percent at the same times.

The average hides two user-facing failures. Some users wait several seconds before seeing anything. Others see long pauses during decode. High GPU utilization does not disprove this diagnosis. It may mean the system is saturated and requests are waiting for access to a full decode batch. High KV-cache occupancy can force eviction, constrained admission, or smaller scheduling choices, depending on the serving system.

I would next split the latency by input-token bucket and traffic class. If the bad TTFT requests have large prompts, prefill contention is likely. If TTFT is acceptable but ITL spikes whenever batch size grows, decode scheduling or memory pressure is more likely. If server timestamps look healthy but client timestamps do not, inspect proxies, TCP buffering, stream flushes, and browser rendering.

Turning measurements into separate SLOs

For interactive traffic, I would define an explicit contract such as:

• 99.9 percent of requests produce a valid stream.
• p95 TTFT is at most 800 milliseconds; p99 is at most 2 seconds.
• p95 ITL is at most 150 milliseconds, with p99 at most 300 milliseconds.
• p95 end-to-end latency is at most 20 seconds for responses capped at 256 output tokens.

Those numbers are policy choices, not universal laws. The important design is that TTFT, ITL, and end-to-end latency are separate objectives. A single end-to-end number cannot tell whether the user waited in the queue or suffered pauses during generation. End-to-end targets should also be conditioned on output length, or they will punish a service for producing a genuinely long answer.

For batch traffic, first define the unit of useful work. A nightly summarization job might contain 10,000 documents and have a ten-minute completion SLO. If each output averages 180 tokens, the workload contains about 1.8 million output tokens, requiring about 3,000 output tokens per second to finish in 600 seconds. The batch SLO could be:

• 99 percent of jobs complete by the ten-minute deadline.
• At least 3,000 valid output tokens per second for the declared workload.
• At least 99.5 percent of documents produce valid, non-error outputs.
• Retries and dropped work are counted against completion, not quietly removed from the denominator.

Batch traffic usually does not need a 150-millisecond ITL SLO. A batch worker may legally buffer output and optimize for total completion time. It does need throughput, queue wait, worker utilization, deadline compliance, and validity measurements.

A useful goodput definition makes the contract measurable:

goodput = SLO-qualified useful work / wall-clock time

For interactive requests, that might be valid requests whose TTFT, ITL, and end-to-end values all pass. For batch, it might be valid output tokens from jobs completed before the deadline. Keep both the numerator and denominator visible. Otherwise a system can appear to improve goodput by dropping slow requests.

The senior-level trade-off

Do not “fix” streaming by blindly reducing batch size. That may improve p99 TTFT and ITL while cutting output throughput and increasing cost. Conversely, increasing batch size may look excellent in tokens per second while making interactive users stare at a blank screen.

The usual production pattern is isolation: reserve capacity or scheduling priority for interactive traffic, give batch work separate workers or an explicit lower priority, and enforce admission limits before the KV cache and queue become unstable. Optimize prefill and decode separately. Track quality as well as speed; a fast but truncated or invalid answer is not goodput.

What they’ll ask next

How do you know whether the problem is the model server or the network?
Compare server-generated first-token and token-gap timestamps with gateway and client-received timestamps. A widening client-only gap points toward buffering, transport, or rendering.

Why not use GPU utilization as the main SLO?
Utilization measures occupied hardware, not satisfied users. A GPU can be 95 percent busy processing batch work while interactive requests wait several seconds in the queue.

Would you use one SLO for all model requests?
No. Separate by user-visible interactive work and deadline-driven batch work, and usually by model or response-length class. Their acceptable waiting patterns and capacity economics are different.

One line to say in the room

“I would replace the comforting average with tail TTFT and ITL distributions, then protect interactive p95 and p99 SLOs from batch throughput optimization while measuring goodput as useful, contract-compliant work.”