Inference metrics: TTFT, ITL & goodput

Cost and latency engineering looked at inference from the application’s side — price per token, perceived speed. Running the service needs a sharper instrument panel, because “fast” and “high throughput” are not the same number and often pull against each other. A server tuned for maximum tokens-per-second can deliver a miserable per-user experience, and one tuned for snappy responses can waste most of the GPU. The metrics below are how you tell which you have — and they all rest on one distinction.

Two phases: prefill vs decode

A single request runs in two very different phases, and conflating them is the root of most serving confusion:

• Prefill — process the whole prompt at once to build its KV cache. Every prompt token is handled in parallel, so this phase is compute-bound (it saturates the GPU’s math units). It happens once, and its duration is what the user waits for before anything appears.
• Decode — generate the answer one token at a time, each step attending to the cached K/V. Each step does very little math but must stream the whole KV cache and weights through memory, so decode is memory-bandwidth-bound. It repeats once per output token.

Compute-bound prefill and memory-bound decode have opposite hardware appetites — a fact this lesson’s metrics expose and the next lesson (disaggregated serving) exploits.

The four numbers

• TTFT — time to first token. The prefill latency: how long until the user sees anything. Set by prompt length and queueing. This is what streaming hides — the first word lands fast even if the rest takes seconds.
• ITL — inter-token latency (a.k.a. TPOT, time per output token). The decode step time: how fast the answer streams once it starts. A smooth, low ITL feels responsive; a jittery one feels broken.
• Throughput. System-level: total tokens per second across all concurrent requests. This is what you’re paying the GPU for.
• Goodput. The throughput of requests that actually meet their SLO (e.g. “TTFT under 500 ms and ITL under 25 ms”). Tokens delivered too slowly to satisfy the latency target don’t count — and this is the number that matters.

The latency-throughput frontier

Here is the tension. Batching more requests together raises throughput — the GPU does more useful work per pass — but it also makes each decode step slower, so per-request ITL rises. Push the batch far enough and you blow past your latency SLO: throughput is still climbing while goodput collapses to zero because no request meets the target.

def step_ms(B): return 10 + 0.5 * B          # decode step time grows with batch (toy)
SLO_ITL = 25                                  # per-token latency budget (ms)

print("batch  ITL(ms)  throughput(tok/s)  meets 25ms SLO?")
for B in [1, 8, 16, 32, 48, 64]:
    itl = step_ms(B)
    tput = B * 1000 / itl                      # B tokens produced each ITL ms
    print(f"{B:>4}   {itl:>5.1f}     {tput:>8.0f}        {'yes' if itl <= SLO_ITL else 'NO'}")

batch  ITL(ms)  throughput(tok/s)  meets 25ms SLO?
   1    10.5           95        yes
   8    14.0          571        yes
  16    18.0          889        yes
  32    26.0         1231        NO
  48    34.0         1412        NO
  64    42.0         1524        NO

Raw throughput keeps rising all the way to batch 64 (1524 tok/s) — but ITL crosses the 25 ms budget at batch 32. So goodput peaks at batch 16 (889 tok/s, still within SLO) and is zero beyond it: those faster batches produce tokens nobody can use. If you tuned this server by maximising throughput, you would pick batch 64 and ship a service that violates its own latency promise on every request.

In one breath

• Inference has two phases with opposite hardware appetites: prefill (process the prompt in parallel — compute-bound, sets TTFT) and decode (one token at a time — memory-bandwidth-bound, sets ITL).
• TTFT = time to first token (prefill latency); ITL/TPOT = inter-token latency (decode step); end-to-end = TTFT + (N−1)·ITL.
• Throughput is total tokens/sec across all requests; goodput is the throughput of requests that actually meet their SLO — the number that matters.
• Bigger batches raise throughput but worsen ITL: the latency-throughput frontier. In the demo throughput climbs to batch 64 but goodput peaks at batch 16 and then collapses past the SLO.
• Size on goodput at your SLO, report p95/p99 tails, and track TTFT and ITL separately because they break for different reasons.

Quick check

Next

These metrics expose that prefill and decode want different things from the hardware. Disaggregated serving acts on that — splitting the two phases onto separate GPU pools — and the app-side cost view lives in cost & latency engineering.