LLM cost and latency — the numbers that matter

The prototype runs on your laptop and the bill is rounding error. Then you ship to 5,000 users and accounting sends a screenshot. LLM cost and latency optimization isn’t premature — by the time you notice, the patterns are baked in and harder to change. The good news: 90% of the wins come from a small handful of techniques you can apply mechanically.

The cost model: input vs output tokens

Every API charges separately for tokens you send (input) and tokens the model generates (output). Output tokens almost always cost 2-5x more than input because generation is sequential and expensive, while input can be processed in parallel.

List prices, USD per million tokens:

Two things to internalize:

• A 5,000-token prompt that produces a 200-token answer on claude-sonnet-4.6 costs 5000 × $3/M + 200 × $15/M = $0.018. The 200 output tokens are 1/25 the token count of the input, yet cost $0.003 — a fifth of the input’s $0.015, because each output token is priced 5x higher. Output punches far above its token weight.
• Cache hits drop the input rate by ~90%. If you can structure prompts to hit cache, that 5000-token input drops from $0.015 to $0.0015.

Latency: TTFT vs TPS

Two numbers describe perceived speed:

• Time-to-first-token (TTFT) — milliseconds from sending the request to receiving the first output token. Dominated by input length and the model’s prefill speed.
• Tokens-per-second (TPS) — generation throughput after TTFT. This determines how long a long answer takes to finish.

Typical numbers (median, hosted endpoints, modest prompts):

Streaming closes the perceived-latency gap dramatically: a user sees the first word in 250ms even if the full answer takes 8 seconds. If you’re not streaming user-facing responses, you’re shipping a worse product for no reason.

Prompt caching: the 90% discount

Both OpenAI and Anthropic cache repeated prefixes of your prompts. You declare what’s cacheable, the provider hashes it, and subsequent requests with the same prefix pay ~10% of the input rate for that section. Internally, providers skip re-computing the KV cache (the per-token attention state that the model builds while reading your prompt) for the parts it has already seen — that’s where the 90% savings comes from.

The shape that wins:

Big stable prefix → small variable suffix. Almost every RAG, agent, and chat app fits this shape if you re-order the prompt to put the stable parts first.

# Anthropic-style prompt caching — the cost math, not a real call.
# A code-review tool: a fat stable system prompt + a tiny varying question.

# Per-million-token rates for claude-sonnet-4.6
INPUT_PER_M = 3.0
CACHE_HIT_PER_M = 0.30   # ~10% of normal input
OUTPUT_PER_M = 15.0

stable_tokens = 10_200    # cached prefix: system prompt + style guide + examples
variable_tokens = 50      # the user's question
output_tokens = 200

# Cold: nothing cached yet — pay full input rate on everything.
cold_cost = (
    (stable_tokens + variable_tokens) * INPUT_PER_M / 1_000_000
    + output_tokens * OUTPUT_PER_M / 1_000_000
)
# Warm: the stable prefix is a cache hit at ~10% of input rate.
warm_cost = (
    stable_tokens * CACHE_HIT_PER_M / 1_000_000
    + variable_tokens * INPUT_PER_M / 1_000_000
    + output_tokens * OUTPUT_PER_M / 1_000_000
)

print(f"Cold (first call):  ${cold_cost:.5f}")
print(f"Warm (cache hit):   ${warm_cost:.5f}")
print(f"Savings per call:   {(1 - warm_cost/cold_cost)*100:.0f}%")
print()
print("Per 100,000 calls:")
print(f"  cold-only:  ${cold_cost*100_000:.2f}")
print(f"  with cache: ${warm_cost*100_000:.2f}")

Cold (first call):  $0.03375
Warm (cache hit):   $0.00621
Savings per call:   82%

Per 100,000 calls:
  cold-only:  $3375.00
  with cache: $621.00

The cached prefix is 10,200 tokens of unchanging system prompt; the question is 50. Cold, you pay full input rate on all 10,250 tokens; warm, the 10,200 stable tokens bill at a tenth of that. The per-call savings look modest (82%, $0.027) — but at 100,000 calls that prefix is the difference between $3,375 and $621. Caching the stable prefix is the single highest-ROI change here.

Practical rules:

• Put everything stable before anything variable. One swapped variable in the middle invalidates the entire cache below it.
• Minimum cache size is provider-specific (Anthropic: 1024 tokens for most models). Below that, no cache.
• Anthropic caches expire after ~5 minutes idle; OpenAI’s automatic caching has a similar window. High-traffic prompts stay warm; long- tail prompts re-cache cold.

If you have any system that re-uses a fat system prompt — RAG, agents, chat with personas, code review tools — caching is the single highest- ROI change you can make.

The router pattern

Not every query needs your biggest model. A clarification like “what did I just say?” doesn’t need Claude Opus 4.7; a tax-law question maybe does. The router pattern uses a cheap model (or a small classifier) to decide which model handles each query.

In practice, 60-80% of traffic is “easy” and can run on a cheap model. A naive router that costs $0.0001 per classification can save 80% on a workload that previously used Opus for everything.

Two failure modes:

• Router misclassifies. Hard queries get a small model and a bad answer. Mitigate with a confidence threshold: low-confidence routes default to the bigger model.
• Router adds latency. Two LLM calls instead of one. Use a fast classifier (gpt-5-mini with 5 tokens of output, or a fine-tuned encoder) so the routing cost stays under 100ms.

Structured outputs (or how to stop paying for retries)

When a model emits malformed JSON, your code throws, you retry — and that retry is a full second request at full input price. Use the provider’s structured output mode (OpenAI’s response_format with a JSON schema, Anthropic’s tool-use pattern) instead of “please return JSON”. The provider enforces the schema during generation; you stop paying for retries on malformed responses.

# In production:
# response = client.responses.create(
#     model="gpt-5-mini",
#     input=[...],
#     response_format={"type": "json_schema", "json_schema": {...}},
# )

Same idea for tool calling — schema-constrained generation can’t hallucinate the tool name or the argument types. Cheap, mechanical robustness.

Semantic caching

For high-volume apps with repetitive queries (“how do I reset my password?”, “what’s your refund policy?”), a semantic cache stores prior answers keyed by query embedding. New query comes in, embed it, check if any cached query is above some cosine threshold — if so, return the cached answer without calling the LLM at all.

# Pseudocode for a semantic cache layer
qvec = embed(user_query)
hit = cache.search(qvec, threshold=0.95)
if hit:
    return hit.answer            # 0 LLM calls, ~50ms latency
else:
    answer = call_llm(user_query)
    cache.put(qvec, user_query, answer)
    return answer

Pick the threshold carefully. Too loose and unrelated queries collide (“how do I reset my password?” returns the answer for “how do I delete my account?”). 0.92-0.97 cosine is typical; verify on your data.

The cost ladder, in order of ROI

If you have to optimize in priority order:

1. Cache the system prompt — 50-90% cost cut, no quality change.
2. Switch obvious-easy queries to mini / haiku — usually 70%+ of traffic, 10-30x cheaper.
3. Structured outputs everywhere — kill retry cost, gain reliability.
4. Stream user-facing output — no cost change, huge UX win.
5. Router pattern for mixed difficulty — works when traffic is genuinely heterogeneous.
6. Semantic cache — only after you’ve verified repetition in your queries.

Skip steps in this order and you’re optimizing the wrong thing.

In one breath

• Input and output are billed separately; output costs 2–5× more (generation is sequential), and reasoning tokens are hidden output you still pay for.
• TTFT (time-to-first-token) and TPS (tokens/sec) describe perceived speed — stream user-facing output so the first word lands in ~250ms.
• Prompt caching puts the stable prefix first for a ~90% discount on it — the highest-ROI change, worth thousands at scale.
• The router pattern sends the easy 60–80% of traffic to a cheap model; structured outputs kill retry cost; semantic caching skips the LLM on repeats (mind staleness and the threshold).
• Work the cost ladder in order — cache → cheap model → structured outputs → stream → route → semantic cache — and measure before you optimize.

Quick check

Next

You can now ship LLM features without setting fire to the budget. The last question — and the one most teams put off too long — is whether to self-host the model at all.