Build the unit economics for an LLM feature. Which variables dominate cost and latency, and how would you decide among prompt reduction, output limits, caching, batching, quantization, model routing, and self-hosting while preserving the product's quality and reliability targets?

How to think about it

I would model cost per successful task, not per API call, as input-token cost plus output-token cost plus retries, tools, cache misses, and infrastructure; latency is queueing and network time plus prompt processing and sequential output decoding. I would measure the workload distribution first, then use the cheapest intervention that keeps quality, tail-latency, availability, and freshness targets: reduce needless context and output, cache safe repeats, route easy work to smaller models, batch non-interactive work, and consider quantization or self-hosting only when their operational risk is justified.

Why the mechanism matters

The basic cost model is:

cost = input_tokens × input_rate + output_tokens × output_rate + tool_costs + retry_costs + infrastructure_costs

Rates are normally quoted per million tokens, but the exact price depends on the provider, model, region, caching mode, and whether the request is synchronous or batch. Do not put a price into a spreadsheet once and call the result “unit economics.” Provider price cards change, and your token distribution is usually more important than your average.

A long input raises cost and usually raises time to first token, or TTFT: the delay before the model emits its first token. The model must read and process that context before it can answer. A long output raises cost and completion time because decoding is largely sequential. The model can process some prompt work in parallel, but it generally generates output tokens one after another.

A useful latency approximation is:

latency = queue + network + TTFT + output_tokens / decode_rate + tool_calls + retries

This is not a promise that every term scales perfectly. It is a debugging model. A request with a tiny prompt can still be slow if the serving system is overloaded. A fast model can still miss the product SLO if it calls a search tool three times in series.

The variables that usually dominate are:

• Request volume and its peak concurrency.
• Input and output token distributions, especially the long tail.
• Model price and generation throughput.
• Queueing at the provider or your own GPU fleet.
• Cache hit rate.
• Retries, fallbacks, tool calls, and other extra model invocations.
• The cost of a wrong answer, which may appear later as a support ticket, human review, or lost customer.

For an interactive feature, output length often dominates completion latency. For a retrieval-heavy feature with a 30,000-token context, input processing and input charges may dominate cost and TTFT. There is no universal winner. Measure both.

A concrete support-feature calculation

Suppose a customer-support product generates a draft reply. It handles 1 million requests each month. Each request currently sends 1,200 input tokens and produces 250 output tokens.

For arithmetic, assume the provider’s price card charges 30 cents per million input tokens and 1.20 dollars per million output tokens. These are illustrative assumptions, not a current market quote.

The monthly model bill is approximately:

• Input: 1.2 billion tokens, or 360 dollars.
• Output: 250 million tokens, or 300 dollars.
• Total: 660 dollars before retries, tools, caching, and other services.

Now suppose a load test at the target concurrency measures 500 milliseconds to first token and an effective decoding rate of 80 output tokens per second. A 250-token answer takes roughly 3.6 seconds to finish: 0.5 seconds plus 250 divided by 80.

The team makes three changes:

1. The prompt keeps the relevant account policy and the two most useful retrieved messages instead of sending the whole conversation. Input falls from 1,200 to 700 tokens.
2. The product asks for a concise reply and sets an output cap appropriate for the UI. Average output falls from 250 to 180 tokens.
3. An exact response cache handles 20 percent of requests, where the same account, question, policy version, and model settings produce a reusable answer.

Before caching, the new monthly model cost is:

• Input: 700 million tokens at 30 cents per million, or 210 dollars.
• Output: 180 million tokens at 1.20 dollars per million, or 216 dollars.
• Total: 426 dollars.

With a 20 percent cache hit rate, roughly 80 percent of that model work remains, so the model portion is about 341 dollars. The cache itself still has storage and lookup costs, and cache misses still pay the full model cost.

At 80 tokens per second, reducing the output to 180 tokens saves about 875 milliseconds of decoding time. Reducing the input should also improve TTFT, although the amount depends on the model and serving system. The right validation is not “the average got faster.” It is whether p95 completion time stays within the product target during the busiest hour.

Choosing the levers

Lever	What it improves	Main risk
Prompt reduction	Input cost, TTFT, context noise	Removing evidence or instructions
Output limits	Output cost and completion time	Truncated or unhelpfully terse answers
Caching	Repeated-request cost and latency	Stale, private, or incorrect reuse
Batching	GPU utilization and cost for queued work	Higher waiting time and tail latency
Quantization	Model memory and sometimes serving cost	Quality loss or no speedup on the chosen hardware
Model routing	Expected cost and latency	Misclassifying hard requests
Self-hosting	Control and possible variable-cost savings	Capacity, reliability, and operational burden

Prompt reduction is usually the safest first move when the prompt contains duplicated policy, old conversation turns, or irrelevant retrieved documents. I would replace “send everything” with retrieval, structured summaries, and explicit evidence selection. I would not blindly truncate from the front or back. The deleted paragraph may contain the customer’s cancellation date.

Output limits work because every generated token consumes decoding time and usually incurs output billing. A maximum is an upper bound, not a guarantee that the model will naturally be concise. Set it from the task: a classification needs a few tokens; a support draft may need several paragraphs. Test truncation rate, schema validity, and human-rated usefulness. A cap that saves 30 percent while causing 4 percent of replies to end mid-sentence is not automatically a win.

Caching is excellent for deterministic or slowly changing work. A response cache can make an exact repeat nearly free from the model’s perspective and dramatically reduce latency. The cache key must include whatever changes the answer: user or tenant, permissions, locale, model version, prompt version, and relevant policy version. Semantic caching is more dangerous. Two questions that look similar can have different account facts. Prefix or prompt caching may reduce repeated processing of a stable prompt prefix, but its billing and latency behavior is provider-specific.

Batching fits nightly summarization, document enrichment, and offline evaluation. It improves utilization because the hardware processes many requests together instead of waiting between small jobs. It is a poor default for a user staring at a spinner. Waiting for a batch window increases queue time, and dynamic batching can improve throughput while worsening p95 latency.

Quantization stores model weights with fewer bits, reducing memory pressure and sometimes increasing throughput. It does not magically make every model faster. The result depends on the hardware, kernels, sequence lengths, and workload. Evaluate the quantized model on factuality, tool selection, refusal behavior, structured-output validity, and tail latency. If a quantized model needs more retries or human correction, its apparent saving is fiction with a dashboard.

Model routing gives simple requests to a cheaper or faster model and sends difficult cases to a stronger one. A practical router might use task type, context length, required tools, language, or a confidence signal. I would start with conservative rules and a fallback, then compare the routed system with the strong model on a held-out set and live shadow traffic. The unit to optimize is expected cost per successful task, not the percentage sent to the cheap model.

Self-hosting becomes interesting with sustained, predictable volume, strict data residency, specialized models, or a need for control over latency. It is unattractive when traffic is spiky or the feature is still changing. Include GPU reservations, replicas for failure, storage, networking, observability, engineers, upgrades, security, and idle capacity. If an always-on deployment has a total monthly cost of 12,000 dollars, and the hosted baseline costs 0.00066 dollars per request under the earlier assumptions, the simple break-even point is about 18.2 million requests per month. That is only arithmetic. Peak capacity and the reliability design may move the real break-even point far higher.

The senior-level constraint

I would optimize under explicit constraints rather than chase the lowest token bill. The quality constraints might include grounded-answer pass rate, escalation rate, schema validity, and human preference. Reliability constraints might include availability, timeout rate, p95 and p99 latency, and a maximum fallback rate.

The most common failure mode is a cheaper system that looks healthy in averages. The first symptom is often a p99 latency spike during traffic peaks, followed by retries, followed by even more queueing. Another is a quiet quality regression: the cache serves answers from an old policy version, or the router sends a rare but important legal question to the small model. Version cache keys, cap retries, use timeouts and circuit breakers, and keep a strong-model fallback for defined failure cases.

What they’ll ask next

How would you handle a long-context feature?
Measure which tokens affect the answer. Retrieve fewer documents, deduplicate them, summarize stable history, and preserve citations or evidence needed for verification. Compare against the full-context baseline on a fixed evaluation set rather than assuming shorter is better.

How would you prove caching is safe?
Report hit rate, avoided model cost, latency saved, stale-answer rate, and privacy incidents. Make freshness and authorization part of the cache key or bypass the cache. If the answer depends on live account state, do not cache the final answer merely because the wording is similar.

When would you self-host?
When sustained volume, data control, or predictable latency makes the fixed cost worthwhile, and when measured throughput at peak load satisfies the SLO with spare capacity. I would not self-host to save money before measuring utilization and including on-call and redundancy costs.

One line I would use in the room: “I optimize dollars per successful, on-time task—not tokens per request—and I spend the savings only where the quality evaluation and SLO say the product survives.”