Cost & latency control

At 3 a.m., a customer asks whether an order can be refunded. Your support agent:

• reads the conversation;
• looks up the policy;
• fetches the order;
• checks the payment state; and
• writes a reply.

The answer is good.

It also:

• made four model calls;
• carried more conversation into each call;
• waited for two tools; and
• retried once when the order system timed out.

That is the difference between a cheap model call and an expensive agent task. The bill comes from multiplication: number of calls × tokens per call, plus tools and retries.

Suppose the model prices in this example are $3 per million input tokens and $15 per million output tokens. These are illustrative rates, not a vendor quote. The table assumes every input token is uncached and uses the ordinary input rate.

At 100,000 tickets, that is $10,800 before infrastructure, tools, and retries. A 10 percent retry rate adds another $1,080 if the retried run repeats the whole path.

Latency multiplies too. If each model call takes 1.2 seconds and the calls are serial, the model portion alone takes 4.8 seconds. Add tool waits, network time, and queueing, and a six-second response is easy to produce. The slowest requests are usually worse.

The fix is not “always use the cheapest model”. Control the shape of the work:

• how many calls happen;
• how much each call sees;
• which model handles each step; and
• what happens when the run refuses to finish.

The cost model underneath an agent

For a run with calls numbered from 1 through n, the basic model cost is:

total model cost = sum((input tokens × input price per million + output tokens × output price per million) / 1,000,000)

Input and output prices are usually different. Generated reasoning tokens may be reported separately or as a detail inside output usage. Follow the provider’s invoice semantics and count disjoint categories only. For cached input, use the provider’s cache-read category when its price differs:

total model cost = sum((uncached input tokens × input price per million + cached input tokens × cache-read price per million + output tokens × output price per million) / 1,000,000)

A tool result does not sit outside the next model call. If the agent sends the full conversation, system instructions, tool definitions, and previous results each time, all of that becomes input again. That is why the example grows from 2,000 input tokens to 10,000: it is repeatedly repacking the same history.

Record tool charges and waits separately. A database query may add 400 milliseconds; a browser action may add seconds. For serial work:

request time = model time + tool time + queue time + network time

Two independent reads taking 800 milliseconds each cost roughly 1.6 seconds in sequence, but about 800 milliseconds plus coordination overhead in parallel. Parallelism reduces latency, not token cost: you still pay for both operations.

Track p95 latency, the time below which 95 percent of requests finish. p50 describes a typical request; p95 and p99 expose the experience of the slowest users.

Measure the run before changing it

Start with a trace for every run. For each model call, record:

• the model;
• input and output tokens;
• reasoning usage if applicable;
• cache status;
• duration;
• result; and
• retry number.

For each tool call, record:

• duration;
• response size;
• timeout; and
• repetition.

Attach these records to a request ID and tenant ID.

Track:

• Cost per successful task: total cost divided by tasks meeting your quality criterion.
• Tokens per successful task: useful for spotting bloated context independently of model prices.
• p50, p95, and p99 latency: averages hide queueing and retries.
• Success and escalation rate by route: a cheap route may be expensive if it escalates frequently.

Define “successful”. For the refund agent, the answer might need the correct policy, the correct order state, and no unauthorized refund. Connect cost measurements to your agent evaluation results, or the optimizer may make the agent cheaper by making it wrong.

The levers

Measure first, then apply the levers that match your trace. Eliminating unnecessary work and routing often produce the largest savings; caching and context control make the remaining calls cheaper and faster.

Route steps, not just whole requests

Model routing chooses a model for a particular task instead of sending every step to the same model. These steps often need less capability than ambiguous policy reasoning:

• classification;
• extraction;
• tool selection; and
• short rewrites.

In the refund example, a small model can classify “refund status” and select between two read-only tools.

A more capable model can handle policy exceptions, multiple payments, or a disputed decision. A deterministic rule can be cheaper still: if a known order number appears with a shipping-status request, ordinary code may route it without an LLM call.

Routing creates a failure mode: the wrong model gets the task. Do not use self-reported confidence as the only router. Calibrate thresholds against an evaluation set. Escalate uncertain or high-risk cases before a side effect. Measure quality, retries, and human review by route.

Cache the right thing

Prompt caching reuses a stable prefix such as system instructions, tool schemas, and a policy document. It still runs the model and generates an answer, but reduces the repeated prefix cost or time according to the provider’s rules. Put stable content first and changing content later:

1. system instructions;
2. tool definitions and stable policy;
3. slowly changing tenant configuration;
4. conversation, user request, and live tool results.

Eligibility, minimum sizes, expiration, and prices differ by provider. Treat the cache as an optimization, not a correctness dependency: the request must work on a miss.

Semantic caching can skip a model call by reusing an answer judged similar to a previous request. Similar is not equivalent: “Can I return this shirt?” and “Can I return this shirt after wearing it?” may have different answers.

Tool-result caching is often safer for read-only data. Cache a policy lookup by policy version or a product description by product ID and content version. Use a short TTL for changing data. Do not cache balances, inventory, access decisions, or user-dependent results without an explicit key and freshness rule. Monitor hit rate, tokens saved, and stale-result incidents.

Prune and compact context

Context pruning removes irrelevant material before the next call. Compaction replaces a long transcript with a shorter, structured summary. Both reduce input tokens and can improve quality by removing competing instructions.

The next refund call may need:

• the request;
• order ID;
• current policy version;
• payment state;
• previous decisions;
• unresolved questions; and
• supporting evidence.

It does not need the entire 18-message conversation. Keep this state separately from chat history. A useful summary says, “order 8472 was delivered on 12 August; policy allows returns within 30 days; customer reports a defect; approval is still required.”

Do not blindly keep the last few messages or summarize without validation. An old constraint may matter more than a recent pleasantry, and a summary can lose an identifier or negative instruction. Preserve raw records for audit and test summaries against evaluations.

Bound tokens, steps, time, and money

A step budget limits agent cycles. A token budget limits input or output usage. A dollar budget limits spend. A wall-clock deadline limits how long the user waits. They protect against different failures: a token cap does not stop 40 short calls, and a step cap does not stop one 200,000-token context.

Apply limits at several levels:

• A request might have a 12-step and $0.25 budget.
• A tenant might have a daily spend limit.
• The service needs rate limits and alerts.

Reserve enough budget for the final answer.

Enforce limits with a request-scoped budget ledger containing reserved and consumed steps, tokens, tool charges, and dollars. Before every model call, tool call, and retry, check the ledger and reserve the worst-case amount. Afterward, debit actual usage and release the remainder. Tenant quotas require atomic reservation or debit so concurrent requests cannot spend the same remaining balance. Propagate one absolute deadline through tools and retries; a child gets the remaining time, not a fresh timeout.

Fail closed when accounting cannot prove work is allowed. If reservation fails, do not start the operation: return a structured budget-exhausted or temporarily unavailable result and escalate when needed. If post-call accounting fails, stop further work. Preserve the trace and mark the run’s accounting state unknown.

Retry transient network failures only a small number of times, with exponential backoff and jitter. Do not retry invalid tool arguments or failed quality checks forever. Retries must consume the original budget and deadline.

Choose the smallest workflow that fits

An agent is not automatically the right abstraction:

• Use ordinary code or a fixed workflow when the path and rules are known.
• Use one structured model call for one judgment or extraction.
• Use a bounded agent when the required tools or next observations are unknown.
• Use parallel workers only for truly independent tasks; they reduce wall-clock time but multiply usage and coordination.

Planning can reduce a chatty loop, but planning costs a model call and can become stale. Choose the workflow whose variability justifies its extra calls and whose calls fit the latency-critical path. See planning patterns for more detail.

The honest trade-off

Cost control can lower quality, increase engineering complexity, or both. The trade-offs include:

• A smaller model may miss a rare exception.
• A summary may omit the sentence that changes the answer.
• A cache may serve stale policy.
• A deadline may return a partial result.

Every optimization needs a correctness boundary.

Keep side effects behind explicit checks. Read-only lookups can be retried or cached more freely than issuing a refund. For writes, use:

• idempotency keys;
• approval gates; and
• a durable record of what the agent attempted.

Cost controls should stop work safely, not merely stop billing. The reliability for agent side effects lesson covers that boundary.

Roll out in stages:

1. establish a baseline on representative tasks;
2. remove unnecessary calls;
3. add routing and context control;
4. add caching with explicit freshness and authorization rules; and
5. enforce budgets and deadlines before unrestricted traffic.

Compare:

• cost per successful task;
• quality by route;
• p95 latency;
• escalation; and
• retry rate.

Do not compare tokens saved alone.

Quick check

Next

The surrounding production toolkit is agent evaluation, observability, context engineering, and reliability for agent side effects. Together, they turn cost control into an operating discipline.