An agent has a 30-second p95 latency target and a fixed cost per request, but it sometimes enters long tool loops. What controls would you use for budgeting, model routing, caching, parallelism, context growth, and graceful degradation?

How to think about it

I would enforce a per-request deadline and a hard budget ledger for wall-clock time, tool calls, model tokens, and dollars, then add bounded loops, selective model routing, safe caching, bounded parallel fan-out, context compaction, and explicit degraded fallbacks. Every control must be enforced by the runtime, not suggested in the prompt, and it must leave headroom inside the 30-second p95 target.

Why this is a control problem

An agent is a repeated loop: plan, call a tool, observe the result, then plan again. A loop can become expensive in two ways at once. Each iteration consumes time and money, while each tool result makes the next model call larger and slower.

The dangerous case is not an agent that takes 8 seconds every time. It is the 3 a.m. request that hits a slow dependency, retries it, receives an ambiguous result, and keeps planning until the user gives up. The average latency may look healthy while the tail quietly misses the target.

p95 means the 95th percentile: 95 out of 100 requests finish within the measured latency, while the slowest 5 can take longer. A hard timeout at exactly 30 seconds is therefore not a p95 strategy. It leaves no room for network overhead, response serialization, or the gateway returning the timeout cleanly. I would use an internal deadline such as 26 seconds, measure the end-to-end p95, and propagate the remaining time to every model and tool call.

The budget should have several independent limits:

• A wall-clock deadline.
• A maximum number of model turns and tool calls.
• A token or context-size limit.
• A dollar ceiling, with enough reserve for the final response.
• A concurrency limit and a retry limit.

A request stops when any important budget is exhausted. Otherwise, one generous budget can hide a different runaway: six quick tools may be affordable but a single 20-second tool is not.

A concrete design

Consider a refund-support agent. It can look up an order, retrieve the refund policy, check fraud risk, and then issue a refund. The first three operations are read-only. Issuing the refund is a side effect and requires all required evidence.

Assume the product has an internal cost ceiling of $0.08 per request. For illustration, accounting estimates a small model turn at $0.003, a normal model turn at $0.012, an escalation to a stronger model at $0.035, and a tool call at $0.002. Those are accounting assumptions, not universal provider prices.

A normal request might use one router turn, one normal reasoning turn, four tool calls, and a final response:

$0.003 + $0.012 + (4 × $0.002) + $0.015 reserve = $0.038

The reserve matters. Without it, the agent can spend the entire allowance investigating and then fail while composing the answer.

The runtime could set:

Budget	Example control
Time	26-second absolute deadline
Planning	4 model turns maximum
Tools	8 total calls, one retry for an idempotent read
Concurrency	At most 3 read-only calls at once
Cost	$0.08 ceiling, with $0.015 reserved
Repetition	Stop on the same tool and equivalent arguments

The order lookup, policy lookup, and fraud check can run concurrently. If their observed durations are 600 milliseconds, 900 milliseconds, and 1.4 seconds, parallel execution makes the critical path roughly 1.4 seconds plus scheduling overhead, rather than 2.9 seconds sequentially. The real p95 still needs measurement: the slowest member and queueing effects determine the fan-out tail.

The refund mutation must remain after validation. Parallelizing it with the checks might save time, but it could create an unauthorized refund before the agent knows whether the request is eligible.

The controls I would use

Budgeting. Create a request-scoped ledger before the first model call. Before starting an action, reserve its estimated worst-case cost and time. Pass an absolute deadline rather than giving every tool a fresh 5-second timeout; otherwise five sequential calls can consume 25 seconds before the final model call starts.

Count repeated calls separately from unique calls. A loop that calls get_order three times with the same arguments is a control failure even if each call is fast. Stop after a small retry allowance, and retry only operations that are safe to repeat. For writes, use an idempotency key so a timeout does not turn into two refunds.

Model routing. Use a fast, inexpensive model or deterministic code for classification, field extraction, and known workflows. Use a more capable model when the request is ambiguous, the value at risk is high, or validation fails. Escalation must itself pass the remaining time and dollar checks. If only 3 seconds and $0.01 remain, sending the request to the expensive model is not a strategy; it is wishful accounting.

Do not route purely by prompt length. A short request about a legal exception may need more care than a long but routine order lookup. Route on task type, risk, uncertainty signals, and prior failure, then measure success and tail latency by route.

Caching. Cache deterministic, read-only results first: policy documents, product metadata, and order lookups with a short freshness window. A cache key should include the tenant or user authorization scope, the normalized inputs, and relevant policy, tool, or data versions. Add a TTL where the underlying data changes.

Do not cache a refund action. Do not return a cached answer that was authorized for another customer. An entire agent response is cacheable only when the request, permissions, data freshness requirements, and policy version genuinely match. A stale policy answer can be worse than a slow answer because it sounds confident.

Parallelism. Represent the workflow as dependencies. Fan out independent reads, but use a bounded worker pool or equivalent concurrency limit. Unbounded parallelism can reduce one request’s latency while overloading the tool service, increasing queueing and making everyone slower.

Parallelism also spends the budget faster. Three speculative searches may finish quickly, but if only one was needed, the latency win came with extra cost and load. Prefer parallelism for known-required independent calls, not as a substitute for planning.

Context growth. Keep the agent state structured. Store full tool results outside the prompt and pass only the fields needed for the next decision. Remove duplicate observations, truncate low-value logs, and maintain a rolling summary that preserves decisions, constraints, identifiers, and evidence references.

Set a context budget before the model call, not after the provider rejects it. Summarization can reduce prompt size, but it is another model call with its own cost and failure risk. Use it when the context crosses a threshold, and retain the original evidence for audit rather than trusting a summary as the only record.

Graceful degradation. Define levels before production:

1. Complete the normal workflow.
2. Skip non-critical enrichment, such as a recommendation or fraud explanation.
3. Return a partial answer with the evidence actually obtained.
4. Ask the user for one missing fact or tell them to retry.
5. Queue the work for asynchronous completion.
6. Refuse the side effect when a required safety or authorization check is unavailable.

For the refund agent, “I found the order but could not verify the current refund policy, so no refund was issued” is a good degraded result. “Your refund is complete” when the payment tool timed out is a production incident wearing a sentence as a disguise.

The senior nuance

The textbook answer is often “set a maximum number of steps.” That is necessary but not sufficient. Six fast calls and one hung call are different latency risks, so step budgets must sit alongside absolute deadlines, per-tool timeouts, cancellation, and circuit breakers.

Caching, parallelism, and stronger models are not automatically improvements. Caching risks staleness and data leakage. Parallelism risks load and unnecessary spend. A stronger model may reduce retries but still violate the cost cap. If “fixed cost per request” literally means the vendor charges the same amount regardless of usage, routing will not lower that invoice; it still helps latency, quota consumption, and capacity. I would first verify what the fixed cost includes.

A common failure shows up first as rising p95 and rising token spend, while p50 remains unchanged. Traces then reveal the same tool name and arguments repeated, followed by a growing prompt. The fix is a duplicate-call guard, a total-call limit, result caching, and a deadline-aware stop condition—not a longer context window.

What they’ll ask next

“Why use an internal 26-second deadline instead of 30 seconds?”
Because the target is end-to-end p95. The service needs time for gateway overhead, cancellation, serialization, and a clean fallback. The exact reserve comes from latency measurements, but zero reserve is not credible.

“How do you know whether to retry a failed tool?”
Classify the failure. Retry a transient, idempotent read at most once if the remaining deadline justifies it. Do not retry validation failures, permission errors, or non-idempotent writes without an idempotency mechanism.

“How would you prove the controls work?”
Track p50, p95, and p99 latency; cost per request; loop and tool-call counts; context tokens; cache hit rate; model route; deadline cancellations; and degraded-response rate. Break those metrics down by workflow and dependency, then test slow, stale, duplicated, and unavailable tools deliberately.

The line to use in the room

“I would treat the agent as a deadline-driven workflow with a hard cost ledger: bounded actions, selective escalation, safe cached reads, dependency-aware parallelism, compact context, and an honest fallback before the budget runs out.”