How would you protect a multi-tenant LLM API from both denial of service and denial of wallet? Design limits and admission controls for requests, input tokens, output tokens, concurrency, retries, and expensive tools without making legitimate long-context users unusable.

How to think about it

I would protect it with hierarchical, token-aware admission control: enforce per-tenant and global budgets on requests, input tokens, output tokens, concurrency, retries, and tool spend before expensive work starts. I would reserve worst-case spend in controlled increments, charge actual usage afterward, and use fair queues so legitimate long-context calls are slowed predictably rather than quietly banned.

Why request limits alone fail

A request-per-minute limit counts envelopes, not work.

One tenant sending 600 requests containing 100 tokens each is very different from one tenant sending 60 requests containing 100,000 tokens each. The second workload may consume six million input tokens. It can exhaust provider capacity and a large monthly bill while remaining comfortably below a naive request limit.

A denial of service attacks availability: queues grow, model workers run out, and other tenants see timeouts. A denial of wallet attacks spend: an attacker causes enough model, retry, or tool usage to produce an unpleasant invoice. The same request can do both.

Admission control is the decision made before starting work: reject it, queue it, or run it. It must happen before calling the model and before launching a tool. The decision should use several independent dimensions:

Resource	Example control	What it protects
Requests	Per-tenant token bucket	Request overhead and abuse bursts
Input tokens	Token budget per minute or hour	Context processing cost
Output tokens	Reserved generation budget	Unbounded responses
Concurrency	Per-tenant and global semaphores	Worker and provider capacity
Retries	Separate retry budget	Accidental request multiplication
Tools	Per-tool cost, count, time, and spend caps	Search, code, database, and write actions

A token bucket is a counter that refills at a fixed rate and permits a bounded burst. Use one for requests and separate buckets for token consumption. A concurrency semaphore is simpler: it allows only a fixed number of operations to be active at once.

These controls should be hierarchical. Enforce an aggregate limit for the billing tenant, then smaller limits for project, API key, user, and possibly source IP. The tenant limit is the important one; otherwise a customer can create 1,000 API keys and evade the policy. IP limits are useful at an unauthenticated edge, but they are a poor substitute for tenant identity.

The admission path

First apply cheap limits. Reject an oversized HTTP body, malformed request, or impossible context length before spending time tokenizing it. Then authenticate the tenant and identify the target model.

Count the full billable input, not merely the latest user message. That normally includes conversation history, system instructions, tool schemas, and retrieved documents. Use the tokenizer appropriate to the selected model, or a conservative estimate when exact tokenization is unavailable.

Reserve resources atomically:

1. Consume one request credit.
2. Reserve estimated input tokens.
3. Reserve the requested output ceiling, or an initial output tranche.
4. Reserve concurrency.
5. Reserve the planned tool budget.
6. Queue or execute only if all required reservations succeed.

The reservations matter because checking each limit separately creates a race. Ten requests can all observe “one slot left” and then all start. The counter update must be atomic, or protected by a reliable distributed mechanism.

Here is framework-neutral pseudocode. The method names describe the required operations; they are not a vendor API.

def admit(request, tenant, state):
    input_tokens = count_tokens(request.input)
    output_cap = min(request.output_limit,
                     tenant.max_output_tokens_per_request)
    tool_cost = estimate_tool_cost(request.tool_plan)

    if input_tokens > tenant.max_input_tokens_per_request:
        return "reject: input too large"
    if not state.request_bucket.take(1):
        return "429: request rate"
    if not state.input_bucket.reserve(input_tokens):
        return "queue: input budget"
    if not state.output_bucket.reserve(output_cap):
        state.input_bucket.release(input_tokens)
        return "queue: output budget"
    if not state.tool_budget.reserve(tool_cost):
        state.output_bucket.release(output_cap)
        state.input_bucket.release(input_tokens)
        return "queue: tool budget"
    if not state.concurrency.try_acquire():
        state.tool_budget.release(tool_cost)
        state.output_bucket.release(output_cap)
        state.input_bucket.release(input_tokens)
        return "queue: concurrency"

    return {
        "status": "admit",
        "reserved_input": input_tokens,
        "reserved_output": output_cap,
        "reserved_tools": tool_cost,
    }

When the request finishes, settle the reservation. Charge actual input and output usage, release unused capacity, record latency and provider errors, and release the concurrency slot even on cancellation or timeout. Provider usage fields are preferable to counting visible text yourself; some models may meter additional reasoning or cached-token categories.

Reserving the entire output ceiling is safe but can be wasteful. If every request asks for 16,000 output tokens, the system may reject useful work even when most responses finish at 700 tokens. A practical compromise is to reserve an initial tranche, such as 512 output tokens, and extend the reservation in chunks while streaming. If the next chunk cannot be reserved, stop generation cleanly and return a usage-limit status rather than allowing an unbudgeted response.

A concrete tenant policy

Suppose Acme receives this policy:

• 60 admitted requests per minute, with a burst of 10.
• 1,000,000 input tokens per hour.
• 200,000 output tokens per hour.
• Eight concurrent model requests.
• A 128,000-token input limit per request.
• A default 4,000-token output ceiling, with 16,000 available to an approved workload.
• At most two automatic retries per request.
• A retry budget of 20 attempts per minute and no more than 10 percent of its admitted calls.
• Retrieval costs one tool unit, web search costs five, and code execution costs 50.
• A maximum of 20 tool calls or 200 tool units per request.
• No automatic retry for a side-effecting write.

An Acme analyst submits a document question containing 120,000 input tokens and asks for 2,000 output tokens. It consumes one request credit, 120,000 input-token credits, 2,000 output-token credits, one concurrency slot, and whatever tool reservation the plan requires. It is not rejected merely because it is long.

However, it may wait behind currently running work. The scheduler should use weighted fair queuing, ideally accounting for estimated tokens rather than just request count. A 120,000-token job should not permanently block twenty small jobs, and one tenant’s long-context workload should not occupy every global slot. Separate short-request and long-context queues can make this visible and predictable. The long-context queue may have lower throughput, but it should have an explicit budget and service rate.

Do not silently summarize or truncate a user’s context to make the numbers fit. That changes the question. Offer an explicit fallback, such as a user-selected retrieval or summarization mode, and report exactly what was changed.

Retries and expensive tools

Retries are hidden traffic multiplication. A client sends one request, the gateway retries twice, the model provider retries once, and a slow tool is called again after the original actually succeeded. One user action has now become several billable operations.

Retry only transient failures such as transport errors, provider overload, or a documented temporary server error. Do not retry invalid input, authentication failures, context-too-large errors, or policy denials. Use exponential backoff with jitter, respect provider retry hints, attach an idempotency key, and count every attempt against request, concurrency, token, and tool budgets.

A timeout does not prove that the provider did no work. For side effects, use idempotent tool operations or a durable operation ID so a retry cannot charge a card, send an email, or mutate a record twice.

Treat tools as their own resource pool. Allowlist tools by tenant and role. Give each tool a concurrency limit, timeout, maximum result size, maximum call count, and cumulative cost budget. Cap recursion depth so a model cannot call a tool, inspect the result, call another tool, and continue indefinitely. Require explicit confirmation for destructive or financially meaningful actions. Add a circuit breaker—a switch that temporarily stops calls to a failing dependency—when a tool starts timing out or returning errors.

The senior-level trade-off

Hard limits are necessary, but a single global limit is a blunt instrument. If you set the input-token ceiling to 16,000 because most users submit short prompts, the customer doing legitimate contract analysis becomes your false positive.

The better design separates per-request maximum from rate budget. A long request can be legal while consuming more of the tenant’s token allowance. It may enter a long-context queue and consume more virtual scheduling time, but it does not need to be banned.

Queues also need limits. An unbounded queue merely turns denial of service into delayed denial of service. Bound queue depth and wait time, return a clear 429 with Retry-After when the tenant budget is exhausted, and use 503 when the service itself is overloaded. Never promise that a queued request will run after its deadline.

A common failure mode appears as “normal request volume, rapidly increasing spend.” Traces often reveal repeated upstream attempts after client timeouts, or a tool loop producing dozens of calls per answer. The fix is not simply lowering requests per minute. Inspect per-request attempt count, actual token settlement, tool-call graphs, and tenant-level spend; then enforce the missing budget at the point where the multiplication occurs.

What they’ll ask next

How do you estimate input tokens before the model call?
Use the selected model’s tokenizer when available, including messages, tool definitions, and retrieved content. Otherwise use a conservative estimate, reserve it before execution, and settle against provider-reported usage afterward. Also cap raw bytes and tokenization time so token counting cannot become its own denial-of-service path.

What happens when a tenant exceeds a limit?
Do not treat every limit as a rejection. Queue when capacity may return before the request deadline, reject with 429 and Retry-After when the tenant’s budget is exhausted, and return 503 for temporary platform overload. Keep the response specific enough for a client to change its behavior.

How do you keep one large request from starving everyone else?
Use per-tenant concurrency, global concurrency, and weighted fair scheduling based on estimated work. Separate long-context traffic when necessary, while reserving it a real, documented share of capacity rather than making it a decorative feature.

“I would admit work using hierarchical budgets for tokens, concurrency, retries, and tools, then settle actual usage and schedule long-context requests fairly instead of pretending one request equals one unit of work.”