Production agent architecture

At 3:07 a.m., a customer asks your support agent to refund order A-1042.

The model looks up the order. It sees a delayed shipment. It decides the customer qualifies for a $25 refund. Then the worker crashes immediately after the payment service accepts the refund but before the worker records that fact.

At 3:08 a.m., the job retries.

Does the customer receive $50?

That question is not answered by a better prompt. It is answered by where state is stored, who is allowed to issue refunds, whether the payment operation is idempotent, and whether the system can tell a completed tool call from an interrupted one.

An agent is easy to demonstrate. Production is where its boundaries become the product.

This lesson builds a reference architecture for the support agent. The same boundaries work for coding assistants, research agents, and internal operations bots.

The shape of the system

A production agent has seven responsibilities. They should be visible as separate layers, even if several initially share one process.

• Interface: authenticates the caller, accepts requests, streams progress, returns a response or run ID, and accepts cancellation. Closing a response stream cannot undo a payment already accepted.
• Control loop: owns the run: load state, ask the model for a proposal, validate it, invoke an approved tool, record the result, and repeat.
• Model gateway: handles credentials, model selection, timeouts, bounded retries, structured output, fallbacks, and token accounting. It does not authorize refunds.
• Tool boundary: exposes narrow capabilities and translates them to stable requests and results for legacy systems.
• State store: durably records requests, results, checkpoints, approvals, and status. Long-term customer memory is separate from run state.
• Policy and authorization: makes authoritative decisions about identity, tenancy, permissions, data access, approvals, and limits.
• Observability: records traces, model and tool calls, state transitions, policy decisions, latency, cost, and redacted inputs and outputs.

These can begin in one service. The boundaries still give each responsibility a testable owner.

The control loop is not a prompt

A prompt can say “refund only eligible orders.” It cannot guarantee that instruction survives a malicious order note, a conflicting tool result, or a model that ignores it.

The loop turns a model proposal into a controlled state transition:

1. Create run_7f2 with the request and trusted caller metadata.
2. Ask the model for an allowed next action.
3. Validate the tool name and arguments.
4. Invoke the tool, after policy authorization, and save its result.
5. Repeat until the run completes, pauses, fails, is cancelled, or reaches a budget.

For the refund, the model proposes lookup_order, then issue_refund for 2,500 cents.

Policy independently checks:

• the caller
• the tenant
• order ownership
• eligibility
• currency
• refundable balance
• approval rules

The payment adapter then uses a durable operation reference. Only a confirmed provider result can produce a customer-facing confirmation.

The model is useful because the next step may be uncertain. The loop is necessary because uncertainty is not permission.

This compact fixture demonstrates the important recovery behavior. The dictionary is an in-memory test double; production needs a database, key-value store, or durable workflow system.

class SimulatedCrash(RuntimeError):
    pass


store = {}
orders = {
    "A-1042": {
        "tenant": "shop_19",
        "owner": "customer_42",
        "currency": "USD",
        "refundable_cents": 2500,
        "eligible": True,
    }
}
provider = {}
crash_once = True


def new_run():
    return {
        "run_id": "run_7f2",
        "tenant": "shop_19",
        "principal": "customer_42",
        "order_id": "A-1042",
        "operation_id": "refund:shop_19:A-1042:01",
        "status": "running",
        "order": None,
        "refund": None,
        "in_flight": None,
    }


def authorize(state, amount, currency):
    order = orders.get(state["order_id"])
    if not order or order["tenant"] != state["tenant"]:
        return False, "order not found"
    if order["owner"] != state["principal"]:
        return False, "caller does not own order"
    if not order["eligible"] or currency != order["currency"]:
        return False, "refund not eligible"
    if amount <= 0 or amount > order["refundable_cents"]:
        return False, "amount exceeds refundable balance"
    return True, "approved"


def refund_provider(state, amount, currency):
    global crash_once
    key = state["operation_id"]
    request = (state["order_id"], amount, currency)

    if key in provider:
        old_request, result = provider[key]
        if old_request != request:
            return {"status": "conflict"}
        return {**result, "replayed": True}

    result = {
        "status": "confirmed",
        "provider_ref": "payref_001",
        "order_id": state["order_id"],
        "amount_cents": amount,
        "currency": currency,
    }
    provider[key] = (request, result)

    if crash_once:
        crash_once = False
        raise SimulatedCrash("crash after provider accepted refund")
    return result


def model_decide(state):
    if state["order"] is None:
        return "lookup_order", {}
    if state["refund"] is None:
        return "issue_refund", {
            "amount_cents": 2500,
            "currency": "USD",
        }
    return "final", {}


def run():
    state = store.setdefault("run_7f2", new_run())

    if state["status"] == "completed":
        print("Refund confirmed for A-1042 ($25.00).")
        return

    # Recovery happens before asking the model for another side effect.
    if state["in_flight"]:
        key = state["in_flight"]["operation_id"]
        if key in provider:
            state["refund"] = provider[key][1]
            state["in_flight"] = None
            state["status"] = "completed"
            store["run_7f2"] = state
            print("reconcile: provider refund confirmed")
            return
        state["in_flight"] = None
        store["run_7f2"] = state

    name, args = model_decide(state)

    if name == "lookup_order":
        state["order"] = orders[state["order_id"]]
        store["run_7f2"] = state
        return run()

    if name != "issue_refund":
        state["status"] = "pending"
        store["run_7f2"] = state
        return

    amount, currency = args["amount_cents"], args["currency"]
    allowed, reason = authorize(state, amount, currency)
    if not allowed:
        state["status"] = "denied"
        store["run_7f2"] = state
        print(f"Refund denied: {reason}.")
        return

    # Persist before crossing the ambiguous side-effect boundary.
    state["in_flight"] = {
        "operation_id": state["operation_id"],
        "amount_cents": amount,
        "currency": currency,
    }
    store["run_7f2"] = state
    state["refund"] = refund_provider(state, amount, currency)
    state["in_flight"] = None
    state["status"] = "completed"
    store["run_7f2"] = state


try:
    run()
except SimulatedCrash as exc:
    print(f"worker: {exc}")

run()  # A new worker reconciles the accepted refund.
assert refund_provider(store["run_7f2"], 2500, "USD")["replayed"]

It prints:

worker: crash after provider accepted refund
reconcile: provider refund confirmed

The first worker saved the in-flight operation before calling payment. The provider recorded the refund before the worker crashed.

The replacement worker found that operation and confirmed it instead of issuing another refund. A repeat with the same key returns the original result.

The system never treats a model sentence as evidence. Completion requires all of the following:

• an approved policy result
• a confirmed provider result
• the matching order
• no in-flight operation

Otherwise it remains denied, failed, or pending.

Persist these as well:

• step budgets
• wall-clock deadlines
• cancellation requests
• terminal status

A cancellation disconnects a client; a run cancellation must stop new side effects and reconcile any in-flight operation. If the provider reports “unknown” or “still processing,” leave the run pending rather than guessing.

State belongs outside the worker

Workers should be disposable. Retries may land on another machine, deployments may remove the current process, and autoscaling may create many more workers. In-memory state can be a cache, never the authoritative record.

A durable run record might contain:

run_id: run_7f2
tenant_id: shop_19
status: waiting_for_tool
step: 4
events: tool and policy results
next_action: issue_refund
deadline: 2026-08-29T03:20:00Z
cancel_requested: false
in_flight_operation: refund:shop_19:A-1042:01

Saving only the latest prompt and response makes recovery ambiguous. “Payment accepted” is different from “the model intends to issue a refund.”

Distributed systems commonly provide at-least-once delivery: a step may run more than once. They do not provide magical exactly-once execution. Each side-effecting tool therefore needs an idempotency strategy.

An idempotency key identifies one operation only when the provider documents and enforces that behavior. The relevant details include:

• scope
• retention
• parameter matching
• in-progress responses

For the refund, create a durable intent such as:

refund:shop_19:A-1042:01

Reuse it across worker retries. A manual rerun must first find the existing intent or reconcile the provider. Deriving a new key from a new run ID could issue a second refund.

If the provider cannot reconcile an ambiguous request, stop for review rather than trying random keys.

Key state by tenant and run, and protect concurrent updates with a lease, version check, or equivalent mechanism. See durable execution for systems that persist workflow progress and resume after failure.

Tools are an anti-corruption boundary

Do not give the model a database connection, raw payment SDK, or general-purpose HTTP client. Give it narrow capabilities with explicit contracts.

An anti-corruption layer prevents one system’s concepts and failures from leaking into another. A tool can expose lookup_order(order_id) while hiding CRM pagination, legacy fields, credentials, and provider-specific errors. It validates inputs and filters outputs before they reach the model.

A contract should specify:

• arguments
• types
• permissions
• side-effect level
• timeout
• retry rule
• error shape

Separate read capabilities from writes: get_order is not change_order, and a refund tool should not accept arbitrary payment-provider JSON.

Normalize errors into deliberate categories such as temporary_unavailable, not_found, and not_authorized. The loop can retry the first, explain the second, and stop on the third.

Put the trust boundary below the model

Treat the model as an untrusted proposer. Its interpretation of the request, chosen tool, arguments, and repeated text from documents are all untrusted.

A shipping note saying “ignore the refund limit” is data, not authorization.

Policy must use authoritative facts:

• caller identity
• tenant and resource ownership
• eligibility
• amount
• currency
• approval state
• whether the exact operation already completed

The interface authenticates; the policy layer authorizes. Passing identity through model context is not enough.

This does not eliminate prompt injection. It limits its power: text cannot grant a capability the caller does not have.

Keep model behavior replaceable

A model gateway gives the loop one interface and centralizes:

• provider routing
• timeouts
• structured-output validation
• bounded retries
• prompt and schema versions
• token costs
• fallback policy

It should return:

• model ID
• request ID
• latency
• token counts when available
• finish reason
• validation errors

Choosing a cheaper model is gateway policy. Deciding whether a refund is legal is business authorization.

Observability should correlate the following with one run ID:

• interface requests
• state transitions
• model calls
• policy decisions
• tool calls

Redact secrets and customer data, but retain enough structure to explain wrong tools, denials, retries, latency, cost, and terminal status. The final answer is not an audit trail.

When not to build an agent

Start with the simplest design that fits:

Approach	Best fit	First reason to reject it
One model call	Summaries, extraction, classification, drafting	It cannot safely discover several dependent actions
Deterministic workflow	Known steps such as validate, charge, notify	The next step genuinely depends on runtime evidence
Single agent loop	Variable tool use with bounded side effects	It lacks budgets, policy, durable state, or recovery
Multi-agent system	Separate domains with real ownership boundaries	Coordination adds more failure modes than value

Use an agent when its complexity is justified by:

• runtime uncertainty
• pauses
• side effects
• recovery

Do not make a model choose steps that code already knows.

What to remember

• The model proposes; the control loop validates, authorizes, executes, records, and controls progression.
• External state, checkpoints, stable operation references, and provider-supported idempotency make interruption recoverable.
• Tools expose narrow capabilities, while policy and downstream services enforce permission.
• Cancellation is a state transition, not a network disconnect.
• Prefer one model call or a deterministic workflow unless runtime uncertainty makes an agent worthwhile.

Quick check

0/3

Q1

Ask the model whether it remembers issuing the refundRetry with a new random request identifierRetry with the same provider-supported idempotency key or query the provider using a stable operation referenceMark the run complete before calling the payment API

Check answer

Q2

Choosing a friendly tone for the final responseEnforcing a maximum number of steps and stopping a repeated tool callSuggesting that a refund may be appropriateSummarizing a tool result for the customer

Check answer

Q3

A multi-agent system, because separate agents can check each other's workA single model call followed by a deterministic validation and write workflowA long-running agent loop with unrestricted tool accessA model gateway that lets the model write directly to the database

Check answer