Reliability for agent side effects

Reliability for agent side effects

At 2:13 a.m., an account agent handles Acme’s subscription renewal. It calls the payment tool for $49.90. The gateway accepts the charge, but the response is lost when the worker disappears.

The agent restarts. It sees no success message. Its planner produces a sensible next step: try the charge again.

At 2:14 a.m., Acme has paid twice.

The same failure can send two emails, open two support tickets, or delete the same customer record twice. A timeout was enough.

The reliability problem is not merely “make the agent remember where it was.” It is this:

If an action may have happened, can the system safely try it again?

The answer requires a correctness discipline around side effects: actions that change something outside the agent, such as charging a card, sending a message, changing a database row, or deploying code.

Why agents naturally retry actions

A planner is the model or program that chooses the next tool call. After a restart, sampling, changed context, new observations, or a different model response can produce a different call.

Now add a network boundary:

1. The worker asks the payment service to charge $49.90.
2. The service charges the card.
3. The worker crashes before receiving or saving the response.
4. The runtime resumes from its last durable checkpoint.
5. The agent cannot distinguish “the call never left” from “the charge succeeded and the reply vanished.”

Retrying is reasonable. Retrying as a fresh request is dangerous.

Production runtimes therefore generally provide at-least-once delivery: a step may be attempted again until the system sees an acknowledgement. This prevents lost messages from silently becoming lost work, but it does not prevent duplicates.

Durable execution preserves workflow state across crashes. It cannot put your workflow and an external payment provider in one transaction. The model, worker, network, and provider do not share a magical commit button.

The remedy is idempotency: repeating the same logical request produces the same externally visible result as doing it once. A second request to charge order A-1842 should return the first charge result, not create a second charge.

The safe shape is simple:

Give each logical action a stable identity

An idempotency key is a durable name for one logical action, not one attempt.

For Acme’s renewal, suppose the incoming business event has the unique ID renewal-2026-08-28-01. A useful key is:

charge:v1:acme:renewal-2026-08-28-01

Every retry uses that key. A later legitimate renewal uses a different event ID and therefore a different key. Do not generate keys from model wording, timestamps, or attempt counters. Intentional duplicate user requests need separate event IDs; internal retries inherit the original ID.

A key usually combines the tenant or account, business event, operation, and semantic version. Store a hash of the request parameters alongside it. If the first request means $49.90 and a retry means $499.00, reject the mismatch rather than returning the old result.

The dedupe record belongs in durable storage with a uniqueness constraint. It might contain:

tenant_id
action_id
key
operation
request_hash
status
provider_reference
result
created_at
updated_at
lease_until

An in-memory set disappears on restart. An agent-context note is not a transaction. A checkpoint saying “step 4 completed” does not prove that a remote payment committed.

Use two layers:

1. Your database persists the action identity, serializes its state, and rejects duplicate rows.
2. The downstream service receives the same key or provides a reliable query by that key.

Keep records as long as retries, delayed jobs, webhook redelivery, and operator replays can occur. A provider that forgets keys after 24 hours is unsafe if operators can replay workflows after seven days.

A worked charge

The agent charges 4,990 cents for order A-1842 during renewal event renewal-2026-08-28-01. The provider creates ch_1, but the response is lost. The recovery request uses the same tenant-scoped key, so the provider returns ch_1 instead of creating ch_2.

These file-backed SQLite databases are test doubles: they survive the injected failure during this run but are not production storage. The exception occurs after the provider commits and before the local completion update.

import hashlib
import json
import os
import sqlite3
import tempfile


class SimulatedCrash(RuntimeError):
    pass


with tempfile.TemporaryDirectory() as tmp:
    db = sqlite3.connect(os.path.join(tmp, "operations.sqlite"))
    provider_db = sqlite3.connect(os.path.join(tmp, "provider.sqlite"))

    db.execute("""
        CREATE TABLE operations (
            tenant_id TEXT NOT NULL,
            action_id TEXT NOT NULL,
            operation TEXT NOT NULL,
            idem_key TEXT PRIMARY KEY,
            request_hash TEXT NOT NULL,
            status TEXT NOT NULL,
            result_json TEXT,
            UNIQUE (tenant_id, action_id, operation)
        )
    """)
    provider_db.execute("""
        CREATE TABLE provider_operations (
            idem_key TEXT PRIMARY KEY,
            amount_cents INTEGER NOT NULL,
            charge_id TEXT NOT NULL
        )
    """)
    provider_db.commit()

    def provider_charge(idem_key, amount_cents):
        row = provider_db.execute(
            "SELECT charge_id, amount_cents FROM provider_operations "
            "WHERE idem_key = ?", (idem_key,)
        ).fetchone()

        if row:
            charge_id, saved_amount = row
            if saved_amount != amount_cents:
                raise ValueError("provider key used for different amount")
            return {"charge_id": charge_id, "amount_cents": saved_amount}

        count = provider_db.execute(
            "SELECT COUNT(*) FROM provider_operations"
        ).fetchone()[0]
        result = {"charge_id": f"ch_{count + 1}", "amount_cents": amount_cents}
        provider_db.execute(
            "INSERT INTO provider_operations "
            "(idem_key, amount_cents, charge_id) VALUES (?, ?, ?)",
            (idem_key, amount_cents, result["charge_id"]),
        )
        provider_db.commit()
        return result

    def charge_once(tenant_id, renewal_event_id, amount_cents, fail=False):
        operation = "charge"
        action_id = renewal_event_id
        key = f"charge:v1:{tenant_id}:{action_id}"
        request_hash = hashlib.sha256(
            f"{key}|{amount_cents}".encode()
        ).hexdigest()

        row = db.execute(
            "SELECT idem_key, status, request_hash, result_json "
            "FROM operations WHERE tenant_id = ? AND action_id = ? "
            "AND operation = ?",
            (tenant_id, action_id, operation),
        ).fetchone()

        if row:
            saved_key, status, saved_hash, result_json = row
            if saved_key != key or saved_hash != request_hash:
                raise ValueError("action identity or parameters changed")
            if status == "completed":
                return json.loads(result_json)
        else:
            db.execute(
                "INSERT INTO operations "
                "(tenant_id, action_id, operation, idem_key, request_hash, status) "
                "VALUES (?, ?, ?, ?, ?, 'started')",
                (tenant_id, action_id, operation, key, request_hash),
            )
            db.commit()

        result = provider_charge(key, amount_cents)
        if fail:
            raise SimulatedCrash("crash after provider_charge returned")

        db.execute(
            "UPDATE operations SET status = 'completed', result_json = ? "
            "WHERE idem_key = ?",
            (json.dumps(result), key),
        )
        db.commit()
        return result

    tenant_id = "acme"
    event_id = "renewal-2026-08-28-01"

    try:
        charge_once(tenant_id, event_id, 4990, fail=True)
    except SimulatedCrash as exc:
        print(exc)

    print(charge_once(tenant_id, event_id, 4990))
    print(charge_once(tenant_id, event_id, 4990))
    print(provider_db.execute(
        "SELECT COUNT(*) FROM provider_operations"
    ).fetchone()[0])

It prints:

crash after provider_charge returned
{'charge_id': 'ch_1', 'amount_cents': 4990}
{'charge_id': 'ch_1', 'amount_cents': 4990}
1

The first call commits ch_1 and crashes before recording completion. Recovery reuses the persisted identity and provider key, so the provider returns ch_1; the next call reads the completed local row.

A production implementation also needs concurrency leases, stale-worker recovery, key-expiry handling, and reconciliation. Every layer must agree what “the same action” means.

Separate read, plan, and act

A reliable agent does not let a model improvise a fresh payment call after every uncertain timeout.

Read gathers facts such as order status, payment status, inventory, recipient, and policy. Reads are usually safe to repeat, though they may be stale.

Plan turns those facts into a typed intent, such as charge order A-1842 for 4990 cents using pm_7. Persist the intent, action ID, and request hash before acting.

Act is a narrow executor. It validates the intent, checks authorization and limits, supplies the stable key, calls the tool, and records the result. It should not ask the model to rewrite the amount because a response was slow.

A retry may reread and replan when facts change. Before acting, compare the new intent with the persisted one. If the amount, target, or meaning changed, create a new reviewable action rather than silently reusing the old key.

If the previous materially different action is still unknown, do not execute the new side effect merely because it has a new key. Reconcile or compensate it first, unless an authorized human accepts the duplicate risk.

Timeouts, retries, and ceilings

Set connection, provider-response, and overall step deadlines. The overall deadline includes waiting and retrying.

Retry only plausibly transient errors such as connection resets, rate limits, and temporary service failures. Do not retry invalid parameters, authorization failures, or business rejections such as “card declined” unless the business rule says to obtain a new payment method.

Use bounded exponential backoff with jitter. For example, a three-attempt policy might wait about 250 ms, 500 ms, then 1 second, with a small random addition. Backoff reduces load; it does not make a duplicate charge harmless. Use it only when the action is idempotent or reconciled first.

Persist hard ceilings: maximum model and tool steps, wall-clock deadline, token or spend budget, side-effect count, and per-action limits such as maximum refund amount. When a ceiling is reached, pause in needs_review; do not let the model plan one more attempt.

The awkward state: the tool succeeded, the answer vanished

The most dangerous status is not failed. It is unknown.

Represent it explicitly:

action: charge:v1:acme:renewal-2026-08-28-01
status: unknown
attempt: 1
provider_key: charge:v1:acme:renewal-2026-08-28-01

Then:

1. Query by idempotency key, merchant reference, or order ID.
2. Check webhooks or settlement events, allowing for eventual consistency.
3. If the charge exists, record its provider reference and mark it completed.
4. If the provider reliably confirms no charge, retry with the same key.
5. If neither result is trustworthy, stop and escalate.

An empty search immediately after a timeout may mean the provider’s read API has not caught up.

When rollback is impossible: compensating actions

Most side effects cannot be rolled back. You cannot unsend an email, erase a customer’s memory of a ticket, or guarantee that a settled charge can be instantly undone.

A compensating action reduces business harm: refund a charge, release a reservation, send a correction, or restore from backup. A saga is a workflow of local transactions where completed steps have planned compensations if a later step fails.

For example:

1. Reserve inventory.
2. Charge $49.90.
3. Create the shipment.
4. If shipment creation permanently fails, release inventory and issue a refund.

A refund is not a database rollback. It may take days, incur fees, or fail itself. Give compensations their own stable keys, durable status, retry limits, reconciliation, and audit trail.

Dry run, then confirm

For irreversible or high-impact actions, split preparation from authorization.

A dry run computes the exact effect without performing it and shows the target, payload, scope, amount, and policy checks. Confirmation must approve that exact payload, not the vague statement “the agent wants to proceed.”

Bind confirmation to the action ID and request hash. Expire it after a short period, such as 10 minutes, and reread volatile facts before acting. Approval must come from an authenticated actor through a channel the agent cannot forge. A model-generated confirmed: true is not human confirmation.

Decision table: where confirmation belongs

The deciding factors are irreversibility, blast radius, ambiguity, and financial or security impact.

Action	Automatic execution with idempotency	Dry run and confirmation	Human-only or escalation
Read records or search documentation	Yes	No	No
Routine internal ticket or approved message	Usually	For unusual text, recipients, or customer contact	Broad or sensitive audience
Charge, refund, or transfer money	Within an explicit low-value policy	Yes	Above the limit or ambiguous
Delete data or change permissions	Rarely	Yes, with an exact preview	Usually
Production deployment or infrastructure change	Only for narrow, preapproved automation	Yes	Schema, traffic, rollback, or data risk

Human confirmation does not replace authorization, validation, or idempotency. Humans double-click too. The system still needs a stable action key after approval.

The honest limitation

Idempotency is not exactly-once execution. It makes repeated attempts converge on one business result.

If a downstream system accepts a request, offers no idempotency key, exposes no reliable lookup, and emits no durable event, there is a fundamental uncertainty window. Your process can crash after the side effect and before recording it. No retry policy can prove whether the action happened.

The architectural options are to:

• put the effect behind a service you control with durable deduplication;
• use an outbox or queue whose consumer owns the idempotency boundary;
• require a provider reference that can be reconciled;
• make the action compensatable; or
• stop and ask a human when the outcome remains unknown.

This discipline costs latency and operational work through durable records, reconciliation jobs, confirmation screens, and paused workflows. For a read-only research agent, it may be excessive. For an agent that can move $10,000 or delete a customer database, skipping it is the greater risk.

Quick check

0/3

Q1

Retry immediately with a new idempotency keyMark the payment failed and continueKeep the action unknown, wait or reconcile through another reliable signal, then decideAsk the model to estimate whether the charge probably happened

Check answer

Q2

It is random, so each request can be tracedIt stays the same for every attempt of the same logical actionIt contains the model temperature and promptIt changes whenever the workflow is resumed

Check answer

Q3

Create a second label with a new random referenceRetry until one response arrivesSearch by the stable shipment reference first, record the existing label if found, and escalate if the search cannot establish the outcomeDelete the local operation row and start over

Check answer