Durable execution for agents

At 3:07 a.m., an order agent has almost finished its work.

It reserved a laptop, checked the fraud score, waited six hours for a human approval, and charged the customer’s card for $42. Then the worker was killed during a deployment, one step before creating the shipping label.

The obvious recovery is to run the agent again from the beginning. That is also how you charge the card twice.

A normal request-response handler assumes the whole job lasts seconds and can be safely retried. An agent workflow may last days. It mixes reasoning with payments, emails, database writes, webhooks, and pauses for people. Some actions can be repeated; some cannot.

Durable execution runs that workflow so its progress survives crashes, machine replacement, deploys, and long waits. The runtime records what happened, reconstructs the workflow after failure, and continues from the last known point instead of starting over.

The important word is not merely durable. It is execution. You are making the running process recoverable, not just saving a final JSON document after the work is over.

Workflows and activities

This section uses a Temporal-like code-first runtime: the runtime re-executes workflow orchestration code and uses durable history to rebuild its state. That is one form of durable execution, not a universal property of every workflow engine.

A state-machine system, such as AWS Step Functions, persists execution state and resumes from a state or transition. It does not replay arbitrary workflow code. A journaled-step or checkpointing system, such as Restate, Inngest, DBOS, or framework persistence, may memoize or replay work at step or checkpoint boundaries. Determinism, retry behavior, and delivery guarantees depend on the engine.

In the code-first model, a durable-execution engine separates an agent into two kinds of code.

A workflow is orchestration: reserve the item, request approval, charge the card, then ship it. It decides order, branching, retries, and waiting. The workflow must be deterministic: the same recorded inputs must produce the same decisions when the code runs again.

An activity is work that touches the outside world. An LLM call, payment request, database write, email, and shipping-provider call are activities. They may be slow, fail, or have side effects.

The engine stores activity results in a durable event log, an append-only history of facts such as “activity reserve completed with result ok” or “timer approval-deadline fired.” The log is the source of truth for recovery, not merely a human-readable application log.

Suppose the order workflow reaches these events:

1. reserve returns ok.
2. fraud_check returns a score of 0.08.
3. request_approval waits for a human.
4. The approval signal arrives.
5. charge returns $42 charged.
6. The worker crashes before ship starts.

When a new worker picks up order-8472, it runs the workflow code from the beginning. For every activity already in the event log, the engine supplies the recorded result instead of calling the activity again. The workflow rebuilds its local state, reaches ship, and executes only that step.

That process is called replay. Replay reconstructs the decisions that led to the payment; it is not a second attempt at the payment.

Here is the mechanism in a deliberately tiny Python model:

history = {}                          # same-process teaching prop, not durable storage

def activity(step, fn):
    if step in history:               # replay: return the recorded result, don't re-run
        print(f"  {step}: replayed (cached) -> {history[step]}")
        return history[step]
    result = fn()                     # first run: execute the side effect
    history[step] = result
    print(f"  {step}: EXECUTED -> {result}")
    return result

print("attempt 1 (same-process simulation; pretend crash after charge):")
activity("reserve", lambda: "ok")
activity("charge", lambda: "$42 charged")
# A real process would stop here. This demo keeps running with the same history.
print("attempt 2 (same process, reusing history):")
activity("reserve", lambda: "ok")
activity("charge", lambda: "$42 charged")     # replayed — the card is NOT charged again
activity("ship",   lambda: "shipped")          # the new step, executed once

attempt 1 (same-process simulation; pretend crash after charge):
  reserve: EXECUTED -> ok
  charge: EXECUTED -> $42 charged
attempt 2 (same process, reusing history):
  reserve: replayed (cached) -> ok
  charge: replayed (cached) -> $42 charged
  ship: EXECUTED -> shipped

The dictionary is only a teaching prop: it disappears when the process exits. A real engine stores history durably and coordinates workers claiming workflow tasks. The important separation is that the workflow requests reserve, charge, and ship, while the activity wrapper either executes a new request or returns a recorded result.

Why the workflow must be deterministic

Replay is safe only if orchestration makes the same decisions from the same history.

if random.random() < 0.5:
    send_to_manual_review()
else:
    charge_card()

A different random value during replay could change the workflow’s story. The same problem occurs with direct reads of current time, unrecorded UUIDs, thread scheduling, mutable global state, or database reads inside workflow code. A deployment can create it too if old history reaches a branch the new code removed.

Use these boundaries:

• Keep sequencing and policy in the workflow.
• Put randomness, time, network calls, model calls, and database reads behind activities or deterministic runtime functions.
• Record identifiers and external results that affect future decisions.
• Treat workflow changes as compatibility changes.

An LLM call belongs in an activity even when it is “just reasoning.” Record the response used by the workflow; on replay, use that response rather than asking the model again. Otherwise a changed answer could determine whether the agent charges $42, requests approval, or cancels.

Replay still runs orchestration code. It substitutes history at completed activity boundaries, which is why workflow code should be small and explicit.

The uncomfortable part: exactly once is not automatic

“Completed activities run only once” is true only after the runtime knows the activity completed.

Consider a payment:

1. The worker sends charge $42 to the provider.
2. The provider charges the card.
3. The worker dies before reporting the result.
4. The engine sees no completion event and retries.

The provider sees a paid order; the engine sees an unknown attempt. The retry can charge twice. Durable execution cannot make two independent systems commit as one atomic transaction.

A side-effecting activity needs idempotency: repeating the same request produces the same external outcome instead of applying the effect again. Send a stable key such as order-8472-charge-v1 with the original request and every retry. A provider that supports this contract returns the original result for a repeated key.

When no provider idempotency contract exists, reconcile with the provider before retrying, or add deduplication at a destination you control. Email commonly offers no exactly-once send guarantee. An outbox makes an intended message durable in the same database transaction as a business change, but it does not make delivery to an arbitrary external system exactly once.

A saga uses compensating actions, such as releasing a reservation after a later payment failure. Compensation is not an undo button: a payment may be refundable, but an email cannot be unsent.

In a Temporal-like code-first runtime, workflow decisions replay consistently when determinism rules are followed. A scheduled activity with retries may have multiple attempts, including one whose external result is unknown; it may also never execute if the workflow is cancelled before it starts. Verify the chosen engine’s retry and delivery semantics rather than treating them as exactly-once side effects.

Durable timers and human approval

A durable timer stores its deadline in the workflow engine instead of holding it in a live process. The order workflow can request approval with a 24-hour deadline, then let every worker disappear. The engine stores the deadline and creates work when it arrives; no container needs to remain in memory or keep an HTTP request open.

A signal is an external event delivered to a waiting workflow. An approval service might send approved, the approver ID, and a timestamp. The workflow records the signal and proceeds to charge.

Approval and timeout can race. Design one recorded decision and make the losing event harmless or auditable. Correlate approval messages with order-8472, reject stale approvals, and make cancellation idempotent.

A production shape

A typical order workflow is:

reserve → fraud_check → wait_for_approval → charge → create_label → notify_customer

The workflow owns sequence and policy. Activities own inventory, model, payment, carrier, and email interactions.

Use a stable workflow ID and the runtime’s idempotent-start or unique-ID semantics. If unavailable, deduplicate starts in a transactional store. Give each external operation a stable idempotency key when its provider supports one, and reconcile ambiguous outcomes before retrying otherwise. Set activity timeouts, classify permanent versus transient failures, and use backoff rather than retrying a declined card as though it were a network error.

For long activities, use heartbeats when supported. They help detect a dead worker, but they do not replace idempotency.

Keep workflow state small. Store large transcripts or PDFs in durable object storage and keep immutable references in history; replaying large payloads makes recovery slow and expensive. Observe workflows and activities by stable IDs so you can tell whether order-8472 is waiting, retrying charge, or replaying.

Choosing the machinery

Ask what must survive, and for how long.

Verify what the selected engine persists, whether it replays code or resumes states, how step boundaries are memoized, how retries are delivered, and what happens when an activity dies after the external system acts.

Failure modes

Duplicate side effects usually mean an activity acted externally and died before recording success. Use a stable idempotency key, reconcile unkeyed attempts, and do not disable retries as a “fix.”

Replay nondeterminism often appears after deployment when a branch, iteration order, time read, or model response changes. Preserve old behavior for in-flight workflows or use the engine’s versioning mechanism.

Stuck workflows may be waiting for a signal, timer, or retry. Check signal correlation and add deadlines to callbacks; a callback without an expiry is a leak.

Slow or expensive replay means history or payloads have grown too large. Keep state compact, store large immutable data externally, and use history compaction or “continue as new” where supported.

The honest cost

Durable execution adds infrastructure or managed-service cost. Every event consumes storage, and every activity needs timeout, retry, and idempotency decisions. Code-first replay also imposes determinism rules.

It does not make a bad process correct: it can preserve an incorrect recipient or decision with perfect consistency. It does not replace authorization, agent safety controls, business invariants, human approval, or reliability controls around agent side effects.

Use it when losing or repeating work costs more than operating the runtime. A two-second, read-only answer usually fails that test; a three-day workflow that can issue refunds usually passes it.

Quick check

Next

Durable execution keeps long agent runs alive through failure. LangGraph persistence is the lighter in-framework cousin, while observability shows what those runs are doing. For autonomous-action boundaries, read agent safety controls.