datarekha

Durable execution for agents

An agent run isn't a request-response — it's a long workflow of LLM calls, tool calls, waits, and approvals. If the process crashes mid-run you lose all progress, and you can't blindly re-run because some steps already had side effects. Durable execution makes a workflow survive crashes and resume exactly where it left off.

8 min read Advanced Agentic AI Lesson 63 of 71

What you'll learn

  • Why long agent workflows need crash-proof state, not just retries
  • Workflows vs activities, and the durable event log
  • How replay reconstructs state without re-running completed side effects
  • Durable timers, the frameworks (Temporal, Restate), and when to reach for them

Before you start

A chat completion is over in seconds; an agent run is not. It’s a long workflow — a sequence of LLM calls, tool invocations, waits, and human approvals that can stretch from minutes to days. That length creates a reliability problem ordinary request-response code never faces: if the process crashes partway through (a deploy, an OOM kill, a network blip), you lose all the work so far. And you can’t just re-run it from the top, because some steps already had side effects — you must not re-charge the card or re-send the email. Durable execution is the runtime that solves exactly this.

Workflows, activities, and the event log

A durable-execution engine splits your code into two kinds of thing:

  • Workflow — the deterministic orchestration logic (the plan: do this, then that, then wait for approval). It must be replayable, so it can’t contain randomness or direct side effects.
  • Activities — the side-effectful steps: an LLM call, a tool, an API request, a payment. The engine persists each activity’s result to a durable event log as it completes.

That log is the whole trick. When the process dies and restarts, the engine replays the workflow code from the beginning — but instead of re-executing activities it has already recorded, it feeds back their stored results, deterministically rebuilding the state, and then continues from the point of failure. Completed side effects are never repeated:

Replay survives a crash — without re-running side effectsattempt 1reservechargecrashdurable event logreserve → ok · charge → $42attempt 2reservereplayedchargereplayed, not re-runshipexecutedlog feeds replay
The crash kills the process, not the progress. On resume, completed activities are replayed from the log (the card is not charged twice) and execution continues at the failure point.
history = {}                          # the durable event log: step -> recorded result

def activity(step, fn):
    if step in history:               # replay: return the recorded result, don't re-run
        print(f"  {step}: replayed (cached) -> {history[step]}")
        return history[step]
    result = fn()                     # first run: execute the side effect
    history[step] = result
    print(f"  {step}: EXECUTED -> {result}")
    return result

print("attempt 1 (crashes after charge):")
activity("reserve", lambda: "ok")
activity("charge", lambda: "$42 charged")
# ... process crashes here ...
print("attempt 2 (resume from durable history):")
activity("reserve", lambda: "ok")
activity("charge", lambda: "$42 charged")     # replayed — the card is NOT charged again
activity("ship",   lambda: "shipped")          # the new step, executed once
attempt 1 (crashes after charge):
  reserve: EXECUTED -> ok
  charge: EXECUTED -> $42 charged
attempt 2 (resume from durable history):
  reserve: replayed (cached) -> ok
  charge: replayed (cached) -> $42 charged
  ship: EXECUTED -> shipped

On the second attempt reserve and charge are served from the log — the payment isn’t repeated — and only the new ship step runs. The workflow continues as if the crash never happened.

Durable timers and the long wait

The same machinery enables something agents need badly: durable waits. An agent can “sleep” for hours or days — waiting for a human approval, a webhook, a scheduled time — and the engine survives restarts, deploys, and machine failures across that whole window. There’s no process sitting in memory holding a timer; the wait is just another entry in the durable state, resumed when the time comes.

In one breath

  • An agent run is a long workflow (LLM calls, tools, waits, approvals); a crash mid-run loses progress, and you can’t blindly re-run because of side effects (don’t re-charge, don’t re-send).
  • Durable execution splits code into a deterministic workflow and side-effectful activities, persisting each activity’s result to a durable event log.
  • On crash + restart it replays the workflow, feeding back recorded activity results instead of re-running them — reconstructing state and resuming at the failure point, with completed side effects never repeated (the demo: charge isn’t repeated).
  • Durable timers let an agent wait hours or days (for approval/webhook) across restarts, since the wait lives in durable state.
  • Use Temporal / Restate / DBOS / Step Functions (or LangGraph checkpointing) when an agent is long-running, stateful, waits on humans, or takes irreversible actions.

Quick check

Quick check

0/4
Q1Why can't you simply retry a crashed agent workflow from the beginning?
Q2How does durable execution reconstruct state after a crash?
Q3What is the difference between a 'workflow' and an 'activity' in this model?
Q4Which agent scenario most needs durable execution?

Next

Durable execution keeps long agent runs reliable; LangGraph persistence is the lighter in-framework version, and observability is how you watch those runs. For bounding what an autonomous agent is allowed to do, see agent safety controls.

Sign in to track your progress

Completed lessons, your XP, level, and streak save to your account — it's free and takes a few seconds.

Related lessons

Explore further

Skip to content