Your agent process crashes after a tool has completed but before the workflow records the result. When the worker restarts, how should durable execution determine what to replay, what to query, and what must never run twice?

How to think about it

Replay the workflow from its durable history, reuse every result already recorded, and treat an external call with no recorded completion as ambiguous rather than failed. Query the downstream system using a stable operation ID; retry only when the operation is known not to have happened or the downstream API deduplicates retries, and never blindly repeat a non-idempotent side effect such as a charge, order, or email.

Why this is the hard part

A durable workflow is not just a process with a checkpoint. It is a process with a durable history: a record of decisions, inputs, outputs, and external effects that survives worker failure.

The worker may keep ordinary variables in memory:

next_step = "send_confirmation"
payment_status = "captured"

Those variables disappear when the process crashes. The durable history does not.

The key boundary is the external effect. An effect is an operation that changes something outside the workflow: charging a card, creating a ticket, sending an email, deploying a service, or calling a tool that mutates data.

A robust effect boundary records an intent before making the call:

EffectIntent(effect_id="capture:order-1842:v1", kind="capture", amount=4999)

It then invokes the tool, and records the result afterward:

EffectCompleted(effect_id="capture:order-1842:v1", result=...)

That ordering creates three useful recovery states:

Durable history	Recovery action
No effect intent	Record the intent, then perform the effect
Intent exists, completion is missing	Query or reconcile the external system
Completion exists	Replay the recorded result; do not call the tool

The dangerous state is the middle one. A missing completion record does not prove that the tool failed. The tool may have finished successfully, and the worker may have died during the next network write.

This is the interviewer’s real test: can you distinguish replaying computation from repeating an effect?

Replay means running deterministic workflow logic again against the durable history. If the history says that a tool returned “account balance: 120 dollars,” replay should supply that recorded answer. It should not call the banking tool again merely because the worker is rebuilding its in-memory state.

Repeating an effect means asking the outside world to do something again. That may be harmless for a read-only lookup. It may be catastrophic for a payment.

A concrete recovery

Imagine an order-support agent handling order 1842. It is authorized to capture 49.99 dollars from the customer’s payment method.

Before calling the payment provider, the workflow writes:

EffectIntent
effect_id = capture:order-1842:v1
operation = capture payment
amount_cents = 4999

The provider captures the money and returns transaction T-88421. Before the workflow appends EffectCompleted, the worker loses power.

When a new worker starts, it reads the history:

EffectIntent(capture:order-1842:v1)

There is no completion record. It must not simply call “capture 49.99 dollars” again.

Instead, it asks the payment provider for the status of the operation using the merchant reference or idempotency key capture:order-1842:v1.

If the provider says the payment succeeded, the workflow appends the missing completion locally, including transaction T-88421, and continues. The customer is charged once.

If the provider says no such capture exists, the workflow retries the capture using the same operation key. A payment provider that supports idempotency treats that retry as the same logical operation rather than a new charge.

If the provider says the operation is pending, or its status cannot be established, the workflow waits for reconciliation or sends the case to an operational queue. It does not guess. A second charge is worse than a delayed order.

The confirmation email is a separate effect. It needs its own durable intent and stable key, such as email:order-1842:confirmation:v1. Otherwise the workflow can correctly avoid a duplicate charge and still send the customer three identical emails during recovery. Distributed systems enjoy this sort of technicality.

What should replay, and what should not

Replay ordinary workflow logic: branching, validation, selecting the next agent step, and calculating values from recorded inputs. These operations should be deterministic, or their outputs should be recorded.

Replay recorded results for external calls. This includes tool results, model outputs when the workflow requires the same decision, generated timestamps, random identifiers, and other values that would change if recomputed. Calling the model again may produce a different tool choice, cost more tokens, and send recovery down a different branch.

Query systems of record for effects whose intent is durable but whose completion is absent. The system of record might be a payment provider, email service, deployment controller, ticketing system, or the database owned by another service.

Never run an irreversible or non-idempotent effect twice merely because the local history is incomplete. “Non-idempotent” means that doing the operation twice changes the outcome twice. Charging 49.99 dollars twice is the obvious example. Creating two production deployments or issuing two refunds are less obvious examples with equally unpleasant incident reviews.

Read-only queries can usually run again. An idempotent write can also be retried: setting a user’s status to suspended twice has the same final state as doing it once. But idempotence must describe the actual operation, not the developer’s intention. “Create an order” is generally not idempotent just because the caller hopes duplicate orders will be rejected.

The senior-level caveat: exactly once is not magic

A crash can happen between any two durable actions. No local worker can atomically commit “the remote payment succeeded” and “my workflow recorded that it succeeded” unless both participate in the same transaction, which most tool providers do not.

So the usual production guarantee is not literal exactly-once execution. It is at-least-once delivery plus deduplication, which gives exactly-once business effects for operations designed around stable idempotency keys.

The key must identify the logical operation, not the worker attempt. A retry after a crash must reuse capture:order-1842:v1; generating a fresh UUID for every retry defeats deduplication.

For a tool that offers neither idempotency keys nor a status query, the workflow cannot safely distinguish “never happened” from “happened just before the crash.” The honest choices are to add an adapter with a durable outbox and reconciliation, make the operation idempotent, or stop and require manual review. Blind retry is not recovery. It is a duplicate-effect lottery.

A checkpoint is useful for reducing replay time, but it is not the authority by itself. The durable effect history must be written with atomic append semantics, and concurrent workers need an atomic claim or equivalent coordination so two recoverers do not both attempt the same unresolved operation. Coordination reduces races; it does not remove the ambiguous crash window.

The failure symptom is usually immediate and concrete: duplicate charges, repeated emails, two tickets for one request, or a deployment that runs twice after a timeout. The root cause is often one of three mistakes: invoking the tool before persisting intent, generating a new effect ID on every retry, or treating a timeout as proof that the effect did not happen.

What they’ll ask next

What if the provider returns a timeout and has no status endpoint?
Treat the result as unknown. Do not blindly retry a non-idempotent operation. Put the operation into reconciliation or manual review, then redesign the integration around an idempotency key or queryable operation record.

Should every tool call be wrapped this way?
Every external effect should have a recovery policy. Read-only tools can generally be replayed or queried again. Mutating tools need an effect ID, durable intent, recorded completion, and either provider-side deduplication or a safe reconciliation path.

Does a database transaction solve the problem?
Only if the database transaction and the external effect are in the same atomic system. Otherwise, the transaction can record intent or completion, but a crash can still occur between the database commit and the remote call. That is why outboxes, idempotency keys, and reconciliation exist.

The line to say in the room

“Replay the history, not the side effects: a missing completion is an ambiguous operation, so query it by a stable idempotency key and retry only behind a deduplicating boundary.”