A payment tool times out after charging a card, so the agent retries and may charge the customer twice. How would you design idempotency, timeout handling, reconciliation, and compensation for agent-driven side effects?

How to think about it

I would give each business payment one durable idempotency key, enforce it in our database and at the payment provider, and treat a timeout after submission as unknown rather than failed. The agent may retry status checks or the same operation, but never create a new charge; a reconciler resolves unknowns, and a refund or void compensates only a confirmed duplicate or other completed charge.

Why this is difficult

The dangerous case is not an ordinary failure. It is an ambiguous outcome.

At 12:00:02, our service sends a request to charge a card. The payment provider accepts it, charges the card, and then the network drops the response. At 12:00:08, our service sees a read timeout. From the service’s point of view, both of these stories look identical:

• the provider never received the request;
• the provider completed the charge, but we never heard back.

A timeout is therefore not evidence of failure. It is evidence that we do not know the result.

An agent makes this more likely because it is another retrying client. The model may see “tool timed out” and call the tool again. The orchestration framework may retry too. The HTTP client may retry underneath. Three reasonable-looking retry layers can quietly turn one purchase into three payment attempts.

The first design principle is to separate the business operation from the tool call. A business operation is “pay order 8472 once for 4,900 cents.” A tool call is merely one attempt to learn or change its state. Retries can create new tool calls, but they must refer to the same business operation.

Idempotency belongs to the operation, not the model run

An idempotency key is a stable identifier that tells a system, “repeat requests with this key are the same operation.” For order 8472, we might create:

pay:order-8472:purchase:v1

The key should be generated and stored by our backend, not invented by the agent. It should remain stable across agent retries, worker retries, process restarts, and a human reopening the checkout page.

Our database needs a uniqueness rule as well as the provider’s idempotency feature. The database prevents two internal payment records for the same order and purpose. The provider prevents repeated submissions from becoming repeated charges.

CREATE TABLE payment_operations (
  order_id            text NOT NULL,
  purpose             text NOT NULL,
  idempotency_key     text NOT NULL UNIQUE,
  amount_minor        bigint NOT NULL,
  currency            char(3) NOT NULL,
  status              text NOT NULL CHECK (
    status IN (
      'created', 'submitting', 'pending', 'succeeded',
      'declined', 'refund_pending', 'refunded'
    )
  ),
  provider_payment_id text,
  PRIMARY KEY (order_id, purpose)
);

The application transaction creates this row before submitting money movement. A second request for order 8472 finds the existing row and returns its current state. It does not create another payment operation.

The provider must also bind the key to the request parameters. Reusing the same key with 4,900 cents and then trying 5,900 cents should be rejected, not interpreted as an update. Otherwise a stale retry could charge a different amount.

This is not exactly-once execution. Exactly once is generally impossible across our database, a network, an agent runtime, and an external payment system. The practical goal is stronger and more useful: at most one financial effect for one business operation, followed by eventual knowledge of its result.

The concrete flow

Suppose a customer asks an agent to pay a $49.00 invoice.

1. The backend creates operation pay:order-8472:purchase:v1, with amount 4900 and currency USD.
2. It records the operation as created.
3. A worker submits it to the provider using the stored key and a merchant reference for order 8472.
4. The provider accepts the payment at 12:00:02.
5. Our five-second provider request deadline expires without a response.
6. The agent-facing tool returns pending, not failed.

The agent can safely ask for the operation’s status. It can also resubmit the same payment operation using the same idempotency key if the provider supports that behavior. It must not generate pay:order-8472:purchase:v2 merely because the first call timed out.

If the provider says the payment succeeded, we record the provider payment identifier, mark the operation succeeded, and fulfil the invoice. If it says declined, we mark it declined. If the provider cannot answer, the operation remains pending and goes to reconciliation.

The agent should never receive raw authority to call an arbitrary “charge card” endpoint. It should receive a narrow payment tool whose backend enforces the order, amount, currency, customer authorization, state transitions, and idempotency key. The model can request an action. It cannot redefine what “the same action” means.

Timeout handling: distinguish waiting from failure

Use separate deadlines for the agent conversation and the payment workflow.

For example, the user-facing tool may wait eight seconds. The durable worker may continue trying to resolve the operation after the agent turn ends. That lets the agent say, “Payment is still processing,” instead of guessing that the payment failed.

A safe result contract has at least these semantic outcomes:

Result	Meaning	Safe next action
Succeeded	Provider confirmed the payment	Fulfil once
Declined	Provider definitively rejected it	Show failure or request another method
Pending	Submission or provider result is unresolved	Poll, await webhook, or reconcile
Requires customer action	The payment needs a customer step	Ask the customer to complete it
Refunded	Compensation completed	Do not fulfil from this operation

A connection timeout before a request is visibly sent may be less suspicious, but it is not automatically safe to retry. DNS, connection establishment, proxies, and provider gateways can all hide whether the request arrived. The system should use the same operation identity unless it has a provider-backed reason that the operation never existed.

Webhooks help, but they do not eliminate reconciliation. Webhooks can be delayed, duplicated, delivered out of order, or lost. Verify their authenticity, persist a provider event identifier so duplicate events are harmless, and make state transitions monotonic. A late “payment succeeded” event must not be able to resurrect an operation that was already refunded without a deliberate transition.

Reconciliation is the safety net

A reconciler is a scheduled process that compares our payment records with the provider’s records. It examines every pending or otherwise suspicious operation, using the provider payment identifier, merchant reference, or settlement report, depending on what that provider supports.

For the example above, the reconciler might find:

• our database: pending, order 8472, 4,900 USD;
• provider: captured, provider ID P123;
• internal ledger: no captured payment yet.

It then records P123, marks the operation succeeded, and lets fulfilment proceed exactly once.

The important invariant is that fulfilment also needs idempotency. If two workers both observe succeeded, they must not ship two products or grant two subscriptions. Payment idempotency protects the money movement; fulfilment idempotency protects the business consequence.

The first practitioner-visible symptom of a weak design is often a customer saying, “I was charged, but your app says payment failed.” The second is a pair of provider transactions with the same amount and order reference within a few seconds. Monitor both. Alert on unresolved payments older than the normal provider window, duplicate captures, and orders whose payment and fulfilment states disagree.

Compensation is not rollback

A payment cannot be rolled back like a database update. Compensation is a new business action that offsets a completed side effect.

If reconciliation proves that two captures happened, compensate the duplicate. Depending on the payment lifecycle, that may mean voiding an uncaptured authorization or issuing a refund for a captured payment. The refund itself needs a stable idempotency key such as refund:P123:duplicate-v1, and it must pass through the same pending-and-reconcile flow. A refund timeout is another ambiguous outcome, not proof that no refund happened.

Do not automatically refund merely because our request timed out. That can create a worse result: the original charge succeeds, the refund succeeds, and a later retry creates the intended charge again. First determine what happened. Then compensate a confirmed duplicate.

There is a real trade-off here. Keeping an operation pending may delay checkout and annoy a customer. Retrying with a fresh key may improve apparent conversion while risking an extra $49 charge. For money movement, a short period of uncertainty is usually the cheaper failure. Set an explicit escalation policy: for example, after 15 minutes without a provider answer, stop automatic retries and send the case to operations. The exact interval depends on the provider and payment method; it should be measured, documented, and tested.

What they’ll ask next

What if the provider does not support idempotency keys?
I would not claim exactly-once charging. I would use our durable operation record, a unique merchant reference if the provider can search by it, a single serialized payment worker, and reconciliation against provider reports. For high-risk flows, authorization followed by separately controlled capture can reduce the blast radius. Ambiguous cases may require manual review.

How do you stop the agent from causing duplicates?
The backend owns operation identity and state. The agent can request “pay order 8472,” but it cannot supply a new key for each attempt or bypass a pending operation. The tool returns structured states, and policy forbids a second charge while one operation is unresolved.

What if the refund times out too?
Treat the refund as its own idempotent operation, persist its key, query or reconcile its status, and do not submit another refund with a new key. Compensation needs the same reliability design as the original payment.

The sentence I would use in the room: “A timeout after submission means unknown, never failed: I persist one business operation, retry only with its identity, reconcile before compensating, and make both charges and refunds idempotent.”