Retries, Idempotency and Recovery for Long-Running Agents

Evidence check: 25 October 2026. Product details reflect the referenced documentation observed at this cutoff; exact plans, prices, regional eligibility and unverified release dates are not claimed. This is an editorial guide, not a hands-on test.

Distinguish error classes

A transient 429 or network timeout can justify bounded retries with backoff. An invalid schema may justify one repair pass. A missing customer identifier needs human input. An authorization failure must stop, not prompt the model to find a different credential. An unexpected backend exception should surface to an operator. These cases need different recovery paths, and one blanket “retry three times” policy is insufficient.

Exactly-once is difficult at system boundaries

Suppose a payment request reaches a server but the acknowledgement is lost. A naive retry may create a duplicate. Design around idempotency keys, uniqueness constraints, an outbox of intended writes and a read-back reconciliation step. Record an operation identifier before execution and reconcile against the remote system after a timeout. If the service lacks safe deduplication, require operator intervention rather than automated replay.

Persist state deliberately

A state graph can checkpoint at node boundaries; a durable agent runtime can store progress and schedule recovery. Put source inputs, decisions, tool outputs, approval tokens and execution state in separate, scoped fields. Do not assume in-memory variables survive process restarts. The Cloudflare long-running-agent documentation explicitly distinguishes persisted state from timers and in-flight requests that are lost on eviction.

Limit the repair loop

Bound model steps, total wall time, maximum tool calls and spend. For every retry, record the previous error and reason for another attempt. A failing agent that continually reruns a search or browser action can exhaust budget or accidentally rate-limit a customer account. Use dead-letter queues or operator queues after defined thresholds.

Compensation is not always rollback

A refund, deletion or sent email may be irreversible. A compensating workflow can issue a corrective action, but it is a new side effect requiring authorization and audit. Describe the difference in the runbook so that “recovered” does not hide an unresolved external effect.

Operational test cases

Kill a worker after tool execution but before recording completion. Drop the network response. Deliver the same webhook twice. Resume after approval expiry. Revoke credentials during a run. A deployment should not pass simply because the happy-path demo finishes.

Reconcile ambiguous timeouts

Imagine a ticket-creation API receives a request, writes the ticket and times out before acknowledging it. Retrying with a new identifier may create a duplicate. Generate a stable operation ID before the first attempt, pass it through the gateway and query the authoritative system after a timeout. If the API has native idempotency support, use it; otherwise maintain a ledger that maps intent to observed external objects. A model should not choose whether a payment or message can safely be repeated.

Classify failures deliberately

Retry transient transport errors with exponential backoff and a cap. Escalate authentication and permission errors rather than repeatedly asking the model to try alternative tools. Treat schema validation failures as data problems and preserve the malformed payload for redacted debugging. Where operations cannot be undone, choose a forward-correction or operator-reconciliation strategy instead of describing rollback as guaranteed. Test crash recovery at every checkpoint and between a write and its acknowledgment.