Idempotency for Agent Actions

Originally published at Trida AI.

Copy as markdown

An agent issues a refund. The payment call times out. Nothing in that timeout tells the agent whether the money moved. The request may have died on the way out, or the refund may have completed while the response died on the way back. Both worlds look identical from where the agent sits.

So it retries. That is what it was built to do, and in the second world the customer has now been refunded twice.

This is not a reasoning failure. The model cannot reason its way to a fact the network didn't return. It's a plumbing failure, and it has a known fix that predates agents by years. The reason it keeps biting agent systems is that agents retry from more places than ordinary software does, and most tool layers are still built as if a call either happened or it didn't.

Reads forgive. Writes don't.

A retried read costs you a second lookup. A retried write can cost you a second refund, a duplicate order, a duplicate ticket, or two copies of an email a customer will read as carelessness. The dividing line in any agent is exactly there: which of its tools change something outside itself?

Agent stacks multiply the retry sources. The model reconsiders and calls the same tool again with slightly different wording. The framework retries a failed step. The SDK retries a transient error nobody configured. A queue redelivers a message, a worker restarts mid-run and resumes from a checkpoint that predates the side effect. Each layer means well. Stacked together, one intended action can leave the building three or four times, and no single layer sees the others doing it.

Ordinary services learned to name this property. An operation is idempotent when repeating it with the same input produces the same end state: the effect happens once, however many times the request arrives. Agents need that property at every tool that writes, and they need it designed in, because it cannot be prompted in. No instruction in a system prompt survives a worker restart.

One key for the whole job

The mechanism is a key. Before the first attempt, the system mints one identifier for the intended action (this refund, for this order), writes it down somewhere that survives a restart, and sends that same key with every attempt. The receiving side records the outcome against the key. A repeat with the same key gets the recorded outcome back instead of a second execution.

Stripe's API is a reference implementation most teams meet early, and it is explicit that it supports idempotency for safely retrying requests without performing the same operation twice. The details matter more than the headline. The key is client-generated, with V4 UUIDs suggested. In API v1, the result of the first request is saved once execution begins, and retries return that saved result. Reuse the same key with different parameters and the request errors rather than quietly returning the wrong response. And keys can be pruned once they are at least 24 hours old, after which a reused key starts a new request. The provider's memory sets a horizon, and any workflow that might retry beyond a day needs its own record as well.

Two design choices decide whether this works in an agent. First, where the key comes from. A fresh random key per attempt is useless; a fresh random key per run is better, but it still misses the duplicate that matters most in production, where two separate runs get launched for the same job. Deriving the key from the business identity, the order, the ticket, the invoice being acted on, means retries, restarts and duplicate runs all collide on the same key and collapse into one effect. Second, when the key is written down. It has to be persisted before the side effect, not after. A key recorded in a success handler is a key that a crash, mid-action, takes with it.

The action ledger: an intended action gets one key, minted and persisted in the ledger before any call is made; the tool executes downstream once and its result is stored against that key, so a retry carrying the same key is answered from the ledger and never reaches the downstream system a second time.

When there is no key, reconcile

Plenty of the systems agents act on offer nothing of the sort. No key header, no deduplication, just an endpoint that does the thing every time it is called. Sending again and hoping is not a strategy for those; the timeout case has to route somewhere else.

Amazon's Builders' Library walks through the same dilemma for a provisioning call: after a timeout, it may be unclear whether the workload is running, and simply retrying could create a second one, so the caller is left performing a reconciliation to find out what actually happened. Its design answer is a unique client request identifier carried with the request, so a retry can be recognised as the same request rather than a new one.

Where you cannot add that identifier to the downstream call, imitate it from your side. Attach your own reference to the action wherever the target system lets you (a client reference on the record, your key in a notes or metadata field) and make the timeout path a lookup before it is ever a resend: search by that reference, adopt the record if it exists, resend only if it provably doesn't. If the system can't be queried either, the honest options narrow to a status check by other means or a person. An ambiguous write that a human resolves in a minute is cheaper than a duplicate a team unwinds in a week.

One retry owner, or the retries multiply

Even safe retries need a budget and a single owner. When every layer of a stack retries independently, the counts don't add, they multiply: Amazon's retry guidance describes a five-deep call stack with three retries at each layer, where a failing database ends up taking 243x the load, and the same arithmetic applies to side effects when the layers sit in front of a write. Pick the layer that owns retries for each action, turn the others off for that action, and give the owner a cap, exponential backoff and jitter. The agent loop itself is usually the worst owner: it retries semantically, rephrasing as it goes, which defeats exact matching everywhere downstream. Retries belong in the tool layer, where the key and the record live.

One refund, traced

Say the action is a refund for a returned order, and the payment provider's response is lost to a timeout on the first attempt.

One refund traced end to end: the intent and its key are recorded before the call, the first attempt completes downstream while its response is lost, the retry arrives carrying the same key and returns the original refund record instead of refunding again, and the customer ends with one refund and one confirmation.

The intent is recorded first: refund this order, key derived from the order itself, status in progress. The first attempt reaches the provider and the refund is created; the response never arrives, so the agent knows only that the attempt timed out. It retries with the same key and the same parameters. The provider finds the key, returns the original refund, and creates nothing. The ledger marks the action complete with that refund's identifier, and the customer receives one confirmation, because the confirmation step keys off the completed record rather than off the attempt count.

Nothing in that trace required the model to be careful. The safety sits underneath it, which is where safety has to sit in a system whose top layer is probabilistic.

Where this stops helping

A key makes a repeat safe. It does not make the first attempt right. An agent that refunds the wrong order, confidently and exactly once, has still refunded the wrong order. Approval rules and evals carry that problem, not idempotency.

The in-progress case is genuinely hard. If a worker dies after the downstream action completes but before the result is recorded, the record sits at in progress and the next run has to decide whether that means "still happening" or "happened, unrecorded". Mature implementations expire in-progress records after a bounded window: the Powertools for AWS Lambda idempotency utility, for one, expires an in-progress execution if the function times out halfway through. Expiry is a judgment call about how long the action can legitimately take, and getting it wrong in either direction reopens the duplicate or strands the work.

And the window problem cuts both ways. Keys expire, ledgers get pruned, providers forget. A retry that arrives after the record is gone is a new action again. Set retention from your real retry horizon, including the human who re-runs a failed job on Monday, not from a default somebody else picked.

Work with us

If your agents can send, charge, book, create or delete anything, the duplicate-action question is already sitting in your retry paths, whether or not it has fired yet. Working through which actions need keys, ledgers and reconciliation, before a customer meets the same refund twice, is deployment work we do. Tell us about it through Trida AI's intake.