Idempotency Keys in Distributed APIs: Preventing Duplicate Financial Transactions
Implementing robust idempotency with Redis distributed locks, PostgreSQL unique constraints, and replay headers.
1. Why Network Retries Require Idempotency
Every HTTP client library that implements retries — Axios interceptors, AWS SDK's built-in retry policy, a service mesh's retry budget — is making a bet: that resending an identical request is safe if the first attempt's response was never received. That bet is only true if the operation on the server side is idempotent. For a GET, this is free; GET has no side effects by contract. For a POST that charges a credit card or debits a wallet, it is very much not free, and a naive retry after a timeout can charge the customer twice for one purchase.
The failure mode that makes this unavoidable, not just theoretical, is the ambiguous timeout: your client sends a POST, the server processes it successfully and commits the charge, but the response packet is lost, or the connection drops before the client reads it. From the client's perspective this is indistinguishable from the request never reaching the server at all. A client that treats every timeout as 'definitely did not happen' and retries blindly will double-charge exactly in this scenario, and this scenario is common enough in real networks — load balancer connection draining during a deploy, a brief network partition, a client-side timeout set too aggressively — that you should assume it happens routinely, not rarely.
The fix is not to stop retrying — retries are essential for resilience against transient failures — but to make the operation itself safe to repeat. An idempotency key is the mechanism: the client generates a unique token per logical operation (not per HTTP attempt) and the server uses it to recognize a retry of the same logical intent as distinct from a new request, so it can return the original result instead of executing the side effect again.
2. The Idempotency Key Workflow
The client generates a UUID (v4 is fine; the only requirement is effectively-unique per logical operation) once, before the first attempt, and sends it as an `Idempotency-Key` header on every retry of that same operation — never a fresh key per retry, which would defeat the entire mechanism. This key must be generated at the point where the business intent is formed (the moment the user clicks 'Pay'), not regenerated by the retry logic itself.
On the server, the first thing the handler does — before touching any business logic — is attempt to atomically claim the key: `INSERT INTO idempotency_keys (key, status) VALUES ($1, 'processing') ON CONFLICT (key) DO NOTHING RETURNING *`. If the insert succeeds, this is a genuinely new operation and processing proceeds. If it conflicts, a request with this key was already seen, and the handler looks up its recorded outcome instead of re-executing anything.
Once the underlying operation completes, the handler updates that same row with the final status and the serialized response body, so a subsequent retry can replay the exact original response — same status code, same JSON body — rather than a generic 'already processed' message that the client would have to interpret differently from a normal success. This replay fidelity matters because client-side code paths for handling a 200 response should not need a separate branch for 'this was actually a replay'.