The duplicate payout problem

The retry race: a timeout does not prove failure

Your application
      |
Send $500 payout
      |
Payout system creates payment
      |
Response is lost
      |
Your application sees timeout

This is the race that creates most duplicate payouts. The request succeeded, but the response never made it back. From where your application sits, a timeout, a dropped connection, and a request that never arrived all look identical.

If the application retries as a brand-new instruction, the recipient may get another $500. Job queues make the same mistake at scale: a worker crashes after sending but before recording the result, the queue redelivers the job, and a fresh worker pays the same person again.

Idempotency keys give the instruction an identity

Pay Alex $500
Instruction ID: athlete_bonus_82719

An idempotency key is a unique identifier your application attaches to each payout instruction. The payout system stores the key with the result of the first request it processed under that key. If the same instruction arrives again with the same key, the system returns the existing result instead of creating another payment.

That one property changes the math on retries. A retry stops being a gamble and becomes safe by construction: at most one payment can exist for a given key, no matter how many times the request is sent.

Key on the business event, not the request

Generating a fresh key per attempt does not help:

Attempt 1: key_123
Timeout
Attempt 2: key_456

The system sees two requests, and pays twice.

Derive the key from the business event instead: the obligation you are settling, not the HTTP call you are making. One bonus, one royalty period, one prize: one key.

Business event: athlete_bonus_82719

Attempt 1: athlete_bonus_82719
Timeout
Attempt 2: athlete_bonus_82719

This works best when the data model keeps the obligation separate from the attempts made to satisfy it. The obligation carries the stable ID; individual attempts can fail and be retried underneath it without minting new identities.

Obligation
$500 owed

Attempt 1
RTP
Failed

Attempt 2
ACH
Completed

Obligation
Satisfied

On a timeout, query before you retry

Idempotency keys make blind retries safe, but the cleaner habit is to not retry blind at all. When a request times out, first ask the payout system what happened to that instruction. A well-designed payout API lets you look up a payout by the same key you sent, so the timeout resolves into one of two known states: the payment exists, or it does not.

If it exists, record the result and move on. If it does not, retry with the same key. Either way you never act on a guess, which is the same discipline that makes handling failed payouts tractable: resolve the current status first, then decide.

Duplicate detection: the last line of defense

Idempotency stops the machine from duplicating itself. It does not stop people. A file can be uploaded twice. Two teams can initiate the same payout. An operator can retry a payment that already completed. The same recipient can appear in two batches.

So a payout system should also detect likely duplicates before money moves: flag suspicious combinations such as the same recipient, amount, reference, and time period, and hold them for review.

The banking system assumes some duplicates escape anyway. There is a dedicated ACH return code for duplicate entries, R24, which exists precisely because receiving banks see the same payment land twice often enough to need a standard way to send one copy back. Duplicates that slip past every control also show up as unmatched entries during payout reconciliation, where the ledger says one payment and the bank statement says two. Catching them there is late and expensive. Catching them at the key is free.

The rule

Before sending money again, answer one question:

Do we know the first payment did not happen?

If the answer is no, resolve the existing status first.