Retries that help, and retries that make it worse

Almost every integration eventually grows a retry. A call fails, someone adds “retry 3 times”, the noise goes away, the ticket closes.

Sometimes that is right. Sometimes it converts a visible failure into an invisible one, sends a customer four copies of the same invoice, or takes down the service you were calling. The difference is not how many times you retry. It is whether you understood what failed.

Two kinds of failure

A transient failure is one where the same request, sent again later, has a real chance of succeeding. Nothing about the request was wrong. The system was momentarily unable to serve it: a connection reset, a lock timeout, an overloaded upstream.

A permanent failure is one where the same request will fail identically forever, because the request itself, or the state it refers to, is the problem: a malformed payload, a revoked credential, a closed account.

Retrying a transient failure is correct. Retrying a permanent failure is a loop that burns quota, delays the alert, and buries the cause under identical log lines.

So the first thing to build is not the retry. It is the classifier: something has to look at the failure and decide which bucket it is in. Everything else follows from that.

Status	Classification	Action
408, 429, 502, 503, 504	Transient	Retry with backoff
Network reset or timeout	Transient, with care	Retry only if the operation is safe to repeat
400, 422	Permanent	Do not retry, route to review
401, 403	Permanent	Do not retry, alert on the credential
404	Permanent	Do not retry, this is usually a logic bug
500	Ambiguous	Retry once or twice, then treat as permanent

The 500 row needs judgement: it might be a blip, or a deterministic crash on your payload. Two attempts survives a blip without hammering a broken endpoint.

Backoff, and why the jitter matters

Retrying immediately is the worst option, because an instant retry adds load at the moment an overloaded upstream can least absorb it.

Exponential backoff means each attempt waits longer than the last. A base of one second doubling each time gives 1s, 2s, 4s, 8s, 16s. Cap both the attempt count and the maximum delay.

Add jitter, a random offset on each delay. Without it, a hundred clients that failed during the same thirty second outage all retry at second 1, then all at second 3, then all at second 7. That is a synchronised load test against a system that just recovered, and a common way for a brief outage to become a long one. Full jitter is the simplest fix: instead of sleeping exactly base * 2^n, sleep a random duration between zero and base * 2^n.

One more rule that costs nothing: respect Retry-After. When an API sends that header, it is telling you when to come back. Your formula is a guess. The header is not.

Why retrying can be dangerous

Your service posts a payment to a provider. The provider receives it, charges the card, and starts writing the response. The connection drops before the response reaches you. Your client sees a timeout, classifies it as transient, and retries. The provider receives a second identical valid request and charges the card again.

From your side, “it failed” and “it succeeded and I did not hear about it” are indistinguishable. That is the fundamental problem with retrying any operation that changes state, and the same shape applies to sending an email, creating a ticket, or shipping an order. A timeout on a read is boring. A timeout on a write is a decision.

Idempotency keys

The fix is to make the second attempt provably the same operation as the first. You generate a unique key per logical operation and send the identical key on every retry. The server stores the key with the result, and on a repeat returns that result instead of redoing the work.

Two details decide whether this works:

• Generate the key once, when the operation is created, not inside the retry loop. A key generated per attempt is not an idempotency key, it is a random string. This is the most common way the pattern is implemented incorrectly.
• Derive it from something stable: the invoice id, or a UUID persisted with the record before the first call.

If the provider offers no idempotency support, record the intent in your own store before the call and check it before any retry. Weaker, since a crash between write and call leaves you unsure, but far better than nothing.

For operations you can neither make idempotent nor verify: do not retry automatically, reconcile instead. Query the provider for the current state, then decide.

What to do with a permanent failure

Classifying something as permanent means you stop retrying. It does not mean you drop it. Dropping it produces the silent failure this discipline exists to prevent.

A permanent failure should land somewhere with four properties:

1. Durable. A row or a dead letter queue, not a log line that ages out in seven days.
2. Complete. The payload, timestamp, status code, response body, and attempt count. Without the payload, the entry cannot be acted on and the queue becomes a graveyard.
3. Counted and age-alerted. Alert on the rate of new entries, and separately on the age of the oldest. A slow trickle is easy to ignore until it is a quarter of your volume.
4. Replayable. A documented way to push stored items back through once the bug is fixed. Design it before the incident, not during one.

A worked example

An order pipeline posts each paid order to a fulfilment API.

• Every order gets a UUID written to the local orders table before the first call. That UUID is the idempotency key and never changes across attempts.
• 429, 502, 503 and 504 retry with full jitter over a base of 500ms, up to five attempts, honouring Retry-After.
• 400 and 422 do not retry. The order moves to fulfilment_failed with the response body stored, and an operator sees it the same day.
• 401 pages whoever is on call, because a broken credential affects every order, not one.
• A timeout after the request was sent retries, but only because the key makes a duplicate harmless. Without it, the correct behaviour is to stop and reconcile.
• A daily job counts rows in fulfilment_failed older than 24 hours. Zero is expected, so anything else gets attention.

Notice how little of that is the retry. The retry is three lines. The classifier, the key, the failure store, and the alert are the actual work, and they decide whether the retry helps.