Retry storm

A retry storm is a self-inflicted outage. The original fault may last thirty seconds. The storm that follows can last much longer, because every client is now trying harder than it was before anything broke.

The mechanism is synchronisation. A service goes down at 14:00:00. Two thousand clients fail at roughly the same instant. Every one of them waits one second and retries, so at 14:00:01 the service receives two thousand simultaneous requests. It fails again. At 14:00:03 the same thing happens with a bigger queue behind it. Even after the underlying fault is fixed, the service comes back into a wall of traffic several times its normal peak and immediately falls over again.

Three things cause storms, and each has a direct fix.

• No jitter. Fixed delays keep clients in lockstep. Randomising the wait spreads the same volume over a window. See exponential backoff.
• No attempt cap. Infinite retries mean the load never decays, it accumulates.
• Retrying the wrong failures. Clients that retry 400s and 404s add traffic that could never have succeeded.

A concrete case. A webhook receiver goes down for two minutes. Two hundred senders each retry on a fixed 60 second interval with no cap. When the receiver restarts, it gets two hundred requests in the same second, exhausts its connection pool, and returns 502. The senders count that as another failure and retry again 60 seconds later. The system oscillates until somebody manually pauses the senders.

Defending from the server side matters too. A rate limit that sheds excess traffic with a 429 and a Retry-After header protects the service from clients that are behaving badly, and gives well-written clients the information they need to behave well.