Backoff that works: jitter, caps, and the retry storm

Exponential backoff with a cap, full jitter, and a maximum attempt count. Without the jitter, clients that failed together retry together, and the retry itself becomes the outage.

The retry storm

A provider hiccups for two seconds. Four hundred of your workers get an error at the same instant. Each waits exactly one second and retries. Four hundred requests arrive in the same millisecond, the provider rate limits, everybody waits two seconds, and four hundred requests arrive in the same millisecond again.

The backoff is working exactly as designed and making things worse, because it is deterministic and every client shares the clock.

Full jitter

import random

BASE = 0.5          # seconds
CAP = 30.0          # never wait longer than this
MAX_ATTEMPTS = 6

def delay(attempt: int) -> float:
    # Full jitter: a random point in the whole window, not the window
    # plus a wobble. Spreads a synchronised herd across the interval
    # instead of nudging it.
    window = min(CAP, BASE * (2 ** attempt))
    return random.uniform(0, window)

The four parameters

Retry-After beats your arithmetic

When a 429 carries Retry-After, use it. It is not a suggestion and it is more accurate than any formula, because the server knows when its window resets and you are guessing.

What not to retry

Backoff is for temporary failures. Retrying a permanent one - an invalid recipient, a closed window, a rejected template - is a loop that fails identically until the cap. The full table of which is which is in designing a send queue.

Where to put it

In one place. Retry logic scattered across four call sites will have four different caps and one of them will be missing, and you will find out which during an incident.