Last week#

One client learned to wait.

A delay that doubles on every attempt: 100 ms, 200, 400, 800.

This week#

A dependency recovers from an outage.

Every client that failed during it is, correctly, about to retry.

The reflex#

Trust that each client’s own backoff is enough.

Each one is behaving well in isolation.

Why they all return on the same second#

Ten thousand clients retrying on the same second is a thundering herd.

The recovering dependency is hit by a synchronised spike — and falls over again.

No single client did anything wrong.

Backoff never said when#

Week 3 fixed how long one client waits.

It says nothing about when that client waits, relative to the other nine thousand nine hundred and ninety-nine.

The fix#

Randomise each client’s wait inside its own backoff window.

On every attempt — not only the first.

What jitter takes away#

Nothing a single client can feel, which is why it gets left out of the implementation.

What changes is that no client can state its own worst-case wait any more — only the shape of the distribution it draws from.

Who decides#

platform.

No one client can see the other nine thousand nine hundred and ninety-nine. The fix has to be a property the platform requires of every client; one client adding jitter on its own changes nothing about the spike.

In the lab#

1. Run a simulated recovery with ten synchronised clients
2. Watch it fail
3. Add jitter to each client’s backoff, independently
4. Watch the same recovery succeed