Last week#

Ten thousand clients, each backing off correctly on its own.

All retried on the same second anyway.

Each one was correct in isolation, and together they were a spike.

This week: same shape, one request#

A request passes through three services stacked on top of each other.

The innermost one starts failing intermittently.

The reflex#

Let each layer retry the call below it up to three times.

Three retries, at any one layer, seems modest.

Twenty-seven#

Twenty-seven requests reach the failing dependency, for the one request the user made.

The retries compound multiplicatively up the stack. They do not add.

Nobody can see it happening#

Each layer only sees its own three attempts.

The total is only visible from a vantage point no single layer has.

The fix#

One retry budget, shared by the whole call chain.

Held as a fraction of recent calls rather than a count per layer, so retries throttle themselves as the ratio nears the cap.

What a shared budget refuses#

A layer that would have recovered on its second attempt is sometimes refused it, because another layer drew the budget down first.

The reason a call wasn’t retried now lives somewhere the person debugging it isn’t looking.

Who decides#

platform.

A per-layer count can only ever see its own three attempts. Keeping the total sane needs a budget the whole call chain draws from.

In the lab#

1. Trace a three-layer call stack with independent per-layer retries
2. Count the amplification by hand
3. Replace the per-layer counts with one shared retry budget
4. Confirm the total drops to three

Bring the amplified count to the first review.