Watching a breaker trip#
Week 8 — SLOP3092
← Back to the lecture pageLast week#
A classification table: 5xx and 429 retryable, 400 and 403 not.
Each caller still works it out one response at a time.
This week: a dependency down for minutes#
Not seconds. Long enough that no per-request backoff is going to help.
Every failure is correctly classified as retryable, and retrying is still the wrong move.
The reflex#
Keep retrying each request against the dependency individually.
The same schedule that worked for a two-second blip in week 3.
Every caller rediscovers the outage#
Every request waits out its full thirty-second timeout to learn the dependency is down.
The first failure already established that.
Closed. Open. Half-open.#
Closed#
Requests flow through normally.
The breaker is counting failures.
Open#
Enough requests have failed.
The breaker stops sending requests and fails them immediately — no more thirty-second waits to learn the same fact.
Half-open#
After a cooldown, exactly one probe is let through.
It asks whether the dependency is back. Closing again only if that probe succeeds.
The step that actually matters#
Skip half-open, and a breaker either never risks finding out the dependency has recovered, or floods it the instant it reopens.
That’s week 3’s problem again, delayed.
What an open breaker refuses#
While the breaker is open, requests fail that would have worked.
You are refusing calls on the evidence of earlier ones. That failure rate is visible and measurable; the one it replaces was spread across every caller and never counted anywhere.
Who decides#
platform.
A circuit breaker is shared state the whole client needs to see. No individual request can hold it on its own.
In the lab#
- Wire a breaker in front of a dependency seeded to fail for an extended window
- Watch requests stop reaching it once the breaker opens
- Confirm it closes again once the dependency recovers
- Confirm what happens when it doesn’t
This is the defence you’ll give in this week’s retry policy review.
Next week#
You failed requests on purpose, to stop waiting out timeouts for an answer you already had.
Now: what happens when the answer arrives after you’ve already stopped waiting.
← Back to the lecture page