Watching a breaker trip

Week 8 — SLOP3092

← Back to the lecture page

Last week

A classification table: 5xx and 429 retryable, 400 and 403 not.

Each caller still works it out one response at a time.

This week: a dependency down for minutes

Not seconds. Long enough that no per-request backoff is going to help.

Every failure is correctly classified as retryable, and retrying is still the wrong move.

The reflex

Keep retrying each request against the dependency individually.

The same schedule that worked for a two-second blip in week 3.

Every caller rediscovers the outage

Every request waits out its full thirty-second timeout to learn the dependency is down.

The first failure already established that.

Closed. Open. Half-open.

Closed

Requests flow through normally.

The breaker is counting failures.

Open

Enough requests have failed.

The breaker stops sending requests and fails them immediately — no more thirty-second waits to learn the same fact.

Half-open

After a cooldown, exactly one probe is let through.

It asks whether the dependency is back. Closing again only if that probe succeeds.

The step that actually matters

Skip half-open, and a breaker either never risks finding out the dependency has recovered, or floods it the instant it reopens.

That’s week 3’s problem again, delayed.

What an open breaker refuses

While the breaker is open, requests fail that would have worked.

You are refusing calls on the evidence of earlier ones. That failure rate is visible and measurable; the one it replaces was spread across every caller and never counted anywhere.

Who decides

platform.

A circuit breaker is shared state the whole client needs to see. No individual request can hold it on its own.

In the lab

  1. Wire a breaker in front of a dependency seeded to fail for an extended window
  2. Watch requests stop reaching it once the breaker opens
  3. Confirm it closes again once the dependency recovers
  4. Confirm what happens when it doesn’t

This is the defence you’ll give in this week’s retry policy review.

Next week

You failed requests on purpose, to stop waiting out timeouts for an answer you already had.

Now: what happens when the answer arrives after you’ve already stopped waiting.

← Back to the lecture page