Last week#

Two operators, one payout, and a runbook line standing in for a lock.

The failure had no code in it and behaved exactly like the ones that did.

This week: two errors, one retry loop#

Two failures that will never succeed however often they are repeated.

A malformed message a queue consumer cannot parse, and a 400 saying the request itself is wrong.

At the instant they fail, both look like the transient faults every earlier week retried successfully.

The reflex#

The retry wrapper does not read the failure.

An exception is an exception and a status code is an integer, so both go round again.

What that does#

The malformed message goes back onto the queue, blocks the messages behind it, and occupies the consumer for as long as anyone lets it.

The 400 draws down week 5’s shared budget and will never succeed.

Meanwhile a 429 in the same batch — the one response explicitly asking the client to slow down — gets retried immediately.

The fix#

Read the failure before deciding.

A 429’s Retry-After overrides whatever schedule the client had planned. A message that has failed a bounded number of times goes to a dead-letter queue rather than back onto the main one.

What the classification depends on#

Somebody else’s server using status codes correctly. A dependency that answers 400 for a transient fault now fails permanently on the first attempt.

And a dead-letter queue only fixes something if a person reads it. Unread, it is the same poison message with better filing.

Who decides#

library.

A consumer sees one message and a call site sees one response, so neither has the memory to notice it is looping.

The library is where the attempt count and the classification table both live long enough to be right, and the only place where correcting the table corrects every caller at once.

In the lab#

1. Sort a log of mixed failures — malformed payloads, 400s, 429s, timeouts — into retryable and not
2. Wire the classification into both a consumer and a client
3. Confirm the poison message reaches the dead-letter queue
4. Confirm the 429 waits out its Retry-After