Week 7: Not every failure can be retried away

How a status code and a poison message are the same problem, and where the judgement that separates them has to live.

19 April 2027

A malformed payload and a momentarily unreachable database can raise the identical exception, and a loop that catches Exception and retries treats them identically: the malformed payload is retried to the attempt limit every time it arrives, forever. HTTP has already done the classification — a 400 means the request is wrong and will be wrong on the next attempt, a 429 means try later and usually says how much later in Retry-After — but only for a client that reads the status code instead of treating every non-2xx response as one undifferentiated failure. Where there is no status code, the equivalent is a dead-letter queue: somewhere to put a message once further attempts are pointless, which does nothing at all unless somebody reads the queue.

Outline#

• transient failure, permanent failure, and the failures that look like both
• reading Retry-After instead of guessing
• dead-letter queues, and who is supposed to read them