Dead letter queue
A durable holding place for messages or records that could not be processed after their retries were exhausted, kept so they can be inspected and replayed rather than lost.
When a message fails permanently, you have two options: drop it, or keep it. Dropping it is how data goes missing without anyone noticing. A dead letter queue, often shortened to DLQ, is the second option made deliberate.
It does not have to be a message broker feature. A table called failed_events with the right columns does the same job for most automation work. What matters is not the technology but four properties.
- Durable. A row or a message, not a log line that ages out in seven days.
- Complete. The original payload, the timestamp, the error, the status code and the attempt count. Without the payload the entry cannot be replayed, and an entry that cannot be replayed is only a receipt for something you lost.
- Monitored. Alert on the rate of new entries and, separately, on the age of the oldest one. A slow trickle is easy to ignore until it is a quarter of your volume.
- Replayable. A documented, tested way to push entries back through the pipeline after the bug is fixed. Build this before the incident, because during the incident is a terrible time to invent it.
A worked example. An order pipeline posts to a fulfilment API. A schema change makes the API reject orders containing gift notes with HTTP 422. Those orders land in the DLQ with the payload and the response body. An age alert fires the next morning. The mapping is corrected, the eleven stored orders are replayed, and the idempotency keys stored with them guarantee the replay creates no duplicates.
The failure mode to avoid is the queue nobody reads. An unwatched DLQ is not a safety net, it is a silent failure with better bookkeeping. Every queue needs a named owner and an alert, or it is just a place where problems go to be forgotten.