Reading the x-death header: which queue rejected the message, and why

Originally published at warrenops.io.

Every message in a dead-letter queue carries its own incident report. RabbitMQ writes it into the x-death header when it dead-letters the message. Most people have seen it in the management UI as a wall of nested tables and closed the tab. Here is how to read it.

Where it comes from

When RabbitMQ dead-letters a message, it does not just move it. It republishes the message to the queue's dead-letter exchange (set by the x-dead-letter-exchange queue argument or a dead-letter-exchange policy), optionally with a new routing key (x-dead-letter-routing-key). Before publishing, it modifies the headers:

• It adds or updates an x-death header, an array of tables.
• The first time a message is dead-lettered it also sets x-first-death-reason, x-first-death-queue and x-first-death-exchange. These are never changed afterwards.
• Recent RabbitMQ versions also set x-last-death-reason, x-last-death-queue and x-last-death-exchange, updated on every dead-lettering.
• If the message died because it expired and had a per-message TTL, the expiration property is removed (otherwise it would expire again in the DLQ) and preserved in the x-death entry as original-expiration.

Everything else, payload, properties, your own headers, is untouched.

The entry

A typical header, as the management UI or a client shows it:

x-death: [
  {
    "count": 3,
    "reason": "rejected",
    "queue": "orders.process",
    "time": 1759042215,
    "exchange": "orders",
    "routing-keys": ["order.created"]
  },
  {
    "count": 2,
    "reason": "expired",
    "queue": "orders.retry.wait",
    "time": 1759042155,
    "exchange": "orders.retry",
    "routing-keys": ["order.created"]
  }
]
x-first-death-exchange: orders
x-first-death-queue:    orders.process
x-first-death-reason:   rejected

Reading the array

The array is ordered most recent first: x-death[0] is the last thing that happened to the message. In the example above, the story reads:

1. The message was published to exchange orders with key order.created and landed in orders.process.
2. The consumer rejected it. It went to orders.retry.wait, a queue with a TTL and no consumers.
3. It expired there and came back to orders.process. Rejected again. Expired again. Rejected a third time.
4. After the third rejection (count: 3) the message ended up wherever you are looking at it now, presumably a final DLQ, because the consumer stopped requeueing after three attempts or the retry queue's TTL routed it elsewhere.

Note what the header does not tell you: which DLQ the message is in now (you know that, you are looking at it), and what went wrong in the consumer. For that you need the consumer's log at time, and time is the first rejection, not the last. If your retry cycle is 60 seconds and count is 3, the last rejection was about two minutes after time.

Same queue twice with different reasons

Because entries are keyed by queue and reason, a queue can appear twice: once with rejected, once with expired. That usually means a consumer was down for a while (messages expired) and then came back and rejected the same message. Both entries carry their own time, so you can tell when each phase started.

Why time does not update

People look at time, see a timestamp from three days ago, and conclude the message has been sitting in the DLQ for three days. It may have arrived a minute ago after its tenth cycle. To know when a message actually entered the DLQ you need something outside the header: a timestamp property your producer set (that is when it was published, also wrong), a message-level TTL trick, or a tool that remembers when it first saw the message.

What the header looks like for each cause

Messages with no x-death at all

Not every message in a dead-letter queue was dead-lettered by RabbitMQ. Many frameworks catch the exception and republish the message to an error queue themselves. The broker never dead-letters it, so there is no x-death. Instead you get framework headers:

• Spring AMQP (RepublishMessageRecoverer): x-exception-message, x-exception-stacktrace, x-original-exchange, x-original-routingKey.
• MassTransit (_error queues): MT-Fault-Message, MT-Fault-ExceptionType, MT-Fault-StackTrace, MT-Reason.
• NServiceBus: NServiceBus.ExceptionInfo.ExceptionType, NServiceBus.ExceptionInfo.Message, NServiceBus.FailedQ.

These are often more useful than x-death because they carry the actual exception. But the "where to send it back" question then has to be answered from x-original-* or NServiceBus.FailedQ, and there is no count. Your replay tooling should read both conventions.

How to look at it

Management UI

Queue page → Get messages, ack mode Nack message requeue true, and a small count. The UI renders x-death as nested tables. It works for one or two messages; for fifty it is unreadable, and the peek marks every fetched message as redelivered.

CLI

rabbitmqadmin -f raw_json get queue=orders.dlq ackmode=ack_requeue_true count=5 \
  | jq '.[] | .properties.headers["x-death"]'

-f raw_json makes rabbitmqadmin print the messages as JSON instead of a table, and jq pulls out the header. ack_requeue_true puts the messages back; they are marked redelivered afterwards, which is a property of RabbitMQ and cannot be avoided when peeking.

Code

def death_story(headers):
    for d in headers.get("x-death", []):
        keys = ",".join(d.get("routing-keys", []))
        print(f"{d['count']}x {d['reason']:<14} in {d['queue']}  "
              f"(from {d['exchange']!r} / {keys}, first at {d['time']})")

In the Java client the values arrive as List<Map<String,Object>> with LongString values for strings; call toString() on them before comparing.

Two rules for your own consumers

1. Read x-death[0].count to decide when to give up, not your own header. Your own header is lost if the message passes through a broker-side dead-lettering, and the broker's counter is authoritative. Look for the entry whose queue is your own queue, not blindly [0].
2. Strip the death headers when you deliberately republish for a fresh start. A replay tool should do the same. A message that keeps its x-death after a replay will be rejected on arrival by any consumer that follows rule one.

Where Warren fits

Warren shows each dead letter's x-death as a readable timeline: which queue, why, how often, when, and a plain-language verdict such as "rejected three times by orders.process after cycling through orders.retry.wait". Messages a framework republished itself are recognised by their x-exception-* or MT-Fault-* headers, and "dead since" is filled from when Warren first saw the message, which x-death cannot tell you.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · README on GitHub