RabbitMQ 4.3: When Poisonous Messages Stop Reaching the DLQ

A few years ago I wrote about why poisonous message handling matters and how we rely on the x-delivery-limit argument of our quorum queues to send a message that keeps failing to the service’s Dead Letter Queue (DLQ) after 10 requeues, so the 11th delivery goes to the DLQ. That safety net has been quietly doing its job ever since.

While preparing the upgrade from RabbitMQ 4.2 to 4.3, I read the 4.3.0 release notes line by line, as I do for every broker upgrade, and one bullet caught my eye:

Quorum queues now allow unlimited explicit message returns. The delivery limit is based on delivery-count rather than acquired-count, so messages can be explicitly returned to the queue without counting towards the delivery limit.

“Unlimited explicit message returns” sounded a lot like “unlimited retries of a poisonous message”. It turned out to be exactly that for some of our consumers.

What changed in 4.3

Before 4.3, a quorum queue checked x-delivery-limit against acquired-count, a counter that went up every time the message went back to the queue, whatever the reason. When it passed the limit, the message was dead-lettered. Simple.

Since 4.3.0 (rabbitmq-server#13885) the limit is checked against a second counter, delivery-count, that only goes up for returns the queue considers a failed delivery. The AMQP 0-9-1 operations now map like this:

requeue=false dead-letters on both, as before.

The two bold rows are the problem. basic.nack with requeue is now a plain return that the broker does not count. The consumer timeout went the same way. In 4.2 the channel was closed with PRECONDITION_FAILED and every unacked message on it was requeued and counted. In 4.3 the quorum queue itself returns only the timed-out messages, the channel sends the consumer a basic.cancel and stays open, and the return is not counted.

The result is a message redelivered forever instead of landing in the DLQ after 10 requeues. Nothing is lost, but the message holds a prefetch slot and burns CPU on every attempt.

Are you affected?

Through the consumer timeout, everybody with x-delivery-limit on a quorum queue is. Any handler that can legitimately take longer than the timeout (30 minutes by default) used to get the message dead-lettered after about five hours, 10 returns of the default. Now it loops while the original handler may still be running on the first delivery. A quick way to check is to query your traces for consumer spans longer than the timeout. In our case a handful of queues had deliveries between 30 minutes and a bit over 2 hours in the last month, so the loop was not hypothetical.

Through basic.nack, it depends on which operation your client library sends when a handler throws, and that is not something most of us ever look at.

Spring AMQP with SimpleMessageListenerContainer (the default @RabbitListener container) is not affected. When a listener throws, BlockingQueueConsumer.rollbackOnExceptionIfNecessary sends basic.reject with requeue, which 4.3 still counts. It only sends basic.nack for consumer batches. I had to read the bytecode of the spring-rabbit jar to be sure, as the docs do not say which one is used.

Spring AMQP with DirectMessageListenerContainer always nacks, so it is affected.

Raw RabbitMQ Java client code is affected if it calls channel.basicNack(tag, false, true) on an exception. We had exactly that in a small in-house wrapper around the Java client: any exception that escaped a handler, including the wrapper’s own JSON parsing before the handler even ran, ended in a nack with requeue. The one-line stopgap is basicReject(tag, true), which the broker still counts. The real fix is the guard below, which the wrapper got as well.

What the loop looks like in practice

We found out the hard way, in a development environment that was already on 4.3. About a hundred events pointed at records that did not exist in that environment, so the handler threw on every attempt, the wrapper nacked with requeue, and the broker handed the messages straight back. Together they were being redelivered about a thousand times a second.

Nothing paged. The consumer was up, the queue held a hundred messages and the pod was healthy. What gave it away was the telemetry bill: every attempt emitted a span with an error on it, and tail-based sampling keeps traces with errors, so none of it was dropped. By the time we noticed, that one service had produced about 2.5 billion spans.

The other fix was in the handler: it now acknowledges an event whose record does not exist, because retrying cannot make the record appear. A failure you know is deterministic should never be requeued in the first place. The delivery limit is the backstop for the ones you did not predict.

The fix: enforce the limit in the consumer

No broker setting restores the old behaviour, so we bounded the loop on the client side.

Since 4.3 every delivery from a quorum queue carries two headers, x-delivery-count and x-acquired-count. The second one still increments on every return, as acquired-count always did. On 4.0 to 4.2 the broker does not send x-acquired-count, so the guard can ship before the upgrade: it finds no header, does nothing, and the broker keeps enforcing the limit.

The guard is a listener advice that rejects without requeue any delivery whose x-acquired-count is above the delivery limit. Spring AMQP then dead-letters the message through the queue’s existing dead-letter exchange, exactly as the broker used to.

@RequiredArgsConstructor
public class DeliveryLimitInterceptor implements MethodInterceptor {

    static final String ACQUIRED_COUNT_HEADER = "x-acquired-count";

    private final int deliveryLimit;

    @Override
    public Object invoke(MethodInvocation invocation) throws Throwable {
        if (invocation.getArguments()[1] instanceof Message message && exceedsDeliveryLimit(message)) {
            throw new AmqpRejectAndDontRequeueException(
                    "Message acquired more than " + deliveryLimit + " times, dead-lettering it");
        }

        return invocation.proceed();
    }

    private boolean exceedsDeliveryLimit(Message message) {
        return message.getMessageProperties().getHeader(ACQUIRED_COUNT_HEADER) instanceof Number acquiredCount
                && acquiredCount.longValue() > deliveryLimit;
    }
}

The instanceof Message check matters: for a batch listener the second argument is a List, and the advice lets those through untouched.

Wiring it is one line in the container factory, appended after the tracing advice so the rejection still shows up in the right trace:

factory.setAdviceChain(ArrayUtils.addAll(factory.getAdviceChain(), deliveryLimitInterceptor));

The guard bounds the loop, but a handler should still stay under the consumer timeout, or a second consumer processes the same message while the first is still running. For the one listener that legitimately runs for two hours, 4.3 adds the x-consumer-timeout consumer argument, set per listener container through setConsumerArguments. It needs no queue redeclare and overrides both the queue argument and the broker-wide consumer_timeout. Raising the global timeout instead only delays the loop and lets any stuck consumer hold its prefetch for longer.

Do not add it to the DLQ consumer

This was the trap. In my earlier post I argued the DLQ needs its own x-delivery-limit, and we still have one. But the DLQ container must not get this interceptor.

When the broker dead-letters a message for reaching the limit, the delivery count travels with it into the DLQ. A message never returned from the DLQ has no acquired count of its own, so the first delivery reports the delivery count as x-acquired-count as well: already above 10, and the guard would reject it before the DLQ handler could log it.

Testing it against 4.3

Our container tests were still pinned to rabbitmq:4.0.9-alpine, so we bumped the shared Testcontainers image to rabbitmq:4.3.6-alpine and wrote three scenarios against a quorum queue declared with x-delivery-limit=10 and a dead-letter routing key.

The first one proves the problem and the fix in one go: nack the same message with requeue 11 times by hand, check the header reached 11 and the DLQ is still empty (the same assertion fails on 4.2), then start a listener container with the interceptor and assert the message lands in the DLQ without the listener ever being invoked.

@Test
void nackRequeuesAreNotDeadLetteredByTheBrokerButAreByTheInterceptor() {
    rabbitTemplate.convertAndSend(QUEUE, "poison");
    awaitMessageCount(QUEUE, 1);

    long acquiredCount = nackWithRequeue(DELIVERY_LIMIT + 1);

    assertThat(acquiredCount).isEqualTo(DELIVERY_LIMIT + 1);
    assertThat(messageCount(DLQ)).isZero();

    startListenerContainer(message -> listenerInvocations.incrementAndGet());

    awaitMessageCount(DLQ, 1);
    assertThat(listenerInvocations).hasValue(0);
    assertThat(messageCount(QUEUE)).isZero();
}

The second scenario checks that messages within the limit still reach the listener. The third one covers the consumer timeout: a handler that parks for three times the timeout, with the timeout lowered to 500 ms through the x-consumer-timeout consumer argument so the test does not take 30 minutes.

Takeaways

1. A release-note bullet phrased as a new capability can be a removed guarantee. Read them before every broker upgrade, including the quorum queue bullets that look like internals.
2. A delivery limit enforced only by the broker is one version bump away from not being enforced. The x-acquired-count header lets the consumer enforce it whatever operation the library sends on failure.
3. Keep handlers under the consumer timeout, or two consumers end up processing the same message at once. Give the few that cannot a longer timeout with the consumer argument.
4. Test the DLQ path against the real broker version. A test pinned to an old image keeps passing forever.
5. A looping message looks like a healthy consumer with a short queue. Alert on the redelivery rate, or on error spans per queue, not only on queue depth.

Check what your consumer sends before copying the fix.

Happy coding!