As part of the increased resilience for our RabbitMQ cluster, we added the argument x-delivery-limit to our service’s quorum queues in order to avoid poisonous messages in our service queues.

The idea is that if we have a message that cannot be deserialized in our custom object types (or other errors, like NPE), it would try to perform the operation for a finite number of times, in our case 10, and after that, the not consumable message would be routed to the service’s DLQ.

After the message arrives in the DLQ, it would be consumed as a default Message, logged the message metadata for debugging purposes, and after we would just discard the message.

When the DLQ Message Is Also Poisonous

One possibility that we didn’t think of was that even the basic Message maybe was not able to be received by the consumer, being itself a poisonous message in the Dead Letter Queue (DLQ).

That is what happened in one case. What happened was that one of the services published a message with a payload a little bit bigger than 64MB and we had the default value from the rabbitmq-java-client (which comes from Spring AMQP) library which is by default 64 MB.

Java code declaring maxInboundMessageBodySize with a default of 64 MiB in the RabbitMQ client

Default maximum consumer message size in the RabbitMQ Client ConnectionFactory.java class

When the consumer of the DLQ was trying to consume the message, it was throwing the following error message

Log entry with IllegalStateException stack trace saying the message body is too large

Stack trace with error

But despite adding x-delivery-limit to all service queues, we forgot to add it also to the DLQ, so when the message was not able to be delivered, it was returned to the broker and being tried again, in an infinite loop.

A Poisonous Message Can Kill the Connection

But that is not the only and worst problem. What happens is that when the channel is faced with a big message like this, based on the current implementation, the channel is to shut down and stops the connection from receiving work

Java handleFailure method in AMQConnection calling shutdown on both EOF and unexpected exceptions

Method handleFailure in RabbitMQ Client AMQConnection.java class

As described in the docs.

Java shutdown method in AMQConnection with Javadoc saying it terminates all attached channels

shutdown method on RabbitMQ AMQConnection.java class

So, besides being poisonous, it was also killing the connection affecting all other channels and consumers for other queues.

This shutdown of the connection caused multiple messages to be delivered more than one time as the connection was killed before the acknowledgment was delivered to the broker, so it tried to deliver all other messages again.

Add a Delivery Limit to the DLQ Too

That is why it is also important to have an x-delivery-limit for DLQ as they also can drop messages that have a larger body size than allowed by the consumers (or other unforeseen errors).

Happy coding!