I joined another team temporarily to help rewrite one of our services, and the team was deciding whether to use Kafka to replace RabbitMQ to have a better consumer load distribution. As I had never worked with Kafka before, I decided to pick up a book to understand the details and help with the decision.
I have very good experience with RabbitMQ, especially with quorum queues, as I have read a few books about it (like RabbitMQ in Depth) and have been maintaining our RabbitMQ for a while now. I have never used RabbitMQ streams, but I read about them to understand if they suited our use case. While they are good, one of our concerns was the distribution of the partitions between the consumers in the case of super streams.
Looking for a resource to read about Kafka, I came across this book. One of my concerns was that the book is around five years old, so I was worried it would be very outdated: it talks about version 2.8, and now we have 3.x and even 4.x available. But I wanted to focus on the fundamentals, how Kafka works at its core, and not on the latest features. I was especially interested in understanding how the partitions and the consumers would work, so that was my main goal.
I was not disappointed with the book. It gave me a lot of the fundamental understanding I was looking for.
This post of Learning is from a technical book, so it may be a bit different than the other posts of the collection Learnings from books, as when I read technical books I focus more on the parts that I looking to learn and for my current situation.
Learnings from the book Kafka: The Definitive Guide

Book Kafka: The Definitive Guide
Publishing to a topic instead of routing to queues
One of the main differences, at least for our use case, is how messages get to the consumers. In RabbitMQ, we publish a message to an exchange, and as we use a topic exchange, it routes the message by routing key to all the queues bound with that routing key. So more than one service can have its own queue bound to the same routing key, and publishing to one exchange delivers the message to the queues of all those services. In a way, each service subscribes to the routing keys it wants and listens to one queue, or to more if it wants to break the work down and process it in parallel.
In Kafka, we publish to a topic, and the consumer has to listen to that topic to get the messages. Each service uses its own consumer group, so every service gets all the messages of the topic, and there is no routing key to filter them on the broker.
The other big difference from the RabbitMQ queues we use is that reading a message does not delete it (RabbitMQ streams work the same way). Kafka keeps the messages for a retention period (7 days by default) or until the partition reaches a size limit, and each consumer group keeps track of its own position (the offset) in each partition. As I learned in Designing Data-Intensive Applications, this means we can replay messages, something that is really hard to do with RabbitMQ classic and quorum queues once a message is acknowledged.
The message key matters too. The producer hashes the key to pick the partition, so all messages with the same key go to the same partition, and Kafka only guarantees the order inside a partition, not across the whole topic.
Consumer groups distribute the partitions
The reason we chose Kafka for this service is the load distribution. RabbitMQ streams can also be partitioned with super streams, but in our case the load was not well distributed between the consumers. In Kafka, a topic is split into partitions, and inside a consumer group each partition is consumed by only one consumer, which is similar to the single active consumer in RabbitMQ. When a consumer joins or leaves the group, the partitions are rebalanced between the consumers, so the load stays distributed. That also means there is no point in having more consumers than partitions, as the extra consumers will just be idle.
How the rebalance happens depends on the assignment strategy. In an eager rebalance, all the consumers stop, give up all their partitions and get a brand-new assignment, which is a short window where the whole group is not consuming. In a cooperative rebalance (the Cooperative Sticky assignor), only a small subset of the partitions moves, and the consumers keep processing the partitions that were not reassigned.
The number of partitions needs some thought upfront. It can only be increased, never decreased, and as the key decides the partition, adding partitions later changes where the keys go. The book’s advice is to calculate it from the expected future throughput, not the current one, and to start small rather than too large, as each partition also uses resources on the broker.
Reliability on the producer, the broker and the consumer
One of my concerns is always reliability: how we scale while keeping replicas of the data, and how we make sure we have a stable system that doesn’t lose any messages. As our usage is for a financial system, this matters even more.
The book gave me a very strong base on how to get reliability on the producer side, on the broker side with replication, and on the consumer side. And it makes clear that you need all three: “Reliability is a property of a system—not of a single component”. Kafka is flexible enough to support everything from tracking website clicks to credit card payments, which also makes it easy to believe the system is reliable when it is not.
Kafka gives a few basic guarantees: order inside a partition, a message is committed once it is written to all the in-sync replicas, committed messages are not lost as long as one replica is alive, and consumers can only read committed messages. Everything else depends on how we configure the producer, the broker and the consumer.
Committing offsets on the consumer
One thing I will for sure pay attention to when we do the implementation is how the consumer acknowledges the messages. As far as I understood, we consume messages from Kafka in batches and in order, and the offset can be committed at the end of the batch or in between, depending on the configuration.
Before committing an offset, we have to make sure all the previous messages were processed successfully, because the broker assumes everything before a committed offset was consumed correctly. Otherwise, a message could fail with an exception and be lost.
The book has a good example: if we fail to process record #30 and succeed with record #31, we should not commit offset #31, as that marks #30 as processed too. Instead, we commit the last record processed successfully, keep the remaining records in a buffer, call pause() on the consumer so new polls don’t return more data, and keep retrying.
By default, the consumer commits automatically (enable.auto.commit=true), every five seconds, the latest offset returned by the last poll. That is safe as long as all the processing happens inside the poll loop, but we have no control over how many messages get processed twice if the consumer stops before the next commit. If the processing happens in another thread, automatic commit is not an option, as it could commit messages that were read but not processed yet.
Another setting to choose carefully is auto.offset.reset, which decides where the consumer starts when it has no valid committed offset. earliest reads the partition from the beginning, which can process a lot of messages twice but minimizes data loss. latest (the default) starts at the end and almost certainly misses some messages. For a financial system, I would rather deal with duplicates.
Acknowledgements on the producer
On the producer side, I also have to be mindful of the acknowledgement process. In RabbitMQ, we have an asynchronous mechanism called publisher confirms, which tells us when the broker has taken responsibility for a message, and for a quorum queue that means a majority of its replicas have it.
In Kafka, the similar mechanism is acks=all together with the minimum number of in-sync replicas (min.insync.replicas). We have to tune the producer and the broker to always replicate to more than one node, to make sure we don’t lose a message if one node fails.
This matters more than I expected because Kafka does not wait for the message to be written to disk: it relies on replication for durability. The book shows how a reliable broker is not enough on its own. With three replicas and the producer using acks=1, the leader can confirm the message and crash before replicating it, a follower becomes the new leader, and the message is gone while the producer thinks it was written. With acks=all and min.insync.replicas=2, at least two replicas have the message before the producer gets the confirmation. If only one replica is in sync, producers using acks=all get a NotEnoughReplicasException and the partition is effectively read-only, but nothing is silently lost.
A replica is in sync when it has caught up with the leader within replica.lag.time.max.ms (the book says 10 seconds, but the default has been 30 seconds since Kafka 2.5), and only in-sync replicas can become the leader. The exception is an unclean leader election, which elects an out-of-sync replica to bring the partition back and loses every message it did not have.
Retries can create duplicates, so the producer should also enable idempotence (enable.idempotence=true). This is one place where the book is outdated in a good way: since Kafka 3.0, the producer defaults to acks=all and idempotence on. It adds a producer id and a sequence number to each message, and the broker rejects the duplicates caused by the producer’s own retries. It does not catch the application calling send() twice with the same message, so it is better to rely on the producer’s built-in retries than to catch the exception and retry in our code.
Performance also depends on how we send. The producer is always asynchronous: send() returns a Future, and waiting on it for every message (a synchronous send) means the thread sits idle for each round trip, which is why it is common in examples but rare in production. Sending with a callback keeps the throughput and still lets us handle the errors. Messages are sent in batches per partition, and enabling compression (snappy, lz4, gzip or zstd) on the producer makes the batches smaller on the network and on the broker disks.
Exactly-once stops at Kafka
Kafka has exactly-once semantics, but they only apply to messages written to Kafka, in the consume-process-produce pattern of stream processing. If the processing calls a REST API, sends an email or writes to a database, exactly-once does not cover it. For us, that means the consumers still have to be idempotent, and the book points to the outbox pattern when we need the database and Kafka to stay in sync.
Other things I learned
- Retention works on log segments, not individual messages. Each partition is split into segments of 1 GB or a week by default, and only closed segments can be deleted.
- Compacted topics keep only the latest message for each key. To delete a key, the producer sends a message with a null value (a tombstone).
- The broker limits the message size (
message.max.bytes, 1 MB by default), measured after compression. - Kafka Connect is the choice to move data from or to systems whose code we do not control. For our own applications, the producer and consumer clients are the way to go.
- Testing failure scenarios like a broker losing connectivity, a full or hanging disk, a leader election, and rolling restarts of the brokers, consumers and producers shows whether the system actually behaves as planned.
- Event streams are ordered, immutable and replayable. A canceled transaction does not disappear: a new event records the cancelation. That is what makes Kafka a good fit for stream processing.
Favorite quotes
“One of the problems in software engineering is that there is always more than one way to do anything. Platforms such as Apache Kafka provide plenty of flexibility, which is great for experts but makes for a steep learning curve for beginners.”
“Because of its flexibility, it is also easy to accidentally shoot ourselves in the foot when using Kafka—believing that our system is reliable when in fact it is not.”
“Exactly-once semantics in Kafka is the opposite of chess: it is challenging to understand but easy to use.”
“It is currently not possible to perform cluster failover in Kafka without either losing data or having duplicate events. Often both.”
There is for sure a lot more in the book: consumers, producers, how to install, maintain and monitor Kafka, in a way everything you need to run it in production. It is going to be very useful for me as we put this new service in production and start supporting it, understanding the edge cases and the drawbacks of this technology, as well as its positive side.
Once we have the service running in production, I can write a bit more about how the experience was and what the learnings were.
Liked this post? Check out other posts part of the series Learnings from books where my goal is to share what I learned from the book that I read. It is a mixture of review and summary with a bit of my opinion and point of view