Ankur Mhatre
e7e64a047b
Kafka, RabbitMQ and Pulsar measured side by side on ordering, replay and consumer scaling by driving the three client libraries directly, plus an operational-footprint measurement of each broker's own distribution. Nine tests, three brokers started without Docker, and every number in the documentation regenerated by scripts/run-all.sh.
2.5 KiB
1. Three storage models, and everything that follows from them
Almost every difference between these brokers follows from one sentence about where a message lives.
- Kafka is an append-only log, partitioned. The broker keeps every record for the retention period and remembers nothing about individual consumers except an offset. Reading does not remove anything.
- RabbitMQ is a router with queues. The broker owns each message until a consumer acknowledges it, and then deletes it. It can route, expire, and dead-letter on its own.
- Pulsar is a log too, but the cursor and the storage are separate services: brokers are stateless and BookKeeper holds the data. Acknowledgement moves a cursor, and a message is deleted once every subscription has passed it — unless a retention policy says otherwise.
Read those three sentences again before reading any comparison table, including the one in chapter 7. They predict most of it.
| Kafka | RabbitMQ | Pulsar | |
|---|---|---|---|
| unit of parallelism | partition | queue | subscription |
| set when | topic is created | queue is declared | consumer subscribes |
| consuming | moves an offset | deletes the message | moves a cursor |
| a second reader | new consumer group | another queue, bound in advance | new subscription |
The last row is the one people underestimate. In Kafka and Pulsar you can add a reader that sees history you have already processed. In RabbitMQ you cannot add one after the fact at all — the messages are gone — so the decision to have a second consumer has to be made before the messages arrive.
What was measured, and how
Three questions, asked identically of all three brokers by driving the client libraries directly rather than through three different Spring abstractions:
| Question | Chapter |
|---|---|
| What ordering survives when you add a second consumer? | 02 |
| Can you read the same message twice? | 03 |
| How far do consumers scale, and what stops them? | 04 |
| What does each one cost to run? | 05 |
Versions are the ones a Spring Boot 4.1.1 application gets: kafka-clients 4.2.1, amqp-client
5.30.0, pulsar-client 4.2.4, all read from spring-boot-dependencies-4.1.1.pom. The brokers are
Kafka 4.2.1, RabbitMQ 3.10.25 and Pulsar 4.2.4, each started from its own distribution by a script
in scripts/.
next: Ordering