Ankur Mhatre
e1f8aa7402
Kafka, RabbitMQ and Pulsar measured side by side on ordering, replay and consumer scaling by driving the three client libraries directly, plus an operational-footprint measurement of each broker's own distribution. Nine tests, three brokers started without Docker, and every number in the documentation regenerated by scripts/run-all.sh.
81 lines
4.2 KiB
Markdown
81 lines
4.2 KiB
Markdown
# broker-comparison
|
|
|
|
Companion project for **Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with
|
|
Benchmarks** on [ankurm.com](https://ankurm.com).
|
|
|
|
Three brokers, three questions, one set of measurements. Ordering, replay and consumer scaling are
|
|
asked identically of all three by driving the client libraries directly, so that what is compared
|
|
is the brokers' delivery models rather than three sets of Spring defaults. What each Spring
|
|
integration adds on top is [chapter 6](docs/06-what-spring-adds.md).
|
|
|
|
## Verified stack
|
|
|
|
| Component | Version | Source of the number |
|
|
|---|---|---|
|
|
| JDK (tests) | 25.0.4.1+1 (Temurin) | `java -version` |
|
|
| JDK (brokers) | 21.0.12.1+1 (Temurin) | `java -version` |
|
|
| Spring Boot | 4.1.1 | `maven-metadata.xml` on Maven Central |
|
|
| kafka-clients | 4.2.1 | `spring-boot-dependencies-4.1.1.pom` |
|
|
| amqp-client | 5.30.0 | `spring-boot-dependencies-4.1.1.pom` |
|
|
| pulsar-client | 4.2.4 | `spring-boot-dependencies-4.1.1.pom` |
|
|
| Kafka broker | 4.2.1 (KRaft) | `kafka_2.13-4.2.1.tgz` |
|
|
| RabbitMQ broker | 3.10.25 | generic-unix tarball |
|
|
| Pulsar broker | 4.2.4 standalone | `apache-pulsar-4.2.4-bin.tar.gz` |
|
|
|
|
RabbitMQ is pinned at 3.10.25 because the box that produced these transcripts has Erlang 24;
|
|
3.11 and later need Erlang 25. The AMQP 0-9-1 semantics measured here are unchanged in 4.x.
|
|
|
|
## Quickstart
|
|
|
|
The Kafka measurements need no external broker — `spring-kafka-test` starts a real KRaft broker
|
|
in-process. The other two need a broker each, and the scripts start both without Docker and
|
|
without root.
|
|
|
|
```bash
|
|
mvn -Dgroups=kafka test
|
|
|
|
ERL_ROOT=... RABBITMQ_HOME=... scripts/rabbit-broker.sh
|
|
mvn -Dgroups=rabbit test
|
|
|
|
PULSAR_HOME=... scripts/pulsar-broker.sh
|
|
mvn -Dgroups=pulsar test
|
|
|
|
scripts/footprint.sh kafka|rabbit|pulsar # the operational numbers
|
|
scripts/run-all.sh # regenerates every docs/output/ file
|
|
```
|
|
|
|
## Documentation
|
|
|
|
| Chapter | What it settles |
|
|
|---|---|
|
|
| [01 Three models](docs/01-three-models.md) | Where a message lives, and why everything else follows |
|
|
| [02 Ordering](docs/02-ordering.md) | What survives the second consumer, in all three |
|
|
| [03 Replay](docs/03-replay.md) | Reading it twice, and the one broker that cannot |
|
|
| [04 Consumer scaling](docs/04-consumer-scaling.md) | The partition ceiling, and the client buffer that defeats fan-out in the other two |
|
|
| [05 Operational footprint](docs/05-operational-footprint.md) | Startup, memory, ports, processes, configuration surface |
|
|
| [06 What Spring adds](docs/06-what-spring-adds.md) | The three integrations, and the Boot 4 starter trap |
|
|
| [07 The decision table](docs/07-the-decision-table.md) | Every row backed by a measurement, and how to choose |
|
|
|
|
## Captured output
|
|
|
|
| File | What it shows |
|
|
|---|---|
|
|
| [`kafka-ordering.txt`](docs/output/kafka-ordering.txt) | Per-key order kept, global order not — and why A, B and C were the wrong keys to demonstrate it |
|
|
| [`kafka-replay.txt`](docs/output/kafka-replay.txt) | The same 12 records read three times |
|
|
| [`kafka-consumer-scaling.txt`](docs/output/kafka-consumer-scaling.txt) | 5 consumers, 3 partitions, 2 idle |
|
|
| [`rabbit-ordering.txt`](docs/output/rabbit-ordering.txt) | Message 1 finishing tenth |
|
|
| [`rabbit-replay.txt`](docs/output/rabbit-replay.txt) | 12 messages, then 0, then an empty queue |
|
|
| [`rabbit-consumer-scaling.txt`](docs/output/rabbit-consumer-scaling.txt) | 40/0/0/0/0 without `basicQos`, 8 each with it |
|
|
| [`pulsar-ordering.txt`](docs/output/pulsar-ordering.txt) | `Shared` spreading a key, `Key_Shared` pinning it |
|
|
| [`pulsar-replay.txt`](docs/output/pulsar-replay.txt) | `seek(earliest)` and a new subscription |
|
|
| [`pulsar-consumer-scaling.txt`](docs/output/pulsar-consumer-scaling.txt) | The 1000-message receiver queue defeating fan-out |
|
|
| [`footprint.txt`](docs/output/footprint.txt) | Startup, memory, ports and config surface for all three |
|
|
| [`tests.txt`](docs/output/tests.txt) | The nine tests behind all of the above |
|
|
|
|
## What is not measured here
|
|
|
|
Throughput. A messages-per-second number from one 2-core container says nothing about any of
|
|
these brokers, and publishing one would be worse than publishing nothing. The measurements above
|
|
are structural: they hold on any hardware, because they are properties of the delivery models
|
|
rather than of the machine.
|