asmhatre/spring-messaging-demo
1
0
Fork 0
Code Activity

Add the broker-comparison module

Browse Source 
Kafka, RabbitMQ and Pulsar measured side by side on ordering, replay and
consumer scaling by driving the three client libraries directly, plus an
operational-footprint measurement of each broker's own distribution.

Nine tests, three brokers started without Docker, and every number in the
documentation regenerated by scripts/run-all.sh.
This commit is contained in:
Ankur Mhatre
2026-09-01 23:27:08 +05:30
parent f4b0112e2c
commit e7e64a047b
30 changed files with 1836 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
broker-comparison/docs/01-three-models.md Normal file
View File

@@ -0,0 +1,49 @@
[README](../README.md) · next: [Ordering](02-ordering.md)
# 1. Three storage models, and everything that follows from them
Almost every difference between these brokers follows from one sentence about where a message
lives.
- **Kafka** is an append-only log, partitioned. The broker keeps every record for the retention
period and remembers nothing about individual consumers except an offset. Reading does not
remove anything.
- **RabbitMQ** is a router with queues. The broker owns each message until a consumer
acknowledges it, and then deletes it. It can route, expire, and dead-letter on its own.
- **Pulsar** is a log too, but the cursor and the storage are separate services: brokers are
stateless and BookKeeper holds the data. Acknowledgement moves a cursor, and a message is
deleted once every subscription has passed it — unless a retention policy says otherwise.
Read those three sentences again before reading any comparison table, including the one in
[chapter 7](07-the-decision-table.md). They predict most of it.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| unit of parallelism | partition | queue | subscription |
| set when | topic is created | queue is declared | consumer subscribes |
| consuming | moves an offset | deletes the message | moves a cursor |
| a second reader | new consumer group | another queue, bound in advance | new subscription |
The last row is the one people underestimate. In Kafka and Pulsar you can add a reader that sees
history you have already processed. In RabbitMQ you cannot add one after the fact at all — the
messages are gone — so the decision to have a second consumer has to be made *before* the
messages arrive.
## What was measured, and how
Three questions, asked identically of all three brokers by driving the client libraries directly
rather than through three different Spring abstractions:
| Question | Chapter |
|---|---|
| What ordering survives when you add a second consumer? | [02](02-ordering.md) |
| Can you read the same message twice? | [03](03-replay.md) |
| How far do consumers scale, and what stops them? | [04](04-consumer-scaling.md) |
| What does each one cost to run? | [05](05-operational-footprint.md) |
Versions are the ones a Spring Boot 4.1.1 application gets: kafka-clients 4.2.1, amqp-client
5.30.0, pulsar-client 4.2.4, all read from `spring-boot-dependencies-4.1.1.pom`. The brokers are
Kafka 4.2.1, RabbitMQ 3.10.25 and Pulsar 4.2.4, each started from its own distribution by a script
in `scripts/`.
next: [Ordering](02-ordering.md)

66
broker-comparison/docs/02-ordering.md Normal file
View File

@@ -0,0 +1,66 @@
prev: [Three models](01-three-models.md) · [README](../README.md) · next: [Replay](03-replay.md)
# 2. Ordering
Every broker here is FIFO with one producer and one consumer. The question that matters is what
survives the second consumer, because that is the first thing you add.
## Kafka: ordered within a partition, and a key picks the partition
Twelve records, three keys, three partitions
([`docs/output/kafka-ordering.txt`](output/kafka-ordering.txt)):
```
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
```
Each key's values come back in the order they were produced. The global sequence does not: the
consumer drains one partition's buffer before moving to the next, so the delivered order is
2, 5, 8, 11, 1, 4, 7, 10, 3, 6, 9, 12.
That is the whole Kafka ordering guarantee, and it is usually enough, because "in order" nearly
always means "in order per customer / per account / per device" rather than globally.
**A detail worth stealing.** The keys in that transcript are D, A and F, not A, B and C. With
three partitions, murmur2 sends A, B *and* C all to partition 1. Picking three obvious keys to
demonstrate partitioning would have produced a transcript in which global order was accidentally
preserved. Before you conclude that your keys spread evenly, compute
`Utils.toPositive(Utils.murmur2(key)) % partitions` for the ones you actually use.
## RabbitMQ: FIFO per queue, and no key at all
One consumer sees the queue in order. Two consumers do not
([`docs/output/rabbit-ordering.txt`](output/rabbit-ordering.txt)):
```
one queue, one consumer : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
```
Message 1 went to the slower consumer and finished tenth. There is no key and no partition, so
there is nothing to pin related messages to one consumer. Ordering across related messages in
RabbitMQ means one queue and one consumer — which means no horizontal scaling of that queue —
or the consistent-hash exchange, which is a plugin and effectively reintroduces partitions by
hand.
## Pulsar: the subscription type decides, per subscriber
Two consumers on a `Shared` subscription and two on `Key_Shared`, same topic, same messages
([`docs/output/pulsar-ordering.txt`](output/pulsar-ordering.txt)):
```
Shared A -> [consumer-1, consumer-2] B -> [consumer-1, consumer-2] C -> [consumer-1, consumer-2]
Key_Shared A -> [consumer-2] B -> [consumer-1] C -> [consumer-2]
```
Under `Shared`, every key was handled by both consumers, so two messages with the same key can be
in flight at once and no ordering claim survives. Under `Key_Shared`, each key went to exactly one
consumer — Kafka's guarantee, but chosen by the subscriber rather than fixed by a partition count
someone picked when the topic was created, and recomputed as consumers join and leave.
That last property is the strongest single argument for Pulsar: two teams reading the same topic
can make different ordering-versus-throughput trades without negotiating.
next: [Replay](03-replay.md)

59
broker-comparison/docs/03-replay.md Normal file
View File

@@ -0,0 +1,59 @@
prev: [Ordering](02-ordering.md) · [README](../README.md) · next: [Consumer scaling](04-consumer-scaling.md)
# 3. Replay
"Can I read that again?" is asked after a bad deploy, and by then it is too late to change
brokers.
## Kafka
```
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
```
([`docs/output/kafka-replay.txt`](output/kafka-replay.txt)) Consuming moves an offset and removes
nothing. A new group, a reset offset and a `seekToBeginning` all read the same records, for as
long as retention keeps them.
## RabbitMQ
```
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
```
([`docs/output/rabbit-replay.txt`](output/rabbit-replay.txt)) Acknowledging deletes. There is no
offset to rewind and no second reader that can see what the first consumed. Reading a message
twice has to be arranged in advance — a second queue bound to the same exchange, or a copy written
somewhere else — and it cannot be arranged afterwards.
This is not a defect. A router that deletes what it has delivered is much cheaper to operate than
a log, and most work queues genuinely do not need history. It only becomes a defect at the moment
you need history and do not have it.
RabbitMQ streams (3.9+) are a separate, log-shaped feature that does support replay. They are a
different thing living in the same broker, with their own client protocol and their own semantics
— worth knowing about, and not what you get from `queueDeclare`.
## Pulsar
```
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
```
([`docs/output/pulsar-replay.txt`](output/pulsar-replay.txt)) `seek(MessageId)` and
`seek(timestamp)` rewind a live subscription; a new subscription starting from `Earliest` reads
everything still stored.
The difference from Kafka is the **default**, and it is the one that surprises people: Kafka keeps
a record for the retention period regardless of who read it, while Pulsar deletes a message once
every subscription has acknowledged it, unless a retention policy on the namespace says otherwise.
A Pulsar namespace with the default retention and one well-behaved subscription keeps nothing —
so the replay you are counting on requires a policy you have to set.
next: [Consumer scaling](04-consumer-scaling.md)

80
broker-comparison/docs/04-consumer-scaling.md Normal file
View File

@@ -0,0 +1,80 @@
prev: [Replay](03-replay.md) · [README](../README.md) · next: [Operational footprint](05-operational-footprint.md)
# 4. Consumer scaling
The question is what happens when you add the fifth consumer.
## Kafka: partitions are a hard ceiling
Five consumers in one group on a three-partition topic
([`docs/output/kafka-consumer-scaling.txt`](output/kafka-consumer-scaling.txt)):
```
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
```
A partition belongs to at most one consumer in a group, so partition count is the ceiling. Two of
the five processes are running, connected, healthy, and doing nothing at all — and they will keep
reporting healthy forever.
Raising the partition count later is possible and is not free: it changes which partition a key
maps to, so the per-key ordering guarantee is broken across the change for every key that moves.
In practice the partition count is a capacity decision made at design time, on incomplete
information, that you then live with.
## RabbitMQ: no ceiling, but prefetch decides whether it is real
Five consumers on one queue, forty messages
([`docs/output/rabbit-consumer-scaling.txt`](output/rabbit-consumer-scaling.txt)):
```
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 consumer-2..5 -> 0
basicQos(1):
consumer-1..5 -> 8 each
```
There is no structural ceiling — but with the AMQP default the broker pushes as many messages as
a consumer will accept, so the first consumer to connect can be handed the entire backlog while
four idle processes wait. `basicQos` is not a tuning knob to postpone; it is what makes the
fan-out exist.
Spring AMQP sets it for you: `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` is 250.
Better than unlimited, and still enough to concentrate any backlog smaller than 250 messages on
one consumer.
## Pulsar: no ceiling either, and the same trap under a different name
Five consumers, one *non-partitioned* topic, forty messages
([`docs/output/pulsar-consumer-scaling.txt`](output/pulsar-consumer-scaling.txt)):
```
receiverQueueSize default (1000):
one or two consumers take everything; the rest receive nothing
receiverQueueSize(1):
consumer-1..5 -> 8 each
```
The topic has no partitions and five consumers still share the work, which in Kafka would require
at least five partitions decided in advance. But the default receiver queue is a thousand
messages, so the first consumers to connect pull the whole backlog into their own buffers first.
Same failure as RabbitMQ's unlimited prefetch, four times the default.
## The shape of the answer
- Kafka gives you ordering per key and pays for it with a fixed parallelism ceiling.
- RabbitMQ gives you unlimited competing consumers and pays for it with no ordering across them.
- Pulsar lets each subscription choose, which is genuinely the best of both — at the operational
cost measured in [the next chapter](05-operational-footprint.md).
And all three brokers have a client-side buffer that will quietly defeat your fan-out if you leave
it at the default and your backlog is small. That is the one finding here that applies to
whichever you pick.
next: [Operational footprint](05-operational-footprint.md)

55
broker-comparison/docs/05-operational-footprint.md Normal file
View File

@@ -0,0 +1,55 @@
prev: [Consumer scaling](04-consumer-scaling.md) · [README](../README.md) · next: [What Spring adds](06-what-spring-adds.md)
# 5. Operational footprint
"Ops burden" cannot be benchmarked. The number of moving parts, the memory floor and the size of
the configuration surface can be, and they are what people are actually asking about.
Each broker started from its shipped distribution with default configuration and a 700 MB heap
cap, on one 2-core / 3.8 GB Linux box, Temurin JDK 21
([`docs/output/footprint.txt`](output/footprint.txt)):
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 standalone |
|---|---|---|---|
| launch to first accepted connection | 10.0 s | 15.1 s | 20.8 s |
| resident memory at idle | 333 MB | 115 MB | 610 MB |
| server processes | 1 | 1 (+ `epmd`) | 1 |
| listening ports | 9092, 9093 | 5672, 4369, 25672 | 6650, 8080, 2181 |
| unpacked distribution | 135 MB | 26 MB | 344 MB |
| settings in the shipped default config | 24 | **0** | 357 |
Read those numbers as orders of magnitude, not as a benchmark: one sample, one small machine.
Three things they say clearly.
**RabbitMQ ships no configuration file at all.** The boot log records `Config file(s): (none)`;
`rabbitmq.conf.example` is entirely commented out. Everything works out of the box, and the
smallest resident footprint here is the one that is not a JVM. Against that, `epmd` on 4369 and
the inter-node port on 25672 are the Erlang distribution, which is also how clustering works and
how clustering goes wrong.
**Pulsar standalone is three systems in one process.** Port 2181 is ZooKeeper and the on-disk
data is BookKeeper's; the standalone distribution hides that behind one command. A real
deployment does not: you operate ZooKeeper (or, from Pulsar 3.x, an alternative metadata store),
BookKeeper bookies and Pulsar brokers as three tiers with three scaling stories. The 357-setting
`standalone.conf` is the honest signal here — that is the configuration surface, and running it
seriously means learning most of it.
**Kafka since KRaft is genuinely simpler than it was.** One process, two ports, 24 settings, no
ZooKeeper. The old "Kafka means also running ZooKeeper" objection is a version behind; anyone
comparing on that basis is comparing to 2022.
## What the numbers do not include
The footprint above is one node at idle. The thing that actually determines ops burden is what
happens at three in the morning:
- **Kafka**: rebalances, consumer lag as the primary signal, partition-count decisions you cannot
cleanly reverse, and a broad ecosystem of tools that assume Kafka.
- **RabbitMQ**: the management plugin is genuinely good, queue depth is a direct and obvious
signal, and the hard problem is network partitions in a cluster — which the Erlang distribution
makes fast to detect and awkward to resolve.
- **Pulsar**: the fewest people on your team will have run it. That is not a technical property
and it is usually the deciding one.
next: [What Spring adds](06-what-spring-adds.md)

52
broker-comparison/docs/06-what-spring-adds.md Normal file
View File

@@ -0,0 +1,52 @@
prev: [Operational footprint](05-operational-footprint.md) · [README](../README.md) · next: [The decision table](07-the-decision-table.md)
# 6. What Spring adds, and what Boot 4 no longer gives you for free
The measurements in this module drive the client libraries directly, so that what is being
compared is the brokers rather than three sets of Spring defaults. In an application you would use
the Spring integrations, and they are not equivalent to one another.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| project | Spring for Apache Kafka 4.1.1 | Spring AMQP 4.1.1 | Spring for Apache Pulsar 2.0.7 |
| starter | `spring-boot-starter-kafka` | `spring-boot-starter-amqp` | `spring-boot-starter-pulsar` |
| listener | `@KafkaListener` | `@RabbitListener` | `@PulsarListener` |
| template | `KafkaTemplate` | `RabbitTemplate` | `PulsarTemplate` |
| retry / DLQ | `DefaultErrorHandler`, `@RetryableTopic` | dead-letter exchange, container error handler | `DeadLetterPolicy` on the listener |
All three are Boot-managed at 4.1.1, so you do not pin their versions.
## The Boot 4 trap that applies to all three
**In Spring Boot 4, depending on a messaging library directly rather than through its Boot starter
means you have no auto-configuration.** The auto-configuration classes moved out of
`spring-boot-autoconfigure` into per-technology modules that only the starters bring:
| depending on | what you lose | how it presents |
|---|---|---|
| `org.springframework.kafka:spring-kafka` | `spring-boot-kafka` | `No qualifying bean of type KafkaTemplate<...>` — the context starts fine |
| `org.springframework.amqp:spring-rabbit` | `spring-boot-amqp` | no `RabbitTemplate`, no `RabbitAdmin` |
| `org.springframework.pulsar:spring-pulsar` | `spring-boot-pulsar` | no `PulsarTemplate` |
Every Boot 3 tutorial gets this wrong now, and the symptom is a missing bean rather than anything
that names the cause. Use the starters.
## The Jackson fork, in both Kafka and RabbitMQ
Boot 4 moved to Jackson 3 (`tools.jackson`), and both messaging projects ship converters for
both generations. **A `2` in the class name means the previous Jackson** — which is the opposite
of the convention you would guess:
- Spring Kafka: `JsonSerializer`/`JsonDeserializer` are Jackson 2; `JacksonJsonSerializer`,
`JacksonJsonDeserializer` and `JacksonJsonSerde` are Jackson 3.
- Spring AMQP: `Jackson2JsonMessageConverter` is Jackson 2; `JacksonJsonMessageConverter` is
Jackson 3.
Picking the wrong one gives you `SerializationException: Can't serialize data`, whose cause is
`Java 8 date/time type java.time.Instant not supported by default` — a message about date types
for what is really a wrong-library problem.
The [kafka-basics](../kafka-basics/README.md) and [rabbitmq](../rabbitmq/README.md) modules in
this repository cover both stacks in detail.
next: [The decision table](07-the-decision-table.md)

59
broker-comparison/docs/07-the-decision-table.md Normal file
View File

@@ -0,0 +1,59 @@
prev: [What Spring adds](06-what-spring-adds.md) · [README](../README.md)
# 7. The decision table
Every row below is either a measurement in `docs/output/` or a statement about operating the
thing. Nothing here is a vendor claim.
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 |
|---|---|---|---|
| ordering with one consumer | FIFO per partition | FIFO per queue | FIFO |
| ordering with many consumers | per key, always | **none** | per key with `Key_Shared`, none with `Shared` |
| who chooses that | topic design | plugin, or one consumer | each subscription, independently |
| replay after the fact | yes, within retention | **no** | yes, if retention is configured |
| second independent reader | new consumer group, any time | must be arranged in advance | new subscription, any time |
| consumer parallelism ceiling | partition count | none | none |
| changing that ceiling | repartition; breaks key→partition | nothing to change | nothing to change |
| the client-side trap | none by default | unlimited prefetch (Spring: 250) | `receiverQueueSize` 1000 |
| memory at idle | 333 MB | **115 MB** | 610 MB |
| processes to operate | 1 (KRaft) | 1 + `epmd` | broker + BookKeeper + metadata store |
| shipped default settings | 24 | **0** | 357 |
| Spring project maturity | very high | very high | good, much smaller community |
## Choosing
**Choose RabbitMQ** when the work is a queue of tasks: each message is an instruction, order
between instructions does not matter, and once it is done it is done. It is the smallest thing
that works, it has the best out-of-the-box operability of the three, and queue depth is a metric
anyone can interpret. Choose it in the knowledge that you are giving up replay permanently —
adding it later is a broker migration, not a configuration change.
**Choose Kafka** when the messages are events other people will want to read: when more than one
consumer will exist, when someone will need to reprocess history after a bug, or when ordering per
entity is part of the contract. Pay for it with a partition count decided too early, consumers
that scale only as far as that number, and an operational model where lag is the thing you watch.
KRaft has removed the old ZooKeeper objection.
**Choose Pulsar** when you genuinely need what neither of the others gives you: per-subscription
choice of ordering versus fan-out, consumer counts not bounded by a number chosen at topic
creation, or multi-tenancy with real isolation. It is the most capable design here. It is also
three systems, 357 settings, five times RabbitMQ's memory floor, and the one your team has least
experience with — and that last point decides more incidents than the first three prevent.
**If the honest answer is "we do not know yet"**, that argues for Kafka, on grounds that have
nothing to do with the technology: you can hire for it, your monitoring vendor supports it, and
the failure modes are documented by thousands of people who hit them first. That is a real
engineering argument and it is fine to make it out loud.
## The question that dissolves the choice
A surprising number of these decisions are made for a system that has one producer, one consumer
and fewer than a hundred messages a second. At that volume all three brokers work, none of the
measurements above will ever be reached, and the decision is entirely about what your team can
operate at three in the morning.
The measurements matter when you can name which row you are relying on. If you cannot, you are
picking an operational burden, not a broker — and the cheapest one to operate is the one in the
`115 MB` cell.
[README](../README.md)

40
broker-comparison/docs/output/footprint.txt Normal file
View File

@@ -0,0 +1,40 @@
== Operational footprint, measured on one 2-core / 3.8 GB Linux box ==
Each broker started from its shipped distribution with default configuration and a
700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as
orders of magnitude, not as a benchmark.
broker : kafka
time from launch to first
accepted connection : 10035 ms
resident memory at idle : 333 MB
server processes : 1
listening ports : 9092 (broker), 9093 (controller)
unpacked distribution size : 135 MB
settings in shipped default
configuration file : 24 (server.properties)
broker : rabbit
time from launch to first
accepted connection : 15053 ms
resident memory at idle : 115 MB
server processes : 1
listening ports : 5672 (AMQP), 4369 (epmd), 25672 (inter-node)
unpacked distribution size : 26 MB
settings in shipped default
configuration file : 0 (rabbitmq.conf.example)
epmd processes : 1
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
entirely commented out and the boot log records "Config file(s): (none)".
Everything in that file is documentation, not a default.
broker : pulsar
time from launch to first
accepted connection : 20801 ms
resident memory at idle : 610 MB
server processes : 1
listening ports : 6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)
unpacked distribution size : 344 MB
settings in shipped default
configuration file : 357 (standalone.conf)

16
broker-comparison/docs/output/kafka-consumer-scaling.txt Normal file
View File

@@ -0,0 +1,16 @@
== Kafka: partitions are the ceiling on consumer parallelism ==
topic 'orders-scaling', 3 partitions, 5 consumers in one group
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
consumers with no partitions: 2
A partition is assigned to at most one consumer in a group, so the number
of partitions is a hard ceiling on consumer parallelism. Adding consumers
beyond it adds idle processes, not throughput.

30
broker-comparison/docs/output/kafka-ordering.txt Normal file
View File

@@ -0,0 +1,30 @@
== Kafka: ordering is per partition ==
produced 12 records, keys D/A/F round-robin, values 1..12 in order
(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C
all hash to partition 1 with three partitions, which is worth knowing
before you decide your keys are well distributed.)
consumed in this order:
F=2@p2
F=5@p2
F=8@p2
F=11@p2
A=1@p1
A=4@p1
A=7@p1
A=10@p1
D=3@p0
D=6@p0
D=9@p0
D=12@p0
per key:
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
Order is preserved within each key because a key hashes to one partition.
Across keys it is not: the values above are not 1..12 in order, because a
consumer drains one partition's buffer before the next.

10
broker-comparison/docs/output/kafka-replay.txt Normal file
View File

@@ -0,0 +1,10 @@
== Kafka: the log is the storage ==
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
Consuming does not remove anything. A consumer group is a cursor over a log
that the broker keeps until retention expires, so a new group, a reset
offset or a seek all read the same records again.

30
broker-comparison/docs/output/pulsar-consumer-scaling.txt Normal file
View File

@@ -0,0 +1,30 @@
== Pulsar: no partition ceiling, but the receiver queue decides who gets the work ==
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
receiverQueueSize left at the default (1000):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
receiverQueueSize(1):
consumer-1 -> 8 messages
consumer-2 -> 8 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 8 messages
consumers that received nothing : 0
The topic has no partitions and five consumers can still share the work --
in Kafka the same shape needs at least five partitions, chosen when the
topic was created. But the default receiver queue is 1000 messages, so the
first consumers to connect pull the whole 40-message backlog into their own
buffers before the rest ask for anything, and the subscription looks
broken. Which consumers win is a race and varies between runs -- one
consumer taking all forty, or two taking twenty each -- but the consumers
that lose it see nothing at all. This is the same trap as RabbitMQ's
unbounded prefetch, with a different name and a much larger default.

24
broker-comparison/docs/output/pulsar-ordering.txt Normal file
View File

@@ -0,0 +1,24 @@
== Pulsar: the subscription type decides ==
12 messages, keys A/B/C, values 1..12 in order, two consumers per
subscription, receiverQueueSize 1 so the first consumer cannot take the
whole backlog.
Shared subscription, which consumers saw each key:
A -> [consumer-1, consumer-2]
B -> [consumer-1, consumer-2]
C -> [consumer-1, consumer-2]
Key_Shared subscription, which consumers saw each key:
A -> [consumer-1]
B -> [consumer-1]
C -> [consumer-1]
A Shared subscription round-robins individual messages, so messages with
the same key end up on different consumers and can be processed at the
same time: there is no per-key order left to speak of. Key_Shared hashes
the key to one consumer, which is Kafka's guarantee -- except that the
assignment belongs to the subscription and is recomputed as consumers come
and go, rather than being fixed by a partition count chosen when the topic
was created.

14
broker-comparison/docs/output/pulsar-replay.txt Normal file
View File

@@ -0,0 +1,14 @@
== Pulsar: acknowledged, but still there ==
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
Acknowledgement moves a cursor; the message itself lives in the managed
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
which Kafka can also do by resetting offsets. What differs is the default:
Pulsar deletes a message once every subscription has acknowledged it,
unless a retention policy on the namespace says otherwise, whereas Kafka
keeps it for the retention period regardless of who read it. A Pulsar
topic with no retention policy and no subscriptions keeps nothing.

34
broker-comparison/docs/output/rabbit-consumer-scaling.txt Normal file
View File

@@ -0,0 +1,34 @@
== RabbitMQ: consumers scale, and prefetch decides whether they actually do ==
one queue, 40 messages, 5 consumers, each taking 10 ms per message
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
basicQos(1):
consumer-1 -> 8 messages
consumer-2 -> 9 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 7 messages
consumers that received nothing : 0
Every consumer on a queue competes for the same messages, so adding
consumers adds throughput and there is no structural ceiling of the kind
Kafka's partition count imposes. But with the AMQP default the broker
pushes as many messages as a consumer will take, so whichever consumer
connects first can be handed the entire backlog while the others sit idle.
basicQos is not a tuning knob you get to postpone; it is what makes the
fan-out real. Spring AMQP sets it for you --
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
better than unlimited and still large enough to concentrate a small
backlog on one consumer.
The price of all this is the ordering measurement: there is no key, so
nothing constrains related messages to one consumer.

18
broker-comparison/docs/output/rabbit-ordering.txt Normal file
View File

@@ -0,0 +1,18 @@
== RabbitMQ: FIFO per queue, per consumer ==
one queue, one consumer, 12 messages
completion order : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
completion order : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
out-of-order steps: 1
A queue is FIFO and a single consumer sees it that way. The moment a second
consumer is added the broker hands the next message to whichever consumer
is free, so the order in which work finishes is no longer the order in
which it was published. There is no key: RabbitMQ has no notion of a
partition to which related messages could be pinned. Ordering across
related messages means one queue and one consumer, and therefore no
horizontal scaling for that queue -- or a consistent-hash exchange, which
is a plugin.

12
broker-comparison/docs/output/rabbit-replay.txt Normal file
View File

@@ -0,0 +1,12 @@
== RabbitMQ: there is nothing to replay ==
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
Acknowledging a message deletes it. The broker is a router with buffers,
not a log: there is no offset to rewind and no second reader that can see
what the first one consumed. Reading the same message twice means
arranging it in advance -- a second queue bound to the same exchange, or a
copy written somewhere else -- and it cannot be arranged after the fact.

12
broker-comparison/docs/output/tests.txt Normal file
View File

@@ -0,0 +1,12 @@
[INFO] Running com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.88 s -- in com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.732 s -- in com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.00 s -- in com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS