asmhatre/spring-messaging-demo
1
0
Fork 0
Code Activity

Compare commits

base: asmhatre/spring-messaging-demo:main
Branches Tags
asmhatre/spring-messaging-demo:main
..
compare: asmhatre/spring-messaging-demo:2919282a5c8da79a417c5089976522fd7cda3569
Branches Tags
asmhatre/spring-messaging-demo:main
2 Commits
main .. 2919282a5c
Author SHA1 Message Date
asmhatre 2919282a5c Add the kafka-error-handling module 2026-08-29 10:14:23 +05:30
asmhatre 8b5587cf88 Add the rabbitmq module 2026-08-29 09:58:34 +05:30
227 changed files with 17 additions and 9452 deletions
Expand all files Collapse all files
README.md
+10 -23
View File
@@ -1,34 +1,21 @@
# spring-messaging-demo # spring-messaging-demo
Companion code for the messaging series on [ankurm.com](https://ankurm.com). Each directory is a Companion code for the messaging series on [ankurm.com](https://ankurm.com). Each directory is a
self-contained Maven project for one article, with its own `pom.xml` and its own captured self-contained Maven project for one article, with its own `pom.xml`, its own numbered
output, regenerated from the test suite rather than typed by hand. Most modules keep that documentation chapters, and its own captured output under `docs/output/` — regenerated by that
output under `docs/output/` alongside numbered documentation chapters; `saga/` and `share-groups/` keep it module's `scripts/run-all.sh`, never typed by hand.
directly under `output/` instead, with the equivalent depth in the article's own accordion sections.
| Module | Article | What it demonstrates | | Module | Article | What it demonstrates |
|---|---|---| |---|---|---|
| [`kafka-basics/`](kafka-basics/README.md) | [Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/) | The on-ramp: what the starter gives you, the two Jackson serializer families, where a key lands and why, and which defaults are Kafka's rather than Spring's | | [`kafka-basics/`](kafka-basics/README.md) | [Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/) | The on-ramp: what the starter gives you, the two Jackson serializer families, where a key lands and why, and which defaults are Kafka's rather than Spring's |
| [`kafka-error-handling/`](kafka-error-handling/README.md) | [Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/) | What the default error handler really does, why a poison pill stops a partition, and what blocking retries cost that retry topics do not | | [`kafka-error-handling/`](kafka-error-handling/README.md) | [Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/) | What the default error handler really does, why a poison pill stops a partition, and what blocking retries cost that retry topics do not |
| [`rabbitmq/`](rabbitmq/README.md) | [Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/) | All four exchange types against a real broker, manual acknowledgement, and a dead-letter path exercised through both rejection and TTL expiry | | [`rabbitmq/`](rabbitmq/README.md) | [Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/) | All four exchange types against a real broker, manual acknowledgement, and a dead-letter path exercised through both rejection and TTL expiry |
| [`sse-websocket/`](sse-websocket/README.md) | [Server-Sent Events and WebSocket on Spring Boot 4: SseEmitter, STOMP, and Which to Pick](https://ankurm.com/spring-boot-4-sse-websocket-stomp/) | The two browser-facing options side by side: the SSE lifecycle and the three ways a stream ends, STOMP's routing defaults, and the size limit that is enforced by Tomcat rather than by Spring |
| [`protocol-comparison/`](protocol-comparison/README.md) | [RSocket vs gRPC vs WebSocket on Spring Boot 4.1: When Each One Wins](https://ankurm.com/rsocket-vs-grpc-vs-websocket-spring-boot-4-1/) | One application serving the same two operations over all three, benchmarked in one JVM, ending in the only measurement that transfers off the box: what each server produces for a consumer that asked for a hundred |
| [`broker-comparison/`](broker-comparison/README.md) | [Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with Benchmarks](https://ankurm.com/kafka-vs-rabbitmq-vs-pulsar-java-decision-framework/) | The three brokers measured side by side on ordering, replay, consumer scaling and operational footprint, ending in a decision table where every row has a transcript behind it |
| [`saga/`](saga/README.md) | Saga Pattern in Spring Boot: Orchestration vs Choreography with Kafka | The same order/payment/inventory saga built twice against a real broker — choreography and orchestration — driven through the same inventory failure so the compensating transaction can be compared side by side, plus a real duplicate-delivery test for the idempotent-consumer problem |
| [`share-groups/`](share-groups/README.md) | [Queues for Kafka: Share Groups with Spring for Apache Kafka 4](https://ankurm.com/kafka-share-groups-spring-kafka-4-queues-for-kafka/) | Share groups against a real Kafka 4.3.1 broker: six consumers on two partitions, the acquire mode that decides whether they really share, the three acknowledgement types, a group that starts at the end of the topic, and the ordering you give up |
| [`kafka-streams/`](kafka-streams/README.md) | Kafka Streams with Spring Boot 4: Real-Time Fraud Detection with Vector Similarity | A fraud topology tested entirely under `TopologyTestDriver`: cosine-similarity scoring, a 5-minute windowed count that alerts exactly once, and a KTable blocklist join with a tombstone, plus the Spring wiring that needs no broker to load |
The brokers make an instructive set. Kafka's consumer holds an offset and the broker remembers The two brokers make an instructive pair. Kafka's consumer holds an offset and the broker
nothing about individual records; RabbitMQ's broker owns the message until it is acknowledged and remembers nothing about individual records; RabbitMQ's broker owns the message until it is
can route, expire and dead-letter it on its own; Pulsar keeps the log but lets each subscriber acknowledged and can route, expire and dead-letter it on its own. Almost every difference in how
choose how it is read. Almost every difference in how you handle failure follows from those three you handle failure follows from that one sentence — which is why Kafka needs a retry topic
sentences — which is why Kafka needs a retry topic to do what RabbitMQ does with a queue to do what RabbitMQ does with a queue argument.
argument, and why `broker-comparison` is mostly an argument about where a message lives.
The last two modules are about the *other* kind of messaging — a connection rather than a
broker — and they are deliberately adjacent to the broker modules, because the question
"should this be a topic or a stream?" is answered by whether anything needs to survive a dropped
connection. Nothing in `sse-websocket` or `protocol-comparison` persists a single message.
They are meant to be read in order. `kafka-basics` establishes that the default acknowledgement They are meant to be read in order. `kafka-basics` establishes that the default acknowledgement
mode is `BATCH` and therefore that delivery is at-least-once; everything the error-handling mode is `BATCH` and therefore that delivery is at-least-once; everything the error-handling
@@ -42,8 +29,8 @@ Central and from Boot's own `spring-boot-dependencies` POM, rather than from rel
announcements. announcements.
Every module runs its broker without Docker, so the transcripts can be regenerated on any Every module runs its broker without Docker, so the transcripts can be regenerated on any
machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ and machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ via a
Pulsar via real brokers started by the modules' own scripts. Each module also ships a Testcontainers real broker started by the module's own scripts. Each module also ships a Testcontainers
configuration for the cases where the deployed image is what you need to test against. configuration for the cases where the deployed image is what you need to test against.
## Licence ## Licence
broker-comparison/README.md
-80
View File
@@ -1,80 +0,0 @@
# broker-comparison
Companion project for **Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with
Benchmarks** on [ankurm.com](https://ankurm.com).
Three brokers, three questions, one set of measurements. Ordering, replay and consumer scaling are
asked identically of all three by driving the client libraries directly, so that what is compared
is the brokers' delivery models rather than three sets of Spring defaults. What each Spring
integration adds on top is [chapter 6](docs/06-what-spring-adds.md).
## Verified stack
| Component | Version | Source of the number |
|---|---|---|
| JDK (tests) | 25.0.4.1+1 (Temurin) | `java -version` |
| JDK (brokers) | 21.0.12.1+1 (Temurin) | `java -version` |
| Spring Boot | 4.1.1 | `maven-metadata.xml` on Maven Central |
| kafka-clients | 4.2.1 | `spring-boot-dependencies-4.1.1.pom` |
| amqp-client | 5.30.0 | `spring-boot-dependencies-4.1.1.pom` |
| pulsar-client | 4.2.4 | `spring-boot-dependencies-4.1.1.pom` |
| Kafka broker | 4.2.1 (KRaft) | `kafka_2.13-4.2.1.tgz` |
| RabbitMQ broker | 3.10.25 | generic-unix tarball |
| Pulsar broker | 4.2.4 standalone | `apache-pulsar-4.2.4-bin.tar.gz` |
RabbitMQ is pinned at 3.10.25 because the box that produced these transcripts has Erlang 24;
3.11 and later need Erlang 25. The AMQP 0-9-1 semantics measured here are unchanged in 4.x.
## Quickstart
The Kafka measurements need no external broker — `spring-kafka-test` starts a real KRaft broker
in-process. The other two need a broker each, and the scripts start both without Docker and
without root.
```bash
mvn -Dgroups=kafka test
ERL_ROOT=... RABBITMQ_HOME=... scripts/rabbit-broker.sh
mvn -Dgroups=rabbit test
PULSAR_HOME=... scripts/pulsar-broker.sh
mvn -Dgroups=pulsar test
scripts/footprint.sh kafka|rabbit|pulsar # the operational numbers
scripts/run-all.sh # regenerates every docs/output/ file
```
## Documentation
| Chapter | What it settles |
|---|---|
| [01 Three models](docs/01-three-models.md) | Where a message lives, and why everything else follows |
| [02 Ordering](docs/02-ordering.md) | What survives the second consumer, in all three |
| [03 Replay](docs/03-replay.md) | Reading it twice, and the one broker that cannot |
| [04 Consumer scaling](docs/04-consumer-scaling.md) | The partition ceiling, and the client buffer that defeats fan-out in the other two |
| [05 Operational footprint](docs/05-operational-footprint.md) | Startup, memory, ports, processes, configuration surface |
| [06 What Spring adds](docs/06-what-spring-adds.md) | The three integrations, and the Boot 4 starter trap |
| [07 The decision table](docs/07-the-decision-table.md) | Every row backed by a measurement, and how to choose |
## Captured output
| File | What it shows |
|---|---|
| [`kafka-ordering.txt`](docs/output/kafka-ordering.txt) | Per-key order kept, global order not — and why A, B and C were the wrong keys to demonstrate it |
| [`kafka-replay.txt`](docs/output/kafka-replay.txt) | The same 12 records read three times |
| [`kafka-consumer-scaling.txt`](docs/output/kafka-consumer-scaling.txt) | 5 consumers, 3 partitions, 2 idle |
| [`rabbit-ordering.txt`](docs/output/rabbit-ordering.txt) | Message 1 finishing tenth |
| [`rabbit-replay.txt`](docs/output/rabbit-replay.txt) | 12 messages, then 0, then an empty queue |
| [`rabbit-consumer-scaling.txt`](docs/output/rabbit-consumer-scaling.txt) | 40/0/0/0/0 without `basicQos`, 8 each with it |
| [`pulsar-ordering.txt`](docs/output/pulsar-ordering.txt) | `Shared` spreading a key, `Key_Shared` pinning it |
| [`pulsar-replay.txt`](docs/output/pulsar-replay.txt) | `seek(earliest)` and a new subscription |
| [`pulsar-consumer-scaling.txt`](docs/output/pulsar-consumer-scaling.txt) | The 1000-message receiver queue defeating fan-out |
| [`footprint.txt`](docs/output/footprint.txt) | Startup, memory, ports and config surface for all three |
| [`tests.txt`](docs/output/tests.txt) | The nine tests behind all of the above |
## What is not measured here
Throughput. A messages-per-second number from one 2-core container says nothing about any of
these brokers, and publishing one would be worse than publishing nothing. The measurements above
are structural: they hold on any hardware, because they are properties of the delivery models
rather than of the machine.
broker-comparison/docs/01-three-models.md
-49
View File
@@ -1,49 +0,0 @@
[README](../README.md) · next: [Ordering](02-ordering.md)
# 1. Three storage models, and everything that follows from them
Almost every difference between these brokers follows from one sentence about where a message
lives.
- **Kafka** is an append-only log, partitioned. The broker keeps every record for the retention
period and remembers nothing about individual consumers except an offset. Reading does not
remove anything.
- **RabbitMQ** is a router with queues. The broker owns each message until a consumer
acknowledges it, and then deletes it. It can route, expire, and dead-letter on its own.
- **Pulsar** is a log too, but the cursor and the storage are separate services: brokers are
stateless and BookKeeper holds the data. Acknowledgement moves a cursor, and a message is
deleted once every subscription has passed it — unless a retention policy says otherwise.
Read those three sentences again before reading any comparison table, including the one in
[chapter 7](07-the-decision-table.md). They predict most of it.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| unit of parallelism | partition | queue | subscription |
| set when | topic is created | queue is declared | consumer subscribes |
| consuming | moves an offset | deletes the message | moves a cursor |
| a second reader | new consumer group | another queue, bound in advance | new subscription |
The last row is the one people underestimate. In Kafka and Pulsar you can add a reader that sees
history you have already processed. In RabbitMQ you cannot add one after the fact at all — the
messages are gone — so the decision to have a second consumer has to be made *before* the
messages arrive.
## What was measured, and how
Three questions, asked identically of all three brokers by driving the client libraries directly
rather than through three different Spring abstractions:
| Question | Chapter |
|---|---|
| What ordering survives when you add a second consumer? | [02](02-ordering.md) |
| Can you read the same message twice? | [03](03-replay.md) |
| How far do consumers scale, and what stops them? | [04](04-consumer-scaling.md) |
| What does each one cost to run? | [05](05-operational-footprint.md) |
Versions are the ones a Spring Boot 4.1.1 application gets: kafka-clients 4.2.1, amqp-client
5.30.0, pulsar-client 4.2.4, all read from `spring-boot-dependencies-4.1.1.pom`. The brokers are
Kafka 4.2.1, RabbitMQ 3.10.25 and Pulsar 4.2.4, each started from its own distribution by a script
in `scripts/`.
next: [Ordering](02-ordering.md)
broker-comparison/docs/02-ordering.md
-66
View File
@@ -1,66 +0,0 @@
prev: [Three models](01-three-models.md) · [README](../README.md) · next: [Replay](03-replay.md)
# 2. Ordering
Every broker here is FIFO with one producer and one consumer. The question that matters is what
survives the second consumer, because that is the first thing you add.
## Kafka: ordered within a partition, and a key picks the partition
Twelve records, three keys, three partitions
([`docs/output/kafka-ordering.txt`](output/kafka-ordering.txt)):
```
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
```
Each key's values come back in the order they were produced. The global sequence does not: the
consumer drains one partition's buffer before moving to the next, so the delivered order is
2, 5, 8, 11, 1, 4, 7, 10, 3, 6, 9, 12.
That is the whole Kafka ordering guarantee, and it is usually enough, because "in order" nearly
always means "in order per customer / per account / per device" rather than globally.
**A detail worth stealing.** The keys in that transcript are D, A and F, not A, B and C. With
three partitions, murmur2 sends A, B *and* C all to partition 1. Picking three obvious keys to
demonstrate partitioning would have produced a transcript in which global order was accidentally
preserved. Before you conclude that your keys spread evenly, compute
`Utils.toPositive(Utils.murmur2(key)) % partitions` for the ones you actually use.
## RabbitMQ: FIFO per queue, and no key at all
One consumer sees the queue in order. Two consumers do not
([`docs/output/rabbit-ordering.txt`](output/rabbit-ordering.txt)):
```
one queue, one consumer : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
```
Message 1 went to the slower consumer and finished tenth. There is no key and no partition, so
there is nothing to pin related messages to one consumer. Ordering across related messages in
RabbitMQ means one queue and one consumer — which means no horizontal scaling of that queue —
or the consistent-hash exchange, which is a plugin and effectively reintroduces partitions by
hand.
## Pulsar: the subscription type decides, per subscriber
Two consumers on a `Shared` subscription and two on `Key_Shared`, same topic, same messages
([`docs/output/pulsar-ordering.txt`](output/pulsar-ordering.txt)):
```
Shared A -> [consumer-1, consumer-2] B -> [consumer-1, consumer-2] C -> [consumer-1, consumer-2]
Key_Shared A -> [consumer-2] B -> [consumer-1] C -> [consumer-2]
```
Under `Shared`, every key was handled by both consumers, so two messages with the same key can be
in flight at once and no ordering claim survives. Under `Key_Shared`, each key went to exactly one
consumer — Kafka's guarantee, but chosen by the subscriber rather than fixed by a partition count
someone picked when the topic was created, and recomputed as consumers join and leave.
That last property is the strongest single argument for Pulsar: two teams reading the same topic
can make different ordering-versus-throughput trades without negotiating.
next: [Replay](03-replay.md)
broker-comparison/docs/03-replay.md
-59
View File
@@ -1,59 +0,0 @@
prev: [Ordering](02-ordering.md) · [README](../README.md) · next: [Consumer scaling](04-consumer-scaling.md)
# 3. Replay
"Can I read that again?" is asked after a bad deploy, and by then it is too late to change
brokers.
## Kafka
```
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
```
([`docs/output/kafka-replay.txt`](output/kafka-replay.txt)) Consuming moves an offset and removes
nothing. A new group, a reset offset and a `seekToBeginning` all read the same records, for as
long as retention keeps them.
## RabbitMQ
```
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
```
([`docs/output/rabbit-replay.txt`](output/rabbit-replay.txt)) Acknowledging deletes. There is no
offset to rewind and no second reader that can see what the first consumed. Reading a message
twice has to be arranged in advance — a second queue bound to the same exchange, or a copy written
somewhere else — and it cannot be arranged afterwards.
This is not a defect. A router that deletes what it has delivered is much cheaper to operate than
a log, and most work queues genuinely do not need history. It only becomes a defect at the moment
you need history and do not have it.
RabbitMQ streams (3.9+) are a separate, log-shaped feature that does support replay. They are a
different thing living in the same broker, with their own client protocol and their own semantics
— worth knowing about, and not what you get from `queueDeclare`.
## Pulsar
```
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
```
([`docs/output/pulsar-replay.txt`](output/pulsar-replay.txt)) `seek(MessageId)` and
`seek(timestamp)` rewind a live subscription; a new subscription starting from `Earliest` reads
everything still stored.
The difference from Kafka is the **default**, and it is the one that surprises people: Kafka keeps
a record for the retention period regardless of who read it, while Pulsar deletes a message once
every subscription has acknowledged it, unless a retention policy on the namespace says otherwise.
A Pulsar namespace with the default retention and one well-behaved subscription keeps nothing —
so the replay you are counting on requires a policy you have to set.
next: [Consumer scaling](04-consumer-scaling.md)
broker-comparison/docs/04-consumer-scaling.md
-80
View File
@@ -1,80 +0,0 @@
prev: [Replay](03-replay.md) · [README](../README.md) · next: [Operational footprint](05-operational-footprint.md)
# 4. Consumer scaling
The question is what happens when you add the fifth consumer.
## Kafka: partitions are a hard ceiling
Five consumers in one group on a three-partition topic
([`docs/output/kafka-consumer-scaling.txt`](output/kafka-consumer-scaling.txt)):
```
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
```
A partition belongs to at most one consumer in a group, so partition count is the ceiling. Two of
the five processes are running, connected, healthy, and doing nothing at all — and they will keep
reporting healthy forever.
Raising the partition count later is possible and is not free: it changes which partition a key
maps to, so the per-key ordering guarantee is broken across the change for every key that moves.
In practice the partition count is a capacity decision made at design time, on incomplete
information, that you then live with.
## RabbitMQ: no ceiling, but prefetch decides whether it is real
Five consumers on one queue, forty messages
([`docs/output/rabbit-consumer-scaling.txt`](output/rabbit-consumer-scaling.txt)):
```
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 consumer-2..5 -> 0
basicQos(1):
consumer-1..5 -> 8 each
```
There is no structural ceiling — but with the AMQP default the broker pushes as many messages as
a consumer will accept, so the first consumer to connect can be handed the entire backlog while
four idle processes wait. `basicQos` is not a tuning knob to postpone; it is what makes the
fan-out exist.
Spring AMQP sets it for you: `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` is 250.
Better than unlimited, and still enough to concentrate any backlog smaller than 250 messages on
one consumer.
## Pulsar: no ceiling either, and the same trap under a different name
Five consumers, one *non-partitioned* topic, forty messages
([`docs/output/pulsar-consumer-scaling.txt`](output/pulsar-consumer-scaling.txt)):
```
receiverQueueSize default (1000):
one or two consumers take everything; the rest receive nothing
receiverQueueSize(1):
consumer-1..5 -> 8 each
```
The topic has no partitions and five consumers still share the work, which in Kafka would require
at least five partitions decided in advance. But the default receiver queue is a thousand
messages, so the first consumers to connect pull the whole backlog into their own buffers first.
Same failure as RabbitMQ's unlimited prefetch, four times the default.
## The shape of the answer
- Kafka gives you ordering per key and pays for it with a fixed parallelism ceiling.
- RabbitMQ gives you unlimited competing consumers and pays for it with no ordering across them.
- Pulsar lets each subscription choose, which is genuinely the best of both — at the operational
cost measured in [the next chapter](05-operational-footprint.md).
And all three brokers have a client-side buffer that will quietly defeat your fan-out if you leave
it at the default and your backlog is small. That is the one finding here that applies to
whichever you pick.
next: [Operational footprint](05-operational-footprint.md)
broker-comparison/docs/05-operational-footprint.md
-55
View File
@@ -1,55 +0,0 @@
prev: [Consumer scaling](04-consumer-scaling.md) · [README](../README.md) · next: [What Spring adds](06-what-spring-adds.md)
# 5. Operational footprint
"Ops burden" cannot be benchmarked. The number of moving parts, the memory floor and the size of
the configuration surface can be, and they are what people are actually asking about.
Each broker started from its shipped distribution with default configuration and a 700 MB heap
cap, on one 2-core / 3.8 GB Linux box, Temurin JDK 21
([`docs/output/footprint.txt`](output/footprint.txt)):
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 standalone |
|---|---|---|---|
| launch to first accepted connection | 10.0 s | 15.1 s | 20.8 s |
| resident memory at idle | 333 MB | 115 MB | 610 MB |
| server processes | 1 | 1 (+ `epmd`) | 1 |
| listening ports | 9092, 9093 | 5672, 4369, 25672 | 6650, 8080, 2181 |
| unpacked distribution | 135 MB | 26 MB | 344 MB |
| settings in the shipped default config | 24 | **0** | 357 |
Read those numbers as orders of magnitude, not as a benchmark: one sample, one small machine.
Three things they say clearly.
**RabbitMQ ships no configuration file at all.** The boot log records `Config file(s): (none)`;
`rabbitmq.conf.example` is entirely commented out. Everything works out of the box, and the
smallest resident footprint here is the one that is not a JVM. Against that, `epmd` on 4369 and
the inter-node port on 25672 are the Erlang distribution, which is also how clustering works and
how clustering goes wrong.
**Pulsar standalone is three systems in one process.** Port 2181 is ZooKeeper and the on-disk
data is BookKeeper's; the standalone distribution hides that behind one command. A real
deployment does not: you operate ZooKeeper (or, from Pulsar 3.x, an alternative metadata store),
BookKeeper bookies and Pulsar brokers as three tiers with three scaling stories. The 357-setting
`standalone.conf` is the honest signal here — that is the configuration surface, and running it
seriously means learning most of it.
**Kafka since KRaft is genuinely simpler than it was.** One process, two ports, 24 settings, no
ZooKeeper. The old "Kafka means also running ZooKeeper" objection is a version behind; anyone
comparing on that basis is comparing to 2022.
## What the numbers do not include
The footprint above is one node at idle. The thing that actually determines ops burden is what
happens at three in the morning:
- **Kafka**: rebalances, consumer lag as the primary signal, partition-count decisions you cannot
cleanly reverse, and a broad ecosystem of tools that assume Kafka.
- **RabbitMQ**: the management plugin is genuinely good, queue depth is a direct and obvious
signal, and the hard problem is network partitions in a cluster — which the Erlang distribution
makes fast to detect and awkward to resolve.
- **Pulsar**: the fewest people on your team will have run it. That is not a technical property
and it is usually the deciding one.
next: [What Spring adds](06-what-spring-adds.md)
broker-comparison/docs/06-what-spring-adds.md
-52
View File
@@ -1,52 +0,0 @@
prev: [Operational footprint](05-operational-footprint.md) · [README](../README.md) · next: [The decision table](07-the-decision-table.md)
# 6. What Spring adds, and what Boot 4 no longer gives you for free
The measurements in this module drive the client libraries directly, so that what is being
compared is the brokers rather than three sets of Spring defaults. In an application you would use
the Spring integrations, and they are not equivalent to one another.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| project | Spring for Apache Kafka 4.1.1 | Spring AMQP 4.1.1 | Spring for Apache Pulsar 2.0.7 |
| starter | `spring-boot-starter-kafka` | `spring-boot-starter-amqp` | `spring-boot-starter-pulsar` |
| listener | `@KafkaListener` | `@RabbitListener` | `@PulsarListener` |
| template | `KafkaTemplate` | `RabbitTemplate` | `PulsarTemplate` |
| retry / DLQ | `DefaultErrorHandler`, `@RetryableTopic` | dead-letter exchange, container error handler | `DeadLetterPolicy` on the listener |
All three are Boot-managed at 4.1.1, so you do not pin their versions.
## The Boot 4 trap that applies to all three
**In Spring Boot 4, depending on a messaging library directly rather than through its Boot starter
means you have no auto-configuration.** The auto-configuration classes moved out of
`spring-boot-autoconfigure` into per-technology modules that only the starters bring:
| depending on | what you lose | how it presents |
|---|---|---|
| `org.springframework.kafka:spring-kafka` | `spring-boot-kafka` | `No qualifying bean of type KafkaTemplate<...>` — the context starts fine |
| `org.springframework.amqp:spring-rabbit` | `spring-boot-amqp` | no `RabbitTemplate`, no `RabbitAdmin` |
| `org.springframework.pulsar:spring-pulsar` | `spring-boot-pulsar` | no `PulsarTemplate` |
Every Boot 3 tutorial gets this wrong now, and the symptom is a missing bean rather than anything
that names the cause. Use the starters.
## The Jackson fork, in both Kafka and RabbitMQ
Boot 4 moved to Jackson 3 (`tools.jackson`), and both messaging projects ship converters for
both generations. **A `2` in the class name means the previous Jackson** — which is the opposite
of the convention you would guess:
- Spring Kafka: `JsonSerializer`/`JsonDeserializer` are Jackson 2; `JacksonJsonSerializer`,
`JacksonJsonDeserializer` and `JacksonJsonSerde` are Jackson 3.
- Spring AMQP: `Jackson2JsonMessageConverter` is Jackson 2; `JacksonJsonMessageConverter` is
Jackson 3.
Picking the wrong one gives you `SerializationException: Can't serialize data`, whose cause is
`Java 8 date/time type java.time.Instant not supported by default` — a message about date types
for what is really a wrong-library problem.
The [kafka-basics](../kafka-basics/README.md) and [rabbitmq](../rabbitmq/README.md) modules in
this repository cover both stacks in detail.
next: [The decision table](07-the-decision-table.md)
broker-comparison/docs/07-the-decision-table.md
-59
View File
@@ -1,59 +0,0 @@
prev: [What Spring adds](06-what-spring-adds.md) · [README](../README.md)
# 7. The decision table
Every row below is either a measurement in `docs/output/` or a statement about operating the
thing. Nothing here is a vendor claim.
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 |
|---|---|---|---|
| ordering with one consumer | FIFO per partition | FIFO per queue | FIFO |
| ordering with many consumers | per key, always | **none** | per key with `Key_Shared`, none with `Shared` |
| who chooses that | topic design | plugin, or one consumer | each subscription, independently |
| replay after the fact | yes, within retention | **no** | yes, if retention is configured |
| second independent reader | new consumer group, any time | must be arranged in advance | new subscription, any time |
| consumer parallelism ceiling | partition count | none | none |
| changing that ceiling | repartition; breaks key→partition | nothing to change | nothing to change |
| the client-side trap | none by default | unlimited prefetch (Spring: 250) | `receiverQueueSize` 1000 |
| memory at idle | 333 MB | **115 MB** | 610 MB |
| processes to operate | 1 (KRaft) | 1 + `epmd` | broker + BookKeeper + metadata store |
| shipped default settings | 24 | **0** | 357 |
| Spring project maturity | very high | very high | good, much smaller community |
## Choosing
**Choose RabbitMQ** when the work is a queue of tasks: each message is an instruction, order
between instructions does not matter, and once it is done it is done. It is the smallest thing
that works, it has the best out-of-the-box operability of the three, and queue depth is a metric
anyone can interpret. Choose it in the knowledge that you are giving up replay permanently —
adding it later is a broker migration, not a configuration change.
**Choose Kafka** when the messages are events other people will want to read: when more than one
consumer will exist, when someone will need to reprocess history after a bug, or when ordering per
entity is part of the contract. Pay for it with a partition count decided too early, consumers
that scale only as far as that number, and an operational model where lag is the thing you watch.
KRaft has removed the old ZooKeeper objection.
**Choose Pulsar** when you genuinely need what neither of the others gives you: per-subscription
choice of ordering versus fan-out, consumer counts not bounded by a number chosen at topic
creation, or multi-tenancy with real isolation. It is the most capable design here. It is also
three systems, 357 settings, five times RabbitMQ's memory floor, and the one your team has least
experience with — and that last point decides more incidents than the first three prevent.
**If the honest answer is "we do not know yet"**, that argues for Kafka, on grounds that have
nothing to do with the technology: you can hire for it, your monitoring vendor supports it, and
the failure modes are documented by thousands of people who hit them first. That is a real
engineering argument and it is fine to make it out loud.
## The question that dissolves the choice
A surprising number of these decisions are made for a system that has one producer, one consumer
and fewer than a hundred messages a second. At that volume all three brokers work, none of the
measurements above will ever be reached, and the decision is entirely about what your team can
operate at three in the morning.
The measurements matter when you can name which row you are relying on. If you cannot, you are
picking an operational burden, not a broker — and the cheapest one to operate is the one in the
`115 MB` cell.
[README](../README.md)
broker-comparison/docs/output/footprint.txt
-40
View File
@@ -1,40 +0,0 @@
== Operational footprint, measured on one 2-core / 3.8 GB Linux box ==
Each broker started from its shipped distribution with default configuration and a
700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as
orders of magnitude, not as a benchmark.
broker : kafka
time from launch to first
accepted connection : 10035 ms
resident memory at idle : 333 MB
server processes : 1
listening ports : 9092 (broker), 9093 (controller)
unpacked distribution size : 135 MB
settings in shipped default
configuration file : 24 (server.properties)
broker : rabbit
time from launch to first
accepted connection : 15053 ms
resident memory at idle : 115 MB
server processes : 1
listening ports : 5672 (AMQP), 4369 (epmd), 25672 (inter-node)
unpacked distribution size : 26 MB
settings in shipped default
configuration file : 0 (rabbitmq.conf.example)
epmd processes : 1
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
entirely commented out and the boot log records "Config file(s): (none)".
Everything in that file is documentation, not a default.
broker : pulsar
time from launch to first
accepted connection : 20801 ms
resident memory at idle : 610 MB
server processes : 1
listening ports : 6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)
unpacked distribution size : 344 MB
settings in shipped default
configuration file : 357 (standalone.conf)
broker-comparison/docs/output/kafka-consumer-scaling.txt
-16
View File
@@ -1,16 +0,0 @@
== Kafka: partitions are the ceiling on consumer parallelism ==
topic 'orders-scaling', 3 partitions, 5 consumers in one group
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
consumers with no partitions: 2
A partition is assigned to at most one consumer in a group, so the number
of partitions is a hard ceiling on consumer parallelism. Adding consumers
beyond it adds idle processes, not throughput.
broker-comparison/docs/output/kafka-ordering.txt
-30
View File
@@ -1,30 +0,0 @@
== Kafka: ordering is per partition ==
produced 12 records, keys D/A/F round-robin, values 1..12 in order
(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C
all hash to partition 1 with three partitions, which is worth knowing
before you decide your keys are well distributed.)
consumed in this order:
F=2@p2
F=5@p2
F=8@p2
F=11@p2
A=1@p1
A=4@p1
A=7@p1
A=10@p1
D=3@p0
D=6@p0
D=9@p0
D=12@p0
per key:
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
Order is preserved within each key because a key hashes to one partition.
Across keys it is not: the values above are not 1..12 in order, because a
consumer drains one partition's buffer before the next.
broker-comparison/docs/output/kafka-replay.txt
-10
View File
@@ -1,10 +0,0 @@
== Kafka: the log is the storage ==
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
Consuming does not remove anything. A consumer group is a cursor over a log
that the broker keeps until retention expires, so a new group, a reset
offset or a seek all read the same records again.
broker-comparison/docs/output/pulsar-consumer-scaling.txt
-30
View File
@@ -1,30 +0,0 @@
== Pulsar: no partition ceiling, but the receiver queue decides who gets the work ==
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
receiverQueueSize left at the default (1000):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
receiverQueueSize(1):
consumer-1 -> 8 messages
consumer-2 -> 8 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 8 messages
consumers that received nothing : 0
The topic has no partitions and five consumers can still share the work --
in Kafka the same shape needs at least five partitions, chosen when the
topic was created. But the default receiver queue is 1000 messages, so the
first consumers to connect pull the whole 40-message backlog into their own
buffers before the rest ask for anything, and the subscription looks
broken. Which consumers win is a race and varies between runs -- one
consumer taking all forty, or two taking twenty each -- but the consumers
that lose it see nothing at all. This is the same trap as RabbitMQ's
unbounded prefetch, with a different name and a much larger default.
broker-comparison/docs/output/pulsar-ordering.txt
-24
View File
@@ -1,24 +0,0 @@
== Pulsar: the subscription type decides ==
12 messages, keys A/B/C, values 1..12 in order, two consumers per
subscription, receiverQueueSize 1 so the first consumer cannot take the
whole backlog.
Shared subscription, which consumers saw each key:
A -> [consumer-1, consumer-2]
B -> [consumer-1, consumer-2]
C -> [consumer-1, consumer-2]
Key_Shared subscription, which consumers saw each key:
A -> [consumer-1]
B -> [consumer-1]
C -> [consumer-1]
A Shared subscription round-robins individual messages, so messages with
the same key end up on different consumers and can be processed at the
same time: there is no per-key order left to speak of. Key_Shared hashes
the key to one consumer, which is Kafka's guarantee -- except that the
assignment belongs to the subscription and is recomputed as consumers come
and go, rather than being fixed by a partition count chosen when the topic
was created.
broker-comparison/docs/output/pulsar-replay.txt
-14
View File
@@ -1,14 +0,0 @@
== Pulsar: acknowledged, but still there ==
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
Acknowledgement moves a cursor; the message itself lives in the managed
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
which Kafka can also do by resetting offsets. What differs is the default:
Pulsar deletes a message once every subscription has acknowledged it,
unless a retention policy on the namespace says otherwise, whereas Kafka
keeps it for the retention period regardless of who read it. A Pulsar
topic with no retention policy and no subscriptions keeps nothing.
broker-comparison/docs/output/rabbit-consumer-scaling.txt
-34
View File
@@ -1,34 +0,0 @@
== RabbitMQ: consumers scale, and prefetch decides whether they actually do ==
one queue, 40 messages, 5 consumers, each taking 10 ms per message
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
basicQos(1):
consumer-1 -> 8 messages
consumer-2 -> 9 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 7 messages
consumers that received nothing : 0
Every consumer on a queue competes for the same messages, so adding
consumers adds throughput and there is no structural ceiling of the kind
Kafka's partition count imposes. But with the AMQP default the broker
pushes as many messages as a consumer will take, so whichever consumer
connects first can be handed the entire backlog while the others sit idle.
basicQos is not a tuning knob you get to postpone; it is what makes the
fan-out real. Spring AMQP sets it for you --
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
better than unlimited and still large enough to concentrate a small
backlog on one consumer.
The price of all this is the ordering measurement: there is no key, so
nothing constrains related messages to one consumer.
broker-comparison/docs/output/rabbit-ordering.txt
-18
View File
@@ -1,18 +0,0 @@
== RabbitMQ: FIFO per queue, per consumer ==
one queue, one consumer, 12 messages
completion order : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
completion order : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
out-of-order steps: 1
A queue is FIFO and a single consumer sees it that way. The moment a second
consumer is added the broker hands the next message to whichever consumer
is free, so the order in which work finishes is no longer the order in
which it was published. There is no key: RabbitMQ has no notion of a
partition to which related messages could be pinned. Ordering across
related messages means one queue and one consumer, and therefore no
horizontal scaling for that queue -- or a consistent-hash exchange, which
is a plugin.
broker-comparison/docs/output/rabbit-replay.txt
-12
View File
@@ -1,12 +0,0 @@
== RabbitMQ: there is nothing to replay ==
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
Acknowledging a message deletes it. The broker is a router with buffers,
not a log: there is no offset to rewind and no second reader that can see
what the first one consumed. Reading the same message twice means
arranging it in advance -- a second queue bound to the same exchange, or a
copy written somewhere else -- and it cannot be arranged after the fact.
broker-comparison/docs/output/tests.txt
-12
View File
@@ -1,12 +0,0 @@
[INFO] Running com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.88 s -- in com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.732 s -- in com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.00 s -- in com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
broker-comparison/pom.xml
-66
View File
@@ -1,66 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<!-- Every client version below is managed by spring-boot-dependencies 4.1.1, so the three
brokers are compared at the versions a Boot 4.1 application would actually get:
kafka-clients 4.2.1, amqp-client 5.30.0, pulsar-client 4.2.4. Read from
spring-boot-dependencies-4.1.1.pom, not from release announcements. -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>broker-comparison</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<!-- The three client libraries, driven directly. The comparison is between the brokers'
delivery models, and going through three different Spring abstractions would put three
different sets of defaults between the measurement and the thing being measured.
What each Boot starter adds on top is covered in docs/06-what-spring-adds.md. -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- Provides the in-process KRaft broker used by the Kafka measurements: a real Kafka
broker, no Docker. -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<!-- Each broker is started by its own script and the matching tagged tests are run on
their own; see scripts/run-all.sh, which passes -Dgroups=kafka|rabbit|pulsar. Running
"mvn test" with no group runs all three and needs all three brokers up. -->
</project>
broker-comparison/scripts/footprint.sh
-73
View File
@@ -1,73 +0,0 @@
#!/usr/bin/env bash
# Measures the operational footprint of one broker: how long it takes to accept a connection,
# how much memory it holds at idle, how many processes and listening ports it opens, and how
# large its shipped default configuration is.
#
# These are proxies, not a benchmark. "Ops burden" is not directly measurable, but the number of
# moving parts, the memory floor and the size of the configuration surface are, and they are what
# people are actually asking about when they ask which broker is heavier to run.
#
# scripts/footprint.sh kafka|rabbit|pulsar
set -euo pipefail
BROKER="${1:?kafka|rabbit|pulsar}"
extra=""
port_open() { (echo > "/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
rss_mb() { ps -eo rss,cmd | grep -F "$1" | grep -v grep | awk '{s+=$1} END {printf "%.0f", s/1024}'; }
procs() { ps -eo cmd | grep -F "$1" | grep -cv grep; }
settings() { grep -cE '^[a-zA-Z]' "$1" 2>/dev/null || echo 0; }
start=$(date +%s%3N)
case "$BROKER" in
kafka)
: "${KAFKA_HOME:?set KAFKA_HOME}"
# Wipe whatever log.dirs points at, so the format below is not refused with
# "Invalid cluster.id ... Expected X, but read Y" from a previous run.
KAFKA_LOG_DIRS=$(grep -E '^log.dirs=' "$KAFKA_HOME/config/server.properties" | cut -d= -f2)
rm -rf ${KAFKA_LOG_DIRS//,/ }
"$KAFKA_HOME/bin/kafka-storage.sh" format --standalone -t "$("$KAFKA_HOME/bin/kafka-storage.sh" random-uuid)" \
-c "$KAFKA_HOME/config/server.properties" --ignore-formatted >/dev/null
start=$(date +%s%3N) # after formatting: kafka-storage.sh is two more JVM starts and is a
# one-off, not part of what a restart costs
setsid nohup "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/server.properties" \
> /tmp/kafka-start.log 2>&1 < /dev/null &
for _ in $(seq 1 90); do port_open 9092 && break; sleep 1; done
ready=$(( $(date +%s%3N) - start ))
marker="kafka.Kafka"; ports="9092 (broker), 9093 (controller)"; conf="$KAFKA_HOME/config/server.properties"
dist="$KAFKA_HOME"
;;
rabbit)
: "${RABBITMQ_HOME:?set RABBITMQ_HOME}"; : "${ERL_ROOT:?set ERL_ROOT}"
scripts/rabbit-broker.sh >/dev/null
ready=$(( $(date +%s%3N) - start ))
# The broker is an Erlang VM, so the process to measure is beam.smp, not anything named
# "rabbitmq". epmd is a second, tiny process and is counted separately below.
marker="beam.smp"; ports="5672 (AMQP), 4369 (epmd), 25672 (inter-node)"
conf="$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf.example"; dist="$RABBITMQ_HOME"
extra="epmd processes : $(procs epmd)
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
entirely commented out and the boot log records \"Config file(s): (none)\".
Everything in that file is documentation, not a default."
;;
pulsar)
: "${PULSAR_HOME:?set PULSAR_HOME}"
scripts/pulsar-broker.sh >/dev/null
ready=$(( $(date +%s%3N) - start ))
marker="PulsarStandaloneStarter"; ports="6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)"
conf="$PULSAR_HOME/conf/standalone.conf"; dist="$PULSAR_HOME"
;;
*) echo "unknown broker $BROKER" >&2; exit 2 ;;
esac
cat <<EOF
broker : $BROKER
time from launch to first
accepted connection : ${ready} ms
resident memory at idle : $(rss_mb "$marker") MB
server processes : $(procs "$marker")
listening ports : $ports
unpacked distribution size : $(du -sm "$dist" | cut -f1) MB
settings in shipped default
configuration file : $(settings "$conf") ($(basename "$conf"))
${extra:-}
EOF
broker-comparison/scripts/pulsar-broker.sh
-25
View File
@@ -1,25 +0,0 @@
#!/usr/bin/env bash
# Start a real Apache Pulsar broker with no Docker and no root.
#
# The standalone distribution bundles ZooKeeper, BookKeeper and the broker in one process, which
# is why a single tarball is enough. -nss skips the stream storage (BookKeeper's table service,
# needed only by stateful Pulsar Functions) and -nfw skips the functions worker; both save
# several hundred megabytes of heap and a few seconds of startup.
#
# PULSAR_HOME must point at an unpacked apache-pulsar-<version>-bin directory. Pulsar 4.2
# supports Java 17 and 21; the transcripts here were produced on Temurin 21.
set -eu
: "${PULSAR_HOME:?set PULSAR_HOME to an unpacked apache-pulsar-*-bin directory}"
export PULSAR_MEM="${PULSAR_MEM:--Xms384m -Xmx700m -XX:MaxDirectMemorySize=384m}"
cd "$PULSAR_HOME"
setsid nohup bin/pulsar standalone -nss -nfw > /tmp/pulsar-start.log 2>&1 < /dev/null &
for _ in $(seq 1 90); do
if [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/admin/v2/brokers/version)" = "200" ]; then
echo "broker ready on 6650 (admin 8080), version $(curl -s http://127.0.0.1:8080/admin/v2/brokers/version)"
exit 0
fi
sleep 2
done
echo "broker did not start; see /tmp/pulsar-start.log" >&2
exit 1
broker-comparison/scripts/rabbit-broker.sh
-31
View File
@@ -1,31 +0,0 @@
#!/usr/bin/env bash
# Start a real RabbitMQ broker with no Docker and no root.
#
# RabbitMQ is an Erlang application, so an Erlang runtime and epmd on the path are the whole
# dependency. Point ERL_ROOT at an Erlang installation and RABBITMQ_HOME at an unpacked
# rabbitmq-server-generic-unix tarball. Ubuntu 22.04 ships Erlang 24, whose newest compatible
# broker is 3.10.25; 3.11 and later need Erlang 25.
#
# The one non-obvious step is starting epmd yourself: rabbitmq-server's own attempt to start it
# fails in a container without a resolvable hostname, and the failure surfaces as a forty-line
# Erlang crash dump ending in {'EXIT',nodistribution}.
set -eu
: "${ERL_ROOT:?set ERL_ROOT to an Erlang installation directory}"
: "${RABBITMQ_HOME:?set RABBITMQ_HOME to an unpacked rabbitmq-server-generic-unix directory}"
export PATH="$ERL_ROOT/bin:$ERL_ROOT/erts-"*/bin":$PATH"
export RABBITMQ_MNESIA_BASE="${RABBITMQ_MNESIA_BASE:-/tmp/rmq/data}"
export RABBITMQ_LOG_BASE="${RABBITMQ_LOG_BASE:-/tmp/rmq/log}"
export RABBITMQ_NODENAME="${RABBITMQ_NODENAME:-rabbit@localhost}"
export HOME="${HOME:-/tmp/rmq}"
mkdir -p "$RABBITMQ_MNESIA_BASE" "$RABBITMQ_LOG_BASE"
epmd -daemon 2>/dev/null || true
sleep 1
setsid nohup "$RABBITMQ_HOME/sbin/rabbitmq-server" > /tmp/rmq/boot.log 2>&1 < /dev/null &
for _ in $(seq 1 60); do
if (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; then echo "broker ready on 5672"; exit 0; fi
sleep 1
done
echo "broker did not start; see /tmp/rmq/boot.log" >&2
exit 1
broker-comparison/scripts/run-all.sh
-37
View File
@@ -1,37 +0,0 @@
#!/usr/bin/env bash
# Regenerates every file under docs/output/. Each broker is started, measured and left running
# only for its own group of tests, because three brokers do not fit comfortably in memory on a
# small machine at the same time.
#
# Required environment:
# KAFKA_HOME unpacked kafka_2.13-4.2.1
# ERL_ROOT an Erlang 24 installation (RabbitMQ 3.10.x)
# RABBITMQ_HOME unpacked rabbitmq-server-generic-unix-3.10.25
# PULSAR_HOME unpacked apache-pulsar-4.2.4-bin
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p docs/output
# Kafka needs no external broker for the measurements: spring-kafka-test starts a real KRaft
# broker in-process. KAFKA_HOME is only used by the footprint measurement.
mvn -B -Dgroups=kafka test 2>&1 | grep -E 'Running |Tests run:|BUILD ' > docs/output/tests.txt
scripts/rabbit-broker.sh
mvn -B -Dgroups=rabbit test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
scripts/pulsar-broker.sh
mvn -B -Dgroups=pulsar test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
{
echo "== Operational footprint, measured on one 2-core / 3.8 GB Linux box =="
echo
echo "Each broker started from its shipped distribution with default configuration and a"
echo "700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as"
echo "orders of magnitude, not as a benchmark."
echo
scripts/footprint.sh kafka; echo
scripts/footprint.sh rabbit; echo
scripts/footprint.sh pulsar
} > docs/output/footprint.txt
echo "regenerated:"; ls -1 docs/output
broker-comparison/src/main/java/com/ankurm/brokers/BrokerComparisonApplication.java
-15
View File
@@ -1,15 +0,0 @@
package com.ankurm.brokers;
/**
* There is deliberately no Spring application here. The measurements in src/test drive the three
* client libraries directly so that the thing being compared is the broker's delivery model and
* not three different sets of Spring defaults.
*
* <p>What each Spring Boot starter adds on top &mdash; and what it silently does not add, which
* in Boot 4 is more than people expect &mdash; is in docs/06-what-spring-adds.md.
*/
public final class BrokerComparisonApplication {
private BrokerComparisonApplication() {
}
}
broker-comparison/src/test/java/com/ankurm/brokers/Capture.java
-23
View File
@@ -1,23 +0,0 @@
package com.ankurm.brokers;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
/** Writes a transcript under docs/output/. Nothing in the article is typed by hand. */
public final class Capture {
private Capture() {
}
public static void write(String fileName, String heading, String body) {
Path dir = Path.of(System.getProperty("user.dir"), "docs", "output");
try {
Files.createDirectories(dir);
Files.writeString(dir.resolve(fileName), "== " + heading + " ==\n\n" + body + "\n");
}
catch (IOException ex) {
throw new IllegalStateException("could not write " + fileName, ex);
}
}
}
broker-comparison/src/test/java/com/ankurm/brokers/KafkaComparisonTest.java
-232
View File
@@ -1,232 +0,0 @@
package com.ankurm.brokers;
import java.time.Duration;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.kafka.test.EmbeddedKafkaKraftBroker;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Kafka measured against a real broker: an in-process KRaft broker from spring-kafka-test, one
* node, one topic, three partitions.
*
* <p>The three questions are the same for all three brokers: what ordering survives, what can be
* replayed, and how far consumers scale.
*/
@Tag("kafka")
class KafkaComparisonTest {
private static final String ORDERING = "orders-ordering";
private static final String REPLAY = "orders-replay";
private static final String SCALING = "orders-scaling";
private static EmbeddedKafkaKraftBroker broker;
@BeforeAll
static void startBroker() {
broker = new EmbeddedKafkaKraftBroker(1, 3, ORDERING, REPLAY, SCALING);
broker.afterPropertiesSet();
}
@AfterAll
static void stopBroker() {
broker.destroy();
}
// ---------------------------------------------------------------- ordering
@Test
void orderingIsPerPartitionOnly() {
produce(ORDERING, 12);
List<ConsumerRecord<String, String>> received = consumeAll(ORDERING, "ordering-group", 12);
Map<String, List<String>> perKey = new LinkedHashMap<>();
Map<String, Set<Integer>> partitionsPerKey = new LinkedHashMap<>();
List<String> globalOrder = new ArrayList<>();
for (ConsumerRecord<String, String> record : received) {
perKey.computeIfAbsent(record.key(), (k) -> new ArrayList<>()).add(record.value());
partitionsPerKey.computeIfAbsent(record.key(), (k) -> new java.util.TreeSet<>())
.add(record.partition());
globalOrder.add(record.key() + "=" + record.value() + "@p" + record.partition());
}
// Per key, the sequence numbers must come back in the order they were produced. Compared
// as numbers: as strings, "10" sorts before "2" and the assertion is meaningless.
perKey.forEach((key, values) -> assertThat(values.stream().map(Integer::parseInt).toList())
.isSorted());
// Each key went to exactly one partition, which is why that holds...
partitionsPerKey.forEach((key, partitions) -> assertThat(partitions).hasSize(1));
// ...and the three keys are spread over all three partitions, so the transcript below is
// actually testing something.
assertThat(partitionsPerKey.values().stream().flatMap(Set::stream).distinct().count())
.isEqualTo(3);
List<Integer> asDelivered = globalOrder.stream()
.map((entry) -> Integer.parseInt(entry.split("=")[1].split("@")[0])).toList();
assertThat(asDelivered).isNotEqualTo(
java.util.stream.IntStream.rangeClosed(1, 12).boxed().toList());
StringBuilder body = new StringBuilder();
body.append("produced 12 records, keys D/A/F round-robin, values 1..12 in order\n")
.append("(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C\n")
.append(" all hash to partition 1 with three partitions, which is worth knowing\n")
.append(" before you decide your keys are well distributed.)\n\n");
body.append("consumed in this order:\n ").append(String.join("\n ", globalOrder))
.append("\n\nper key:\n");
perKey.forEach((key, values) -> body.append(" ").append(key).append(" -> partition ")
.append(partitionsPerKey.get(key)).append(" values ").append(values).append('\n'));
body.append("\nOrder is preserved within each key because a key hashes to one partition.\n")
.append("Across keys it is not: the values above are not 1..12 in order, because a\n")
.append("consumer drains one partition's buffer before the next.\n");
Capture.write("kafka-ordering.txt", "Kafka: ordering is per partition", body.toString());
}
// ---------------------------------------------------------------- replay
@Test
void anythingStillWithinRetentionCanBeReadAgain() {
produce(REPLAY, 12);
int first = consumeAll(REPLAY, "replay-group-1", 12).size();
int again = consumeAll(REPLAY, "replay-group-2", 12).size();
List<ConsumerRecord<String, String>> third;
try (KafkaConsumer<String, String> consumer = consumer("replay-group-1")) {
consumer.subscribe(List.of(REPLAY));
consumer.poll(Duration.ofSeconds(2));
consumer.seekToBeginning(consumer.assignment());
third = drain(consumer, 12);
}
assertThat(first).isEqualTo(12);
assertThat(again).isEqualTo(12);
assertThat(third).hasSize(12);
Capture.write("kafka-replay.txt", "Kafka: the log is the storage",
"""
group replay-group-1, first read : %d records
group replay-group-2, brand new group : %d records
group replay-group-1, after seekToBeginning: %d records
Consuming does not remove anything. A consumer group is a cursor over a log
that the broker keeps until retention expires, so a new group, a reset
offset or a seek all read the same records again.
""".formatted(first, again, third.size()));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void consumerParallelismIsCappedByPartitionCount() throws Exception {
produce(SCALING, 12);
List<KafkaConsumer<String, String>> consumers = new ArrayList<>();
Map<String, Set<Integer>> assignment = new LinkedHashMap<>();
try {
for (int i = 1; i <= 5; i++) {
KafkaConsumer<String, String> consumer = consumer("scaling-group");
consumer.subscribe(List.of(SCALING));
consumers.add(consumer);
}
// Poll each consumer until the group has settled and every member knows its share.
for (int round = 0; round < 8; round++) {
for (KafkaConsumer<String, String> consumer : consumers) {
consumer.poll(Duration.ofMillis(500));
}
}
for (int i = 0; i < consumers.size(); i++) {
Set<Integer> partitions = new java.util.TreeSet<>();
for (TopicPartition tp : consumers.get(i).assignment()) {
partitions.add(tp.partition());
}
assignment.put("consumer-" + (i + 1), partitions);
}
}
finally {
consumers.forEach(KafkaConsumer::close);
}
long idle = assignment.values().stream().filter(Set::isEmpty).count();
assertThat(assignment).hasSize(5);
assertThat(idle).isEqualTo(2);
StringBuilder body = new StringBuilder("topic 'orders-scaling', 3 partitions, 5 consumers in one group\n\n");
assignment.forEach((name, partitions) -> body.append(" ").append(name).append(" -> ")
.append(partitions.isEmpty() ? "no partitions (idle)" : "partitions " + partitions)
.append('\n'));
body.append("\nconsumers with no partitions: ").append(idle).append('\n')
.append("\nA partition is assigned to at most one consumer in a group, so the number\n")
.append("of partitions is a hard ceiling on consumer parallelism. Adding consumers\n")
.append("beyond it adds idle processes, not throughput.\n");
Capture.write("kafka-consumer-scaling.txt",
"Kafka: partitions are the ceiling on consumer parallelism", body.toString());
}
// ---------------------------------------------------------------- helpers
private void produce(String topic, int count) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// D, A and F, not A, B and C: with three partitions murmur2 maps A, B and C all to
// partition 1, so the obvious choice of keys would have produced a transcript in which
// global order happened to be preserved and proved nothing. D->0, A->1, F->2.
String[] keys = { "D", "A", "F" };
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
for (int i = 1; i <= count; i++) {
producer.send(new ProducerRecord<>(topic, keys[i % keys.length], String.valueOf(i)));
}
producer.flush();
}
}
private KafkaConsumer<String, String> consumer(String group) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ConsumerConfig.GROUP_ID_CONFIG, group);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
return new KafkaConsumer<>(props);
}
private List<ConsumerRecord<String, String>> consumeAll(String topic, String group, int expected) {
try (KafkaConsumer<String, String> consumer = consumer(group)) {
consumer.subscribe(List.of(topic));
return drain(consumer, expected);
}
}
private List<ConsumerRecord<String, String>> drain(KafkaConsumer<String, String> consumer, int expected) {
List<ConsumerRecord<String, String>> received = new ArrayList<>();
for (int i = 0; i < 20 && received.size() < expected; i++) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
records.forEach(received::add);
}
return received;
}
}
broker-comparison/src/test/java/com/ankurm/brokers/PulsarComparisonTest.java
-298
View File
@@ -1,298 +0,0 @@
package com.ankurm.brokers;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
import org.apache.pulsar.client.api.SubscriptionType;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Pulsar measured against a real standalone broker on localhost:6650, started by
* scripts/pulsar-broker.sh.
*
* <p>Pulsar is the interesting third option because it does not force the trade the other two
* force. The subscription type is chosen per subscriber rather than being a property of the
* topic, so ordering and consumer fan-out are separate decisions.
*/
@Tag("pulsar")
class PulsarComparisonTest {
private static PulsarClient client;
@BeforeAll
static void connect() throws Exception {
client = PulsarClient.builder().serviceUrl("pulsar://127.0.0.1:6650").build();
}
@AfterAll
static void close() throws Exception {
client.close();
}
// ---------------------------------------------------------------- ordering
@Test
void sharedSpreadsAKeyAndKeySharedPinsIt() throws Exception {
String topic = "persistent://public/default/ordering-" + System.nanoTime();
produce(topic, 12);
Map<String, Set<String>> shared = consumersPerKey(topic, "sub-shared", SubscriptionType.Shared, 12);
Map<String, Set<String>> keyShared = consumersPerKey(topic, "sub-keyshared",
SubscriptionType.Key_Shared, 12);
// Key_Shared: every key was handled by exactly one consumer, so per-key order is safe.
keyShared.forEach((key, handlers) -> assertThat(handlers).hasSize(1));
// Shared: at least one key was handled by both, so two messages with that key were in
// flight on two consumers and nothing orders them.
assertThat(shared.values().stream().anyMatch((handlers) -> handlers.size() > 1)).isTrue();
Capture.write("pulsar-ordering.txt", "Pulsar: the subscription type decides",
"""
12 messages, keys A/B/C, values 1..12 in order, two consumers per
subscription, receiverQueueSize 1 so the first consumer cannot take the
whole backlog.
Shared subscription, which consumers saw each key:
%s
Key_Shared subscription, which consumers saw each key:
%s
A Shared subscription round-robins individual messages, so messages with
the same key end up on different consumers and can be processed at the
same time: there is no per-key order left to speak of. Key_Shared hashes
the key to one consumer, which is Kafka's guarantee -- except that the
assignment belongs to the subscription and is recomputed as consumers come
and go, rather than being fixed by a partition count chosen when the topic
was created.
""".formatted(render3(shared), render3(keyShared)));
}
// ---------------------------------------------------------------- replay
@Test
void aSubscriptionCanBeRewound() throws Exception {
String topic = "persistent://public/default/replay-" + System.nanoTime();
produce(topic, 12);
int first;
int afterSeek;
try (Consumer<String> consumer = consumer(topic, "replay-sub", SubscriptionType.Exclusive)) {
first = drain(consumer, 12).size();
consumer.seek(MessageId.earliest);
afterSeek = drain(consumer, 12).size();
}
int newSubscription;
try (Consumer<String> consumer = consumer(topic, "replay-sub-2", SubscriptionType.Exclusive)) {
newSubscription = drain(consumer, 12).size();
}
assertThat(first).isEqualTo(12);
assertThat(afterSeek).isEqualTo(12);
assertThat(newSubscription).isEqualTo(12);
Capture.write("pulsar-replay.txt", "Pulsar: acknowledged, but still there",
"""
subscription replay-sub, first read : %d messages
subscription replay-sub, after seek(earliest): %d messages
subscription replay-sub-2, brand new : %d messages
Acknowledgement moves a cursor; the message itself lives in the managed
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
which Kafka can also do by resetting offsets. What differs is the default:
Pulsar deletes a message once every subscription has acknowledged it,
unless a retention policy on the namespace says otherwise, whereas Kafka
keeps it for the retention period regardless of who read it. A Pulsar
topic with no retention policy and no subscriptions keeps nothing.
""".formatted(first, afterSeek, newSubscription));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void sharedSubscriptionsHaveNoPartitionCeilingButHaveAReceiverQueue() throws Exception {
String topicA = "persistent://public/default/scaling-default-" + System.nanoTime();
produce(topicA, 40);
Map<String, Integer> withDefaultQueue = drainAcross(topicA, "scaling-sub", 5, 40, 0);
String topicB = "persistent://public/default/scaling-q1-" + System.nanoTime();
produce(topicB, 40);
Map<String, Integer> withQueueOfOne = drainAcross(topicB, "scaling-sub", 5, 40, 1);
long idleDefault = withDefaultQueue.values().stream().filter((n) -> n == 0).count();
long idleTuned = withQueueOfOne.values().stream().filter((n) -> n == 0).count();
assertThat(idleTuned).isZero();
assertThat(idleDefault).isGreaterThan(0);
Capture.write("pulsar-consumer-scaling.txt",
"Pulsar: no partition ceiling, but the receiver queue decides who gets the work",
"""
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
receiverQueueSize left at the default (1000):
%s consumers that received nothing : %d
receiverQueueSize(1):
%s consumers that received nothing : %d
The topic has no partitions and five consumers can still share the work --
in Kafka the same shape needs at least five partitions, chosen when the
topic was created. But the default receiver queue is 1000 messages, so the
first consumers to connect pull the whole 40-message backlog into their own
buffers before the rest ask for anything, and the subscription looks
broken. Which consumers win is a race and varies between runs -- one
consumer taking all forty, or two taking twenty each -- but the consumers
that lose it see nothing at all. This is the same trap as RabbitMQ's
unbounded prefetch, with a different name and a much larger default.
""".formatted(render2(withDefaultQueue), idleDefault,
render2(withQueueOfOne), idleTuned));
}
// ---------------------------------------------------------------- helpers
private void produce(String topic, int count) throws Exception {
String[] keys = { "A", "B", "C" };
try (Producer<String> producer = client.newProducer(Schema.STRING).topic(topic).create()) {
for (int i = 1; i <= count; i++) {
producer.newMessage().key(keys[i % keys.length]).value(String.valueOf(i)).send();
}
}
}
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type)
throws Exception {
return consumer(topic, subscription, type, 0);
}
/**
* @param receiverQueueSize 0 leaves Pulsar's default of 1000 in place; anything else sets it.
* The default matters more than it looks: it is how many messages a single consumer will pull
* into its own buffer before another consumer on the same subscription gets a chance.
*/
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type,
int receiverQueueSize) throws Exception {
var builder = client.newConsumer(Schema.STRING)
.topic(topic)
.subscriptionName(subscription)
.subscriptionType(type)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
if (receiverQueueSize > 0) {
builder = builder.receiverQueueSize(receiverQueueSize);
}
return builder.subscribe();
}
private List<String> drain(Consumer<String> consumer, int expected) throws Exception {
List<String> received = new ArrayList<>();
while (received.size() < expected) {
Message<String> message = consumer.receive(2, TimeUnit.SECONDS);
if (message == null) {
break;
}
received.add(message.getValue());
consumer.acknowledge(message);
}
return received;
}
/** Which consumers saw each key, with a receiver queue of one so nothing hoards. */
private Map<String, Set<String>> consumersPerKey(String topic, String subscription,
SubscriptionType type, int expected) throws Exception {
Map<String, Set<String>> perKey = new LinkedHashMap<>();
List<Consumer<String>> consumers = new ArrayList<>();
try {
consumers.add(consumer(topic, subscription, type, 1));
consumers.add(consumer(topic, subscription, type, 1));
int received = 0;
while (received < expected) {
boolean any = false;
for (int i = 0; i < consumers.size(); i++) {
Message<String> message = consumers.get(i).receive(500, TimeUnit.MILLISECONDS);
if (message != null) {
perKey.computeIfAbsent(message.getKey(), (k) -> new java.util.TreeSet<>())
.add("consumer-" + (i + 1));
consumers.get(i).acknowledge(message);
received++;
any = true;
}
}
if (!any) {
break;
}
}
}
finally {
for (Consumer<String> consumer : consumers) {
consumer.close();
}
}
return perKey;
}
/** Drains a Shared subscription across {@code n} consumers and reports the distribution. */
private Map<String, Integer> drainAcross(String topic, String subscription, int n, int expected,
int receiverQueueSize) throws Exception {
Map<String, Integer> counts = new LinkedHashMap<>();
List<Consumer<String>> consumers = new ArrayList<>();
try {
for (int i = 1; i <= n; i++) {
consumers.add(consumer(topic, subscription, SubscriptionType.Shared, receiverQueueSize));
counts.put("consumer-" + i, 0);
}
int received = 0;
while (received < expected) {
boolean any = false;
for (int i = 0; i < consumers.size(); i++) {
Message<String> message = consumers.get(i).receive(300, TimeUnit.MILLISECONDS);
if (message != null) {
consumers.get(i).acknowledge(message);
counts.merge("consumer-" + (i + 1), 1, Integer::sum);
received++;
any = true;
}
}
if (!any) {
break;
}
}
}
finally {
for (Consumer<String> consumer : consumers) {
consumer.close();
}
}
return counts;
}
private static String render3(Map<String, Set<String>> perKey) {
StringBuilder out = new StringBuilder();
perKey.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append('\n'));
return out.toString();
}
private static String render2(Map<String, Integer> counts) {
StringBuilder out = new StringBuilder();
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append(" messages\n"));
return out.toString();
}
}
broker-comparison/src/test/java/com/ankurm/brokers/RabbitComparisonTest.java
-286
View File
@@ -1,286 +0,0 @@
package com.ankurm.brokers;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DefaultConsumer;
import com.rabbitmq.client.Envelope;
import com.rabbitmq.client.GetResponse;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/**
* RabbitMQ measured against a real broker on localhost:5672, started by
* scripts/rabbit-broker.sh.
*
* <p>The same three questions as the Kafka measurements, against a broker that owns the message
* until it is acknowledged rather than a log that keeps it.
*/
@Tag("rabbit")
class RabbitComparisonTest {
private static Connection connection;
@BeforeAll
static void connect() throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setPort(5672);
connection = factory.newConnection();
}
@AfterAll
static void close() throws Exception {
connection.close();
}
// ---------------------------------------------------------------- ordering
@Test
void oneConsumerIsFifoAndTwoConsumersAreNot() throws Exception {
String single = declare("order-single");
publish(single, 12);
List<String> fifo = getAll(single, 12);
String shared = declare("order-shared");
publish(shared, 12);
List<String> completionOrder = consumeWithTwoConsumers(shared, 12);
assertThat(fifo).containsExactly("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12");
int inversions = 0;
for (int i = 1; i < completionOrder.size(); i++) {
if (Integer.parseInt(completionOrder.get(i)) < Integer.parseInt(completionOrder.get(i - 1))) {
inversions++;
}
}
Capture.write("rabbit-ordering.txt", "RabbitMQ: FIFO per queue, per consumer",
"""
one queue, one consumer, 12 messages
completion order : %s
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
completion order : %s
out-of-order steps: %d
A queue is FIFO and a single consumer sees it that way. The moment a second
consumer is added the broker hands the next message to whichever consumer
is free, so the order in which work finishes is no longer the order in
which it was published. There is no key: RabbitMQ has no notion of a
partition to which related messages could be pinned. Ordering across
related messages means one queue and one consumer, and therefore no
horizontal scaling for that queue -- or a consistent-hash exchange, which
is a plugin.
""".formatted(fifo, completionOrder, inversions));
}
// ---------------------------------------------------------------- replay
@Test
void anAcknowledgedMessageIsGone() throws Exception {
String queue = declare("replay-probe");
publish(queue, 12);
List<String> first = getAll(queue, 12);
List<String> second = getAll(queue, 12);
int depth = depth(queue);
assertThat(first).hasSize(12);
assertThat(second).isEmpty();
assertThat(depth).isZero();
Capture.write("rabbit-replay.txt", "RabbitMQ: there is nothing to replay",
"""
first drain of the queue : %d messages
second drain of the queue : %d messages
queue depth afterwards : %d
Acknowledging a message deletes it. The broker is a router with buffers,
not a log: there is no offset to rewind and no second reader that can see
what the first one consumed. Reading the same message twice means
arranging it in advance -- a second queue bound to the same exchange, or a
copy written somewhere else -- and it cannot be arranged after the fact.
""".formatted(first.size(), second.size(), depth));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void consumersScaleWithoutAPartitionCeilingButPrefetchDecidesWho() throws Exception {
String unlimited = declare("scaling-unlimited");
publish(unlimited, 40);
Map<String, Integer> withDefaultPrefetch = drainAcross(unlimited, 5, 40, 0);
String throttled = declare("scaling-prefetch-1");
publish(throttled, 40);
Map<String, Integer> withPrefetchOne = drainAcross(throttled, 5, 40, 1);
long idleDefault = withDefaultPrefetch.values().stream().filter((n) -> n == 0).count();
long idleTuned = withPrefetchOne.values().stream().filter((n) -> n == 0).count();
assertThat(idleTuned).isZero();
Capture.write("rabbit-consumer-scaling.txt",
"RabbitMQ: consumers scale, and prefetch decides whether they actually do",
"""
one queue, 40 messages, 5 consumers, each taking 10 ms per message
no basicQos at all (unlimited prefetch, the AMQP default):
%s consumers that received nothing : %d
basicQos(1):
%s consumers that received nothing : %d
Every consumer on a queue competes for the same messages, so adding
consumers adds throughput and there is no structural ceiling of the kind
Kafka's partition count imposes. But with the AMQP default the broker
pushes as many messages as a consumer will take, so whichever consumer
connects first can be handed the entire backlog while the others sit idle.
basicQos is not a tuning knob you get to postpone; it is what makes the
fan-out real. Spring AMQP sets it for you --
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
better than unlimited and still large enough to concentrate a small
backlog on one consumer.
The price of all this is the ordering measurement: there is no key, so
nothing constrains related messages to one consumer.
""".formatted(render(withDefaultPrefetch), idleDefault,
render(withPrefetchOne), idleTuned));
}
// ---------------------------------------------------------------- helpers
private String declare(String name) throws Exception {
try (Channel channel = connection.createChannel()) {
channel.queueDelete(name);
channel.queueDeclare(name, false, false, true, null);
}
return name;
}
private void publish(String queue, int count) throws Exception {
try (Channel channel = connection.createChannel()) {
for (int i = 1; i <= count; i++) {
channel.basicPublish("", queue, null, String.valueOf(i).getBytes(StandardCharsets.UTF_8));
}
}
}
private List<String> getAll(String queue, int max) throws Exception {
List<String> received = new ArrayList<>();
try (Channel channel = connection.createChannel()) {
for (int i = 0; i < max; i++) {
GetResponse response = channel.basicGet(queue, true);
if (response == null) {
break;
}
received.add(new String(response.getBody(), StandardCharsets.UTF_8));
}
}
return received;
}
private int depth(String queue) throws Exception {
try (Channel channel = connection.createChannel()) {
return channel.queueDeclarePassive(queue).getMessageCount();
}
}
private List<String> consumeWithTwoConsumers(String queue, int count) throws Exception {
List<String> completion = new CopyOnWriteArrayList<>();
CountDownLatch done = new CountDownLatch(count);
List<Channel> channels = new ArrayList<>();
for (int i = 1; i <= 2; i++) {
long delay = (i == 1) ? 60 : 5;
Channel channel = connection.createChannel();
channel.basicQos(1);
channels.add(channel);
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
@Override
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
byte[] body) throws IOException {
try {
Thread.sleep(delay);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
completion.add(new String(body, StandardCharsets.UTF_8));
getChannel().basicAck(envelope.getDeliveryTag(), false);
done.countDown();
}
});
}
done.await(30, TimeUnit.SECONDS);
for (Channel channel : channels) {
channel.close();
}
return List.copyOf(completion);
}
/** Drains a queue across {@code n} competing consumers. prefetch 0 means "do not set it". */
private Map<String, Integer> drainAcross(String queue, int n, int expected, int prefetch)
throws Exception {
Map<String, AtomicInteger> perConsumer = new ConcurrentHashMap<>();
CountDownLatch done = new CountDownLatch(expected);
List<Channel> channels = new ArrayList<>();
for (int i = 1; i <= n; i++) {
String name = "consumer-" + i;
Channel channel = connection.createChannel();
if (prefetch > 0) {
channel.basicQos(prefetch);
}
channels.add(channel);
perConsumer.put(name, new AtomicInteger());
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
@Override
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
byte[] body) throws IOException {
perConsumer.get(name).incrementAndGet();
try {
Thread.sleep(10);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
getChannel().basicAck(envelope.getDeliveryTag(), false);
done.countDown();
}
});
}
done.await(30, TimeUnit.SECONDS);
for (Channel channel : channels) {
channel.close();
}
Map<String, Integer> counts = new LinkedHashMap<>();
perConsumer.forEach((name, count) -> counts.put(name, count.get()));
return counts;
}
private static String render(Map<String, Integer> counts) {
StringBuilder out = new StringBuilder();
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append(" messages\n"));
return out.toString();
}
}
kafka-streams/README.md
-44
View File
@@ -1,44 +0,0 @@
# kafka-streams
Companion code for *Kafka Streams with Spring Boot 4: Real-Time Fraud Detection with Vector
Similarity* on [ankurm.com](https://ankurm.com). The whole topology runs under
`TopologyTestDriver`, so **no Kafka broker is needed** to run anything here.
| Stack | Version |
|---|---|
| JDK | 25 (Temurin 25.0.4.1+1) |
| Spring Boot | 4.1.1 |
| Spring for Apache Kafka | 4.1.1 |
| kafka-streams / kafka-streams-test-utils | 4.2.1 (what Boot 4.1.1 manages) |
## Run it
```bash
./scripts/run-all.sh # runs the tests, rewrites output/*.txt
```
## What is where
| File | Role |
|---|---|
| `FraudTopology` | The topology as plain Kafka Streams code: similarity scoring, 5-minute velocity count, blocklist KTable join |
| `Embedder` | A hand-built 6-number feature vector and cosine similarity (not a learned model) |
| `FraudStreamsConfig` | The only Spring wiring: `@EnableKafkaStreams` plus handing the `StreamsBuilder` to the topology |
| `FraudTopologyTest` | Five tests; each one writes the matching `output/NN-*.txt` |
| `ContextLoadsTest` | Loads the Spring context with `auto-startup=false` to show the wiring without a broker |
## Captured output
Every file in `output/` is written by a test that also asserts it.
| File | Shows |
|---|---|
| `01-similarity-scores.txt` | Scores for five sample transactions against the 0.98 threshold |
| `02-similarity-alerts.txt` | Only the fraud-shaped transaction produces an alert |
| `03-velocity-window.txt` | Six transactions in two minutes give one alert; the next window starts from one |
| `04-blocklist-table-join.txt` | Join before blocking, after blocking, and after a tombstone |
| `05-topology-description.txt` | `Topology.describe()` of the whole graph |
| `06-spring-context.txt` | The Spring factory bean is wired and not running |
| `07-ambiguous-null-tombstone.txt` | The real `javac` error for `pipeInput("c5", null)`; source in `src/broken/`, regenerated by `scripts/compile-broken.sh` |
The article carries the explanations; there is deliberately no `docs/` folder.
kafka-streams/output/01-similarity-scores.txt
-6
View File
@@ -1,6 +0,0 @@
c1,4700.0,3,crypto score=1.0000
c1,3000.0,2,electronics score=1.0000
c1,45.0,18,grocery score=-0.1185
c1,120.0,13,electronics score=0.1387
c1,2500.0,14,crypto score=0.2136
threshold: 0.98
kafka-streams/output/02-similarity-alerts.txt
-2
View File
@@ -1,2 +0,0 @@
alerts <- c1 SIMILAR:1.0000:c1,4700.0,3,crypto
alerts emitted: 1
kafka-streams/output/03-velocity-window.txt
-5
View File
@@ -1,5 +0,0 @@
alerts <- c9 VELOCITY:4 in 5m
alerts emitted for 6 transactions in 2 minutes: 1
alerts after one more transaction 10 minutes later: 0
state store window 2026-10-14T10:00:00Z count=6
state store window 2026-10-14T10:10:00Z count=1
kafka-streams/output/04-blocklist-table-join.txt
-3
View File
@@ -1,3 +0,0 @@
before blocking, alerts: 0
alerts <- c5 BLOCKED:reported stolen
after the tombstone, alerts: 0
kafka-streams/output/05-topology-description.txt
-39
View File
@@ -1,39 +0,0 @@
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [transactions])
--> KSTREAM-AGGREGATE-0000000005, KSTREAM-JOIN-0000000013, KSTREAM-MAPVALUES-0000000001
Processor: KSTREAM-AGGREGATE-0000000005 (stores: [txn-count-5m])
--> KTABLE-TOSTREAM-0000000006
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAPVALUES-0000000001 (stores: [])
--> score
<-- KSTREAM-SOURCE-0000000000
Processor: KTABLE-TOSTREAM-0000000006 (stores: [])
--> KSTREAM-FILTER-0000000007
<-- KSTREAM-AGGREGATE-0000000005
Processor: KSTREAM-FILTER-0000000007 (stores: [])
--> KSTREAM-MAP-0000000008
<-- KTABLE-TOSTREAM-0000000006
Processor: score (stores: [])
--> KSTREAM-FILTER-0000000003
<-- KSTREAM-MAPVALUES-0000000001
Processor: KSTREAM-FILTER-0000000003 (stores: [])
--> KSTREAM-SINK-0000000004
<-- score
Processor: KSTREAM-JOIN-0000000013 (stores: [blocked-cards-STATE-STORE-0000000010])
--> KSTREAM-SINK-0000000014
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MAP-0000000008 (stores: [])
--> KSTREAM-SINK-0000000009
<-- KSTREAM-FILTER-0000000007
Source: KSTREAM-SOURCE-0000000011 (topics: [blocked-cards])
--> KTABLE-SOURCE-0000000012
Sink: KSTREAM-SINK-0000000004 (topic: alerts)
<-- KSTREAM-FILTER-0000000003
Sink: KSTREAM-SINK-0000000009 (topic: alerts)
<-- KSTREAM-MAP-0000000008
Sink: KSTREAM-SINK-0000000014 (topic: alerts)
<-- KSTREAM-JOIN-0000000013
Processor: KTABLE-SOURCE-0000000012 (stores: [blocked-cards-STATE-STORE-0000000010])
--> none
<-- KSTREAM-SOURCE-0000000011
kafka-streams/output/06-spring-context.txt
-3
View File
@@ -1,3 +0,0 @@
StreamsBuilderFactoryBean present: true
auto-startup: false
running: false
kafka-streams/output/07-ambiguous-null-tombstone.txt
-8
View File
@@ -1,8 +0,0 @@
src/broken/AmbiguousNullTombstone.java:9: error: reference to pipeInput is ambiguous
blocked.pipeInput("c5", null);
^
both method pipeInput(K,V) in TestInputTopic and method pipeInput(V,Instant) in TestInputTopic match
where K,V are type-variables:
K extends Object declared in class TestInputTopic
V extends Object declared in class TestInputTopic
1 error
kafka-streams/pom.xml
-56
View File
@@ -1,56 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<!-- Same parent as the other modules, so spring-kafka and kafka-streams versions are Boot-managed. -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>kafka-streams-fraud</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams-test-utils</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
kafka-streams/scripts/compile-broken.sh
-9
View File
@@ -1,9 +0,0 @@
#!/usr/bin/env bash
# Captures the real javac error for the null-tombstone mistake into output/07-ambiguous-null-tombstone.txt
set -uo pipefail
cd "$(dirname "$0")/.."
mvn -B -q dependency:build-classpath -Dmdep.outputFile=target/cp.txt >/dev/null
mkdir -p target/broken
javac -d target/broken -cp "$(cat target/cp.txt)" src/broken/AmbiguousNullTombstone.java 2>&1 \
| grep -v '^Picked up' > output/07-ambiguous-null-tombstone.txt
cat output/07-ambiguous-null-tombstone.txt
kafka-streams/scripts/run-all.sh
-8
View File
@@ -1,8 +0,0 @@
#!/usr/bin/env bash
# Regenerates every file under output/ by running the test suite. No broker needed.
set -euo pipefail
cd "$(dirname "$0")/.."
rm -f output/*.txt
mvn -B -q test
scripts/compile-broken.sh >/dev/null || true
ls output
kafka-streams/src/broken/AmbiguousNullTombstone.java
-11
View File
@@ -1,11 +0,0 @@
import org.apache.kafka.streams.TestInputTopic;
/**
* Deliberately does not compile; it lives outside src/main and src/test so the build ignores it.
* scripts/compile-broken.sh runs javac on it and captures the compiler's own message.
*/
public class AmbiguousNullTombstone {
static void tombstone(TestInputTopic<String, String> blocked) {
blocked.pipeInput("c5", null);
}
}
kafka-streams/src/main/java/com/ankurm/streams/Embedder.java
-51
View File
@@ -1,51 +0,0 @@
package com.ankurm.streams;
import java.util.List;
/**
* Turns a transaction into a vector and scores it by cosine similarity to known-fraud vectors.
*
* <p>This is a hand-built feature vector, not a learned embedding: [log amount, sin(hour),
* cos(hour), is-crypto, is-electronics, is-grocery]. The topology around it would be identical if
* {@link #embed} called an embedding model; only this one method would change.
*/
public final class Embedder {
private Embedder() {}
/** Prototypes of confirmed fraud: large, late at night, in risky categories. */
static final List<double[]> FRAUD_PROTOTYPES = List.of(
embed(new Txn("proto", 4800, 3, "crypto")),
embed(new Txn("proto", 2900, 2, "electronics")));
public static double[] embed(Txn t) {
double angle = 2 * Math.PI * t.hour() / 24.0;
return new double[] {
Math.log10(Math.max(1, t.amount())) / 5.0,
Math.sin(angle),
Math.cos(angle),
"crypto".equals(t.category()) ? 1 : 0,
"electronics".equals(t.category()) ? 1 : 0,
"grocery".equals(t.category()) ? 1 : 0 };
}
public static double cosine(double[] a, double[] b) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return (na == 0 || nb == 0) ? 0 : dot / (Math.sqrt(na) * Math.sqrt(nb));
}
/** Highest similarity to any fraud prototype, in [-1, 1]. */
public static double fraudScore(Txn t) {
double[] v = embed(t);
double best = -1;
for (double[] p : FRAUD_PROTOTYPES) {
best = Math.max(best, cosine(v, p));
}
return best;
}
}
kafka-streams/src/main/java/com/ankurm/streams/FraudStreamsConfig.java
-20
View File
@@ -1,20 +0,0 @@
package com.ankurm.streams;
import org.apache.kafka.streams.StreamsBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafkaStreams;
/**
* The Spring side is only this: {@code @EnableKafkaStreams} supplies a {@link StreamsBuilder}
* bean, and this method hands it to the plain-Java topology before the streams instance starts.
*/
@Configuration
@EnableKafkaStreams
public class FraudStreamsConfig {
@Autowired
public void wire(StreamsBuilder builder) {
FraudTopology.wire(builder);
}
}
kafka-streams/src/main/java/com/ankurm/streams/FraudTopology.java
-64
View File
@@ -1,64 +0,0 @@
package com.ankurm.streams;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.kstream.Grouped;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.kstream.Produced;
import org.apache.kafka.streams.kstream.TimeWindows;
import java.time.Duration;
/**
* The topology, as plain Kafka Streams code with no Spring in it. That is deliberate: a topology
* that only needs a {@link StreamsBuilder} can be tested with TopologyTestDriver in milliseconds,
* with no broker.
*
* <pre>
* transactions --+--> mapValues(score) --filter(score >= 0.98)--------> alerts "SIMILAR:..."
* |
* +--> groupByKey --> 5-minute tumbling count (store txn-count-5m)
* | --filter(count == 4)------------------------> alerts "VELOCITY:..."
* |
* +--> join(blocked-cards KTable)-----------------------> alerts "BLOCKED:..."
* </pre>
*/
public final class FraudTopology {
public static final String IN = "transactions";
public static final String BLOCKED = "blocked-cards";
public static final String OUT = "alerts";
public static final String COUNT_STORE = "txn-count-5m";
public static final double SIMILARITY_THRESHOLD = 0.98;
public static final long VELOCITY_LIMIT = 4;
private FraudTopology() {}
public static void wire(StreamsBuilder b) {
KStream<String, String> raw = b.stream(IN, Consumed.with(Serdes.String(), Serdes.String()));
// 1. Similarity: score every transaction, keep the ones close to a known-fraud vector.
raw.mapValues(v -> Txn.decode(v))
.mapValues(t -> "SIMILAR:" + String.format("%.4f", Embedder.fraudScore(t)) + ":" + t.encode(),
org.apache.kafka.streams.kstream.Named.as("score"))
.filter((k, v) -> Double.parseDouble(v.split(":")[1]) >= SIMILARITY_THRESHOLD)
.to(OUT, Produced.with(Serdes.String(), Serdes.String()));
// 2. Velocity: count per card in 5-minute tumbling windows, alert on exactly the 4th.
raw.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count(Materialized.as(COUNT_STORE))
.toStream()
.filter((w, c) -> c == VELOCITY_LIMIT)
.map((w, c) -> org.apache.kafka.streams.KeyValue.pair(w.key(), "VELOCITY:" + c + " in 5m"))
.to(OUT, Produced.with(Serdes.String(), Serdes.String()));
// 3. Blocklist: a KTable of blocked cards, joined to each transaction as it arrives.
KTable<String, String> blocked = b.table(BLOCKED, Consumed.with(Serdes.String(), Serdes.String()));
raw.join(blocked, (txn, reason) -> "BLOCKED:" + reason)
.to(OUT, Produced.with(Serdes.String(), Serdes.String()));
}
}
kafka-streams/src/main/java/com/ankurm/streams/StreamsApplication.java
-11
View File
@@ -1,11 +0,0 @@
package com.ankurm.streams;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class StreamsApplication {
public static void main(String[] args) {
SpringApplication.run(StreamsApplication.class, args);
}
}
kafka-streams/src/main/java/com/ankurm/streams/Txn.java
-15
View File
@@ -1,15 +0,0 @@
package com.ankurm.streams;
/** A card transaction. */
public record Txn(String card, double amount, int hour, String category) {
/** Wire format: {@code card,amount,hour,category}. Plain text keeps the serde trivial to read. */
public String encode() {
return card + "," + amount + "," + hour + "," + category;
}
public static Txn decode(String s) {
String[] p = s.split(",", 4);
return new Txn(p[0], Double.parseDouble(p[1]), Integer.parseInt(p[2]), p[3]);
}
}
kafka-streams/src/main/resources/application.properties
-4
View File
@@ -1,4 +0,0 @@
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.streams.application-id=fraud-streams
# Off by default here so the context can load without a broker; set true when you have one.
spring.kafka.streams.auto-startup=false
kafka-streams/src/test/java/com/ankurm/streams/ContextLoadsTest.java
-27
View File
@@ -1,27 +0,0 @@
package com.ankurm.streams;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.context.ApplicationContext;
import org.springframework.kafka.config.StreamsBuilderFactoryBean;
import static org.junit.jupiter.api.Assertions.*;
/** Proves the Spring wiring: the factory bean exists and holds a topology, without a broker. */
@SpringBootTest
class ContextLoadsTest {
@Autowired ApplicationContext ctx;
@Test
void streamsFactoryBeanIsWiredAndNotStarted() throws Exception {
var t = new Transcript("06-spring-context.txt");
var fb = ctx.getBean(StreamsBuilderFactoryBean.class);
t.line("StreamsBuilderFactoryBean present: " + (fb != null));
t.line("auto-startup: " + fb.isAutoStartup());
t.line("running: " + fb.isRunning());
t.write();
assertFalse(fb.isRunning());
}
}
kafka-streams/src/test/java/com/ankurm/streams/FraudTopologyTest.java
-140
View File
@@ -1,140 +0,0 @@
package com.ankurm.streams;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.state.WindowStore;
import org.apache.kafka.streams.state.KeyValueIterator;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Properties;
import static org.junit.jupiter.api.Assertions.*;
class FraudTopologyTest {
private TopologyTestDriver driver;
private TestInputTopic<String, String> txns;
private TestInputTopic<String, String> blocked;
private TestOutputTopic<String, String> alerts;
private Topology topology;
@BeforeEach
void setUp() {
StreamsBuilder b = new StreamsBuilder();
FraudTopology.wire(b);
topology = b.build();
Properties p = new Properties();
p.put(StreamsConfig.APPLICATION_ID_CONFIG, "fraud-test");
p.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1");
driver = new TopologyTestDriver(topology, p, Instant.parse("2026-10-14T10:00:00Z"));
txns = driver.createInputTopic(FraudTopology.IN, new StringSerializer(), new StringSerializer());
blocked = driver.createInputTopic(FraudTopology.BLOCKED, new StringSerializer(), new StringSerializer());
alerts = driver.createOutputTopic(FraudTopology.OUT, new StringDeserializer(), new StringDeserializer());
}
@AfterEach
void tearDown() { driver.close(); }
@Test
void similarityScoresSeparateFraudShapedFromOrdinary() throws Exception {
var t = new Transcript("01-similarity-scores.txt");
var samples = List.of(
new Txn("c1", 4700, 3, "crypto"),
new Txn("c1", 3000, 2, "electronics"),
new Txn("c1", 45, 18, "grocery"),
new Txn("c1", 120, 13, "electronics"),
new Txn("c1", 2500, 14, "crypto"));
for (Txn x : samples) {
t.line(String.format("%-40s score=%.4f", x.encode(), Embedder.fraudScore(x)));
}
t.line("threshold: " + FraudTopology.SIMILARITY_THRESHOLD);
t.write();
assertTrue(Embedder.fraudScore(samples.get(0)) >= FraudTopology.SIMILARITY_THRESHOLD);
assertTrue(Embedder.fraudScore(samples.get(1)) >= FraudTopology.SIMILARITY_THRESHOLD);
assertTrue(Embedder.fraudScore(samples.get(2)) < FraudTopology.SIMILARITY_THRESHOLD);
}
@Test
void similarTransactionsRaiseAlertsOrdinaryOnesDoNot() throws Exception {
var t = new Transcript("02-similarity-alerts.txt");
txns.pipeInput("c1", new Txn("c1", 4700, 3, "crypto").encode());
txns.pipeInput("c2", new Txn("c2", 45, 18, "grocery").encode());
var out = alerts.readKeyValuesToList();
out.forEach(kv -> t.line("alerts <- " + kv.key + " " + kv.value));
t.line("alerts emitted: " + out.size());
t.write();
assertEquals(1, out.size());
assertEquals("c1", out.get(0).key);
assertTrue(out.get(0).value.startsWith("SIMILAR:"));
}
@Test
void velocityAlertFiresOnTheFourthTransactionInAFiveMinuteWindowOnly() throws Exception {
var t = new Transcript("03-velocity-window.txt");
Instant start = Instant.parse("2026-10-14T10:00:30Z");
for (int i = 0; i < 6; i++) {
txns.pipeInput("c9", new Txn("c9", 20, 12, "grocery").encode(), start.plusSeconds(i * 20L));
}
var out = alerts.readKeyValuesToList();
out.forEach(kv -> t.line("alerts <- " + kv.key + " " + kv.value));
t.line("alerts emitted for 6 transactions in 2 minutes: " + out.size());
// The same count in the next window starts again from one.
txns.pipeInput("c9", new Txn("c9", 20, 12, "grocery").encode(), start.plusSeconds(600));
t.line("alerts after one more transaction 10 minutes later: " + alerts.readKeyValuesToList().size());
WindowStore<String, Long> store = driver.getWindowStore(FraudTopology.COUNT_STORE);
try (KeyValueIterator<org.apache.kafka.streams.kstream.Windowed<String>, Long> it =
store.fetchAll(Instant.parse("2026-10-14T09:00:00Z"), Instant.parse("2026-10-14T11:00:00Z"))) {
while (it.hasNext()) {
var kv = it.next();
t.line("state store window " + kv.key.window().startTime() + " count=" + kv.value);
}
}
t.write();
assertEquals(1, out.size());
assertEquals("VELOCITY:4 in 5m", out.get(0).value);
}
@Test
void blocklistTableJoinsEachTransactionAsItArrives() throws Exception {
var t = new Transcript("04-blocklist-table-join.txt");
txns.pipeInput("c5", new Txn("c5", 15, 12, "grocery").encode());
t.line("before blocking, alerts: " + alerts.readKeyValuesToList().size());
blocked.pipeInput("c5", "reported stolen");
txns.pipeInput("c5", new Txn("c5", 15, 12, "grocery").encode());
var out = alerts.readKeyValuesToList();
out.forEach(kv -> t.line("alerts <- " + kv.key + " " + kv.value));
blocked.pipeInput("c5", (String) null); // tombstone removes the card from the table
txns.pipeInput("c5", new Txn("c5", 15, 12, "grocery").encode());
t.line("after the tombstone, alerts: " + alerts.readKeyValuesToList().size());
t.write();
assertEquals(1, out.size());
assertEquals("BLOCKED:reported stolen", out.get(0).value);
}
@Test
void describeTheTopology() throws Exception {
var t = new Transcript("05-topology-description.txt");
t.line(topology.describe().toString().trim());
t.write();
assertTrue(topology.describe().toString().contains(FraudTopology.COUNT_STORE));
}
}
kafka-streams/src/test/java/com/ankurm/streams/Transcript.java
-23
View File
@@ -1,23 +0,0 @@
package com.ankurm.streams;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
/** Collects lines and writes them to output/NN-name.txt, so the test IS the transcript. */
final class Transcript {
private final List<String> lines = new ArrayList<>();
private final String file;
Transcript(String file) { this.file = file; }
void line(String s) { lines.add(s); }
void write() throws IOException {
Path dir = Path.of("output");
Files.createDirectories(dir);
Files.write(dir.resolve(file), lines);
}
}
kafka4-migration/.gitignore
-1
View File
@@ -1 +0,0 @@
target/
kafka4-migration/README.md
-13
View File
@@ -1,13 +0,0 @@
# kafka4-migration: Spring Kafka 3 to 4 on Spring Boot 3.5 to 4.1
Companion code for the ankurm.com post "Spring Kafka 3 to 4 Migration Guide".
| Folder | What it is |
|---|---|
| `legacy/` | Producer, `@KafkaListener` and an `@EmbeddedKafka` test on Boot 3.5.16 / Spring Kafka 3.3.16 / kafka-clients 3.9.2 (JDK 21), using the Jackson 2 `JsonSerializer` |
| `migrated/` | The same app on Boot 4.1.1 / Spring Kafka 4.1.1 / kafka-clients 4.2.1 (JDK 25), using `spring-boot-starter-kafka` and `JacksonJsonSerializer` |
| `scripts/run-all.sh` | Builds the intermediate stages from `legacy/` and regenerates every file in `output/` |
| `output/` | Transcripts for each stage |
Spring Kafka 4.1.1 is the latest GA; 4.2.0-M1 and 4.2.0-M2 are milestones and are not tested here.
No external broker is needed: the tests use `@EmbeddedKafka`. Tested on 28 September 2026.
kafka4-migration/legacy/pom.xml
-21
View File
@@ -1,21 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.5.16</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>kafka-legacy</artifactId>
<version>1.0.0</version>
<properties><java.version>21</java.version></properties>
<dependencies>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter</artifactId></dependency>
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-json</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency>
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka-test</artifactId><scope>test</scope></dependency>
</dependencies>
</project>
kafka4-migration/legacy/src/main/java/com/ankurm/kafka/App.java
-9
View File
@@ -1,9 +0,0 @@
package com.ankurm.kafka;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class App {
public static void main(String[] args) { SpringApplication.run(App.class, args); }
}
kafka4-migration/legacy/src/main/java/com/ankurm/kafka/OrderEvent.java
-3
View File
@@ -1,3 +0,0 @@
package com.ankurm.kafka;
public record OrderEvent(String id, int qty) { }
kafka4-migration/legacy/src/main/java/com/ankurm/kafka/OrderListener.java
-18
View File
@@ -1,18 +0,0 @@
package com.ankurm.kafka;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.List;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;
@Component
public class OrderListener {
public final List<String> received = new CopyOnWriteArrayList<>();
@KafkaListener(topics = "orders")
void on(OrderEvent event, @Header(KafkaHeaders.RECEIVED_KEY) String key) {
received.add("key=" + key + " id=" + event.id() + " qty=" + event.qty());
}
}
kafka4-migration/legacy/src/main/java/com/ankurm/kafka/SerdeProbe.java
-14
View File
@@ -1,14 +0,0 @@
package com.ankurm.kafka;
import java.nio.charset.StandardCharsets;
import org.springframework.kafka.support.serializer.JsonSerializer;
/** Serialises one event with the JSON serializer, using only the runtime classpath (no test jars). */
public class SerdeProbe {
public static void main(String[] args) {
try (JsonSerializer<OrderEvent> s = new JsonSerializer<>()) {
byte[] bytes = s.serialize("orders", new OrderEvent("o-1", 3));
System.out.println("SERIALIZED " + new String(bytes, StandardCharsets.UTF_8));
}
}
}
kafka4-migration/legacy/src/main/resources/application.properties
-8
View File
@@ -1,8 +0,0 @@
spring.kafka.consumer.group-id=orders-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=com.ankurm.kafka
spring.kafka.consumer.properties.spring.json.value.default.type=com.ankurm.kafka.OrderEvent
kafka4-migration/legacy/src/test/java/com/ankurm/kafka/OrderFlowTest.java
-27
View File
@@ -1,27 +0,0 @@
package com.ankurm.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
@SpringBootTest(properties = "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}")
@EmbeddedKafka(topics = "orders", partitions = 1)
class OrderFlowTest {
@Autowired KafkaTemplate<String, OrderEvent> template;
@Autowired OrderListener listener;
@Autowired EmbeddedKafkaBroker broker;
@Test
void sendAndReceive() {
template.send("orders", "k1", new OrderEvent("o-1", 3));
await().untilAsserted(() -> assertThat(listener.received).hasSize(1));
System.out.println("RESULT " + listener.received.get(0));
System.out.println("BROKERS " + broker.getBrokersAsString().replaceAll(":\\d+", ":PORT"));
}
}
kafka4-migration/migrated/pom.xml
-21
View File
@@ -1,21 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>kafka-migrated</artifactId>
<version>1.0.0</version>
<properties><java.version>25</java.version></properties>
<dependencies>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-kafka</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-jackson</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency>
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka-test</artifactId><scope>test</scope></dependency>
</dependencies>
</project>
kafka4-migration/migrated/src/main/java/com/ankurm/kafka/App.java
-9
View File
@@ -1,9 +0,0 @@
package com.ankurm.kafka;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class App {
public static void main(String[] args) { SpringApplication.run(App.class, args); }
}
kafka4-migration/migrated/src/main/java/com/ankurm/kafka/OrderEvent.java
-3
View File
@@ -1,3 +0,0 @@
package com.ankurm.kafka;
public record OrderEvent(String id, int qty) { }
kafka4-migration/migrated/src/main/java/com/ankurm/kafka/OrderListener.java
-18
View File
@@ -1,18 +0,0 @@
package com.ankurm.kafka;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.List;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;
@Component
public class OrderListener {
public final List<String> received = new CopyOnWriteArrayList<>();
@KafkaListener(topics = "orders")
void on(OrderEvent event, @Header(KafkaHeaders.RECEIVED_KEY) String key) {
received.add("key=" + key + " id=" + event.id() + " qty=" + event.qty());
}
}
kafka4-migration/migrated/src/main/java/com/ankurm/kafka/SerdeProbe.java
-14
View File
@@ -1,14 +0,0 @@
package com.ankurm.kafka;
import java.nio.charset.StandardCharsets;
import org.springframework.kafka.support.serializer.JacksonJsonSerializer;
/** Serialises one event with the JSON serializer, using only the runtime classpath (no test jars). */
public class SerdeProbe {
public static void main(String[] args) {
try (JacksonJsonSerializer<OrderEvent> s = new JacksonJsonSerializer<>()) {
byte[] bytes = s.serialize("orders", new OrderEvent("o-1", 3));
System.out.println("SERIALIZED " + new String(bytes, StandardCharsets.UTF_8));
}
}
}
kafka4-migration/migrated/src/main/resources/application.properties
-8
View File
@@ -1,8 +0,0 @@
spring.kafka.consumer.group-id=orders-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JacksonJsonSerializer
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JacksonJsonDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=com.ankurm.kafka
spring.kafka.consumer.properties.spring.json.value.default.type=com.ankurm.kafka.OrderEvent
kafka4-migration/migrated/src/test/java/com/ankurm/kafka/OrderFlowTest.java
-27
View File
@@ -1,27 +0,0 @@
package com.ankurm.kafka;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
@SpringBootTest(properties = "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}")
@EmbeddedKafka(topics = "orders", partitions = 1)
class OrderFlowTest {
@Autowired KafkaTemplate<String, OrderEvent> template;
@Autowired OrderListener listener;
@Autowired EmbeddedKafkaBroker broker;
@Test
void sendAndReceive() {
template.send("orders", "k1", new OrderEvent("o-1", 3));
await().untilAsserted(() -> assertThat(listener.received).hasSize(1));
System.out.println("RESULT " + listener.received.get(0));
System.out.println("BROKERS " + broker.getBrokersAsString().replaceAll(":\\d+", ":PORT"));
}
}
kafka4-migration/output/01-legacy-kafka3.txt
-12
View File
@@ -1,12 +0,0 @@
$ mvn test # Boot 3.5.16, JDK 21
RESULT key=k1 id=o-1 qty=3
BROKERS 127.0.0.1:PORT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.853 s -- in com.ankurm.kafka.OrderFlowTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
org.apache.kafka:kafka-clients:jar:3.9.2:compile
org.springframework.kafka:spring-kafka:jar:3.3.16:compile
$ java -cp target/classes:<runtime classpath> SerdeProbe
SERIALIZED {"id":"o-1","qty":3}
kafka4-migration/output/02-boot4-no-starter.txt
-5
View File
@@ -1,5 +0,0 @@
$ mvn test # only the Boot version changed to 4.1.1
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.728 s <<< FAILURE! -- in com.ankurm.kafka.OrderFlowTest
[ERROR] OrderFlowTest.sendAndReceive ? UnsatisfiedDependency Error creating bean with name 'com.ankurm.kafka.OrderFlowTest': Unsatisfied dependency expressed through field 'template': No qualifying bean of type 'org.springframework.kafka.core.KafkaTemplate<java.lang.String, com.ankurm.kafka.OrderEvent>' available: expected at
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] BUILD FAILURE
kafka4-migration/output/03-starter-added-tests-pass.txt
-15
View File
@@ -1,15 +0,0 @@
$ mvn test # spring-kafka replaced by spring-boot-starter-kafka; nothing else changed
RESULT key=k1 id=o-1 qty=3
BROKERS localhost:PORT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.853 s -- in com.ankurm.kafka.OrderFlowTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
org.apache.kafka:kafka-clients:jar:4.2.1:compile
org.springframework.boot:spring-boot-kafka:jar:4.1.1:compile
org.springframework.kafka:spring-kafka:jar:4.1.1:compile
$ mvn dependency:list | grep jackson # runtime scope, then test scope
runtime: tools.jackson.core:jackson-core:jar:3.1.5:compile
runtime: tools.jackson.core:jackson-databind:jar:3.1.5:compile
test-only: com.fasterxml.jackson.core:jackson-databind:jar:2.21.5:test
kafka4-migration/output/04-runtime-classpath-has-no-jackson2.txt
-3
View File
@@ -1,3 +0,0 @@
$ compile and run SerdeProbe on the runtime classpath only (JsonSerializer from Spring Kafka 4.1.1)
[ERROR] class file for com.fasterxml.jackson.core.type.TypeReference not found
[ERROR] s2/src/main/java/com/ankurm/kafka/SerdeProbe.java:[9,45] cannot access com.fasterxml.jackson.core.type.TypeReference
kafka4-migration/output/05-migrated.txt
-13
View File
@@ -1,13 +0,0 @@
$ mvn test # final migrated project (JacksonJsonSerializer, spring-boot-starter-jackson)
RESULT key=k1 id=o-1 qty=3
BROKERS localhost:PORT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.478 s -- in com.ankurm.kafka.OrderFlowTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
org.apache.kafka:kafka-clients:jar:4.2.1:compile
org.springframework.boot:spring-boot-kafka:jar:4.1.1:compile
org.springframework.kafka:spring-kafka:jar:4.1.1:compile
$ java -cp target/classes:<runtime classpath> SerdeProbe
SERIALIZED {"id":"o-1","qty":3}
kafka4-migration/output/06-serializer-classes.txt
-2
View File
@@ -1,2 +0,0 @@
$ jar tf spring-kafka-4.1.1.jar | grep support/serializer/(Json|Jackson)
JacksonJsonDeserializer.class JacksonJsonSerde.class JacksonJsonSerializer.class JacksonJsonTypeResolver.class JsonDeserializer.class JsonSerde.class JsonSerializer.class JsonTypeResolver.class
kafka4-migration/scripts/run-all.sh
-44
View File
@@ -1,44 +0,0 @@
#!/usr/bin/env bash
# Regenerates output/*.txt. Needs Maven, JDK 21 (J21), JDK 25 (J25) and Maven Central access.
set -u
HERE=$(cd "$(dirname "$0")/.." && pwd); OUT=$HERE/output; W=${WORK:-/tmp/k4-work}
J21=${J21:-/usr/lib/jvm/java-21-openjdk-amd64}; J25=${J25:-/tmp/tools/jdk-25.0.4.1+1}; MVN=${MVN:-mvn}
mkdir -p "$OUT"; rm -rf "$W"; mkdir -p "$W"
mv_() { local jh=$1 d=$2; shift 2; (cd "$d" && JAVA_HOME=$jh $MVN -B "$@" 2>&1 | grep -v JAVA_TOOL_OPTIONS | sed "s#$W/##g"); }
keep='^(RESULT|BROKERS)|Tests run:.*Fail|BUILD|ERROR.*(cannot access|No qualifying)'
stage() { rm -rf "$W/$1"; cp -r "$HERE/$2" "$W/$1"; rm -rf "$W/$1/target"; }
probe() { # probe <javahome> <dir>: compile and run SerdeProbe on the RUNTIME classpath only
local jh=$1 d=$2
(cd "$d" && JAVA_HOME=$jh $MVN -B -q compile dependency:build-classpath -Dmdep.includeScope=runtime -Dmdep.outputFile="$W/cp.txt" 2>&1 | grep -v JAVA_TOOL_OPTIONS | grep -E 'ERROR.*(cannot access|class file)' | sed "s#$W/##g" | sort -u
[ -f "$W/cp.txt" ] && JAVA_HOME=$jh $jh/bin/java -cp target/classes:$(cat "$W/cp.txt") com.ankurm.kafka.SerdeProbe 2>&1 | grep -v JAVA_TOOL_OPTIONS | head -3); rm -f "$W/cp.txt"; }
versions() { (cd "$2" && JAVA_HOME=$1 $MVN -B dependency:list 2>&1 | grep -E 'org.springframework.kafka:spring-kafka:|org.springframework.boot:spring-boot-kafka:|org.apache.kafka:kafka-clients:jar:[0-9]' | sed 's/^\[INFO\] *//; s/ -- module.*//' | sort -u); }
stage s0 legacy
{ echo '$ mvn test # Boot 3.5.16, JDK 21'; mv_ "$J21" "$W/s0" test | grep -E "$keep"
echo; versions "$J21" "$W/s0"
echo; echo '$ java -cp target/classes:<runtime classpath> SerdeProbe'; probe "$J21" "$W/s0"; } > "$OUT/01-legacy-kafka3.txt"
stage s1 legacy; rm "$W/s1/src/main/java/com/ankurm/kafka/SerdeProbe.java"; sed -i 's/3.5.16/4.1.1/; s/<java.version>21/<java.version>25/' "$W/s1/pom.xml"
{ echo '$ mvn test # only the Boot version changed to 4.1.1'; mv_ "$J25" "$W/s1" test | grep -E "$keep" | cut -c1-330; } > "$OUT/02-boot4-no-starter.txt"
cp -r "$W/s1" "$W/s2"; rm -rf "$W/s2/target"
sed -i 's#<groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId>#<groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-kafka</artifactId>#' "$W/s2/pom.xml"
{ echo '$ mvn test # spring-kafka replaced by spring-boot-starter-kafka; nothing else changed'; mv_ "$J25" "$W/s2" test | grep -E "$keep"
echo; versions "$J25" "$W/s2"
echo; echo '$ mvn dependency:list | grep jackson # runtime scope, then test scope'
(cd "$W/s2" && JAVA_HOME=$J25 $MVN -B dependency:list -DincludeScope=runtime 2>&1 | grep -E 'jackson-(core|databind)' | sed 's/^\[INFO\] *//; s/ -- module.*//' | sort -u | sed 's/^/runtime: /'
JAVA_HOME=$J25 $MVN -B dependency:list -DexcludeScope=runtime 2>&1 | grep -E 'com.fasterxml.jackson.core:jackson-databind' | sed 's/^\[INFO\] *//; s/ -- module.*//' | sort -u | sed 's/^/test-only: /'); } > "$OUT/03-starter-added-tests-pass.txt"
cp "$HERE/legacy/src/main/java/com/ankurm/kafka/SerdeProbe.java" "$W/s2/src/main/java/com/ankurm/kafka/"
{ echo '$ compile and run SerdeProbe on the runtime classpath only (JsonSerializer from Spring Kafka 4.1.1)'; probe "$J25" "$W/s2"; } > "$OUT/04-runtime-classpath-has-no-jackson2.txt"
stage m migrated
{ echo '$ mvn test # final migrated project (JacksonJsonSerializer, spring-boot-starter-jackson)'; mv_ "$J25" "$W/m" test | grep -E "$keep"
echo; versions "$J25" "$W/m"
echo; echo '$ java -cp target/classes:<runtime classpath> SerdeProbe'; probe "$J25" "$W/m"; } > "$OUT/05-migrated.txt"
# 06 API facts from the Spring Kafka 4.1.1 jar
J=$HOME/.m2/repository/org/springframework/kafka/spring-kafka/4.1.1/spring-kafka-4.1.1.jar
{ echo '$ jar tf spring-kafka-4.1.1.jar | grep support/serializer/(Json|Jackson)'
env -u JAVA_TOOL_OPTIONS $J25/bin/jar tf "$J" | grep -E 'support/serializer/(Json|Jackson)' | sed 's#.*/##' | sort | tr '\n' ' '; echo; } > "$OUT/06-serializer-classes.txt"
echo done
protocol-comparison/README.md
-91
View File
@@ -1,91 +0,0 @@
# `protocol-comparison` — RSocket vs gRPC vs WebSocket, same use case, measured
Companion project for
[**RSocket vs gRPC vs WebSocket on Spring Boot 4.1: When Each One Wins**](https://ankurm.com/rsocket-vs-grpc-vs-websocket-spring-boot-4-1/)
on ankurm.com.
**One application serves the same two operations over all three protocols.** Same JVM, same heap,
same data generator, same five fields — so a difference in the numbers is a difference in the
transports rather than in three separately-written demos.
[`./scripts/run-all.sh`](scripts/run-all.sh) re-measures everything under
[`docs/output/`](docs/output/).
## Versions
| | Version | Notes |
|---|---|---|
| JDK | 25 (Temurin 25.0.4.1+1) | current LTS |
| Spring Boot | 4.1.1 | latest GA of the 4.1 line |
| Spring Framework | 7.0.9 | Boot-managed |
| grpc-java | **1.83.1** | Boot-managed. Boot 4.1.0 pinned 1.80.0 — it moved |
| protobuf-java | **4.35.1** | Boot-managed. Boot 4.1.0 pinned 4.34.2 |
| rsocket-java | 1.1.5 | Boot-managed |
| Spring gRPC | 1.1.1 | the programming model; separate version from Boot's |
| Jackson | 3.1.5 (`tools.jackson`) | plus `jackson-dataformat-cbor`, RSocket's default |
Read from `repo1.maven.org/.../maven-metadata.xml` and Boot's own `spring-boot-dependencies` POM.
## Quickstart
```bash
./scripts/run-all.sh # six tests, then re-measure docs/output/
mvn test # the same without the capture
mvn spring-boot:run # gRPC 9090, RSocket 7000, WebSocket ws://localhost:8080/quotes
```
Nothing to install: `protoc` and the gRPC codegen plugin resolve as Maven artifacts for your OS
and architecture, and all three servers start inside the test JVM.
## The operations
| | gRPC | RSocket route | WebSocket command |
|---|---|---|---|
| one quote | `QuoteService/GetQuote` | `quote` | `QUOTE <symbol>` |
| N quotes | `QuoteService/StreamQuotes` | `quotes` | `STREAM <symbol> <n> <delayMs>` |
| unbounded | `QuoteService/StreamUnbounded` | `quotes.unbounded` | `UNBOUNDED <symbol>` |
| same, `isReady()`-gated | `QuoteService/StreamUnboundedReady` | — | — |
| server-side produced count | `QuoteService/Produced` | `produced` | `PRODUCED` |
## Documentation
1. [The same use case, three ways](docs/01-three-protocols.md)
2. [How they measure — and why a loopback benchmark flatters the wrong one](docs/02-benchmarks.md)
3. [Payload: the one comparison that needs no benchmark](docs/03-payload.md)
4. [Back-pressure: the measurement that decides it](docs/04-backpressure.md)
5. [Choosing](docs/05-choosing.md)
## Captured output
| File | What it shows |
|---|---|
| [`payload-sizes.txt`](docs/output/payload-sizes.txt) | one message as protobuf, JSON and CBOR, in hex |
| [`request-response.txt`](docs/output/request-response.txt) | 5 000 sequential calls, p50/p99/mean per protocol |
| [`stream-throughput.txt`](docs/output/stream-throughput.txt) | 50 000 messages on one connection |
| [`backpressure.txt`](docs/output/backpressure.txt) | how many the server produced for a client that wanted 100 |
| [`grpc-isready.txt`](docs/output/grpc-isready.txt) | the documented gRPC fix, tested — and it did not work here |
| [`rsocket-codecs.txt`](docs/output/rsocket-codecs.txt) | what Boot configures for RSocket, and why CBOR wins |
| [`tests.txt`](docs/output/tests.txt) | 6 tests |
## Read the benchmark honestly
Client and server are the same process on one loopback interface, two cores. **There is no
network.** That deletes protobuf's size advantage, and it flatters whichever protocol does the
least work per message — which is why raw WebSocket wins both speed benchmarks and why that is
not the recommendation. Treat the ratios as indicative and the absolutes as meaningless off this
box.
The one result that *does* transfer is the back-pressure table, because it is a property of the
protocols rather than of the link.
## Four things this module exists to prove
1. **RSocket's `request(n)` is real and the others have no equivalent.** For a client that took
100 messages then went quiet for two seconds, the servers produced **100**, **1 346 233** and
**311 617** respectively.
2. **`ServerCallStreamObserver.isReady()` did not bound a gRPC producer here.** The documented fix
was tested; the transcript is committed as measured, including its instability.
3. **Raw WebSocket is the fastest and has the worst tail.** p50 of 124 µs against a p99 of
1 448 µs — a 12x spread, where gRPC's is 3.5x.
4. **CBOR keeps the field names, and it wins by list order.** It is binary JSON, not a schema
format: 67 bytes against protobuf's 35 and JSON's 83. Boot puts `JacksonCborEncoder` ahead of
`JacksonJsonEncoder`, and a bare `RSocketStrategies.create()` has neither.
protocol-comparison/docs/01-three-protocols.md
-83
View File
@@ -1,83 +0,0 @@
# 1. The same use case, three ways
Next: [2. How they measure](02-benchmarks.md)
---
One application in this module serves the same two operations over gRPC, RSocket and a raw
WebSocket: fetch one quote, and stream N quotes. Same JVM, same heap, same
[`QuoteSource`](../src/main/java/com/ankurm/protocols/QuoteSource.java), same five fields.
That constraint is the point. Cross-protocol benchmarks published on the internet almost always
compare three different applications, and end up measuring three different serialisation
libraries, three JIT states and three thread pools.
## What each one is, in one sentence
| | gRPC | RSocket | Raw WebSocket |
|---|---|---|---|
| Transport | HTTP/2 | TCP, WebSocket, or others | HTTP/1.1 Upgrade |
| Schema | `.proto`, mandatory, code-generated | none required | none |
| Encoding here | protobuf | CBOR (Spring's default) | JSON |
| Interaction models | 4, declared in the `.proto` | 4, chosen by return type | whatever you invent |
| Demand signalling | none in the API | `request(n)` on the wire | none at all |
| Deadlines | per call, absolute, propagating | none built in | none |
| Browser client | needs grpc-web + a proxy | yes, over the WebSocket transport | yes, natively |
| Boot 4 starters | `spring-boot-starter-grpc-server` / `-client` | `spring-boot-starter-rsocket` | `spring-boot-starter-websocket` |
## The three server sides, side by side
**gRPC** ([`GrpcQuoteService`](../src/main/java/com/ankurm/protocols/GrpcQuoteService.java)) is
generated-class inheritance. There is no registration code: the generated `ImplBase` is a
`BindableService` and Boot registers every such bean.
```java
@Override
public void getQuote(QuoteRequest request, StreamObserver<Quote> observer) {
observer.onNext(toProto(source.at(request.getSymbol(), 0)));
observer.onCompleted();
}
```
**RSocket** ([`RSocketQuoteController`](../src/main/java/com/ankurm/protocols/RSocketQuoteController.java))
is annotation routing, and the interaction model *is the return type*:
```java
@MessageMapping("quote")
public Mono<Quote> quote(String symbol) { ... } // request/response
@MessageMapping("quotes")
public Flux<Quote> quotes(StreamSpec spec) { ... } // request/stream
```
**Raw WebSocket** ([`WebSocketQuoteHandler`](../src/main/java/com/ankurm/protocols/WebSocketQuoteHandler.java))
is a `switch` on a string, because there is nothing else:
```java
switch (parts[0]) {
case "QUOTE" -> send(session, source.at(parts[1], 0));
case "STREAM" -> { for (int i = 0; i < count; i++) { send(...); } }
...
}
```
That third block is the honest summary of what a raw WebSocket gives you: a bidirectional pipe
for text or bytes. Everything above it &mdash; routing, correlation, errors, versioning, demand
&mdash; is yours to invent, and inventing it is how teams end up with a private, undocumented
protocol that only its authors can debug. STOMP exists to stop that, and is covered in
[`../sse-websocket`](../../sse-websocket/README.md).
## Boot 4 version notes
Boot 4.1.1's BOM pins **grpc-java 1.83.1** and **protobuf-java 4.35.1**, both deliberately older
than the newest releases on Maven Central. Overriding them independently is how you get a
`NoSuchMethodError` between grpc-java and its shaded Netty. Note that these have moved since Boot
4.1.0, which pinned 1.80.0 and 4.34.2 &mdash; so a `spring.grpc.*` guide written against 4.1.0 is
already describing different jars.
RSocket is **rsocket-java 1.1.5**, and `spring-boot-starter-rsocket` brings
`jackson-dataformat-cbor` on purpose: CBOR, not JSON, is Spring's default RSocket data mime type.
---
Next: [2. How they measure](02-benchmarks.md)
protocol-comparison/docs/02-benchmarks.md
-74
View File
@@ -1,74 +0,0 @@
# 2. How they measure
Previous: [1. Three protocols](01-three-protocols.md) &middot; Next: [3. Payload](03-payload.md)
---
## Read this before the numbers
Client and server are **the same process on one machine**, talking over loopback: two cores,
3.9&nbsp;GB, JDK 25, Boot 4.1.1. That has three consequences, and ignoring them makes the numbers
say the opposite of the truth.
1. **There is no network.** On a real link, a 35-byte message and an 83-byte message differ by
more than the framing cost, and protobuf's size advantage starts paying. Here it does not.
2. **A protocol that does less looks faster.** WebSocket wins both benchmarks below partly because
it has no flow control, no deadlines, no schema and no per-call metadata. That is a real cost
difference, and it is also exactly what you give up.
3. **Ratios, not absolutes.** Absolute microsecond figures from a two-core sandbox mean nothing
for your hardware. The ordering and the rough factors are what transfer.
Re-measured on every `./scripts/run-all.sh`, and re-measured **in their own JVM invocation**.
That is not tidiness. The back-pressure tests in [chapter 4](04-backpressure.md) leave unbounded
producers spinning, and running the throughput benchmark after them on a two-core box halved
every number — gRPC fell from 44 801 msgs/s to 18 477. If you take one operational lesson from
this module rather than a protocol one, let it be that: a benchmark that shares a machine with
anything is measuring the machine.
## Request/response
5&nbsp;000 sequential calls after 2&nbsp;000 warm-up
([`request-response.txt`](output/request-response.txt)):
```
gRPC n=5000 p50= 291.3 us p99= 1015.3 us mean= 324.4 us ~3,083 calls/s
RSocket n=5000 p50= 300.9 us p99= 1337.4 us mean= 350.9 us ~2,850 calls/s
WebSocket n=5000 p50= 124.3 us p99= 1447.8 us mean= 172.8 us ~5,786 calls/s
```
**gRPC and RSocket are the same speed.** Within noise across runs the two swap places; treat them
as indistinguishable for request/response on a warm connection.
**Raw WebSocket is about twice as fast at the median &mdash; and has the worst tail.** p50 of
124&nbsp;&micro;s against a p99 of 1&nbsp;448&nbsp;&micro;s is a 12x spread. gRPC's is 3.5x. The
median is cheap because a text frame over an already-open socket is nearly free; the tail is bad
because nothing is scheduling anything, so a GC pause or a scheduling hiccup lands on the request
undiluted.
If your SLO is a percentile rather than an average &mdash; and it should be &mdash; that table
does not say what a first glance suggests.
## Server streaming
50&nbsp;000 messages on one connection
([`stream-throughput.txt`](output/stream-throughput.txt)):
```
gRPC 50,000 msgs in 1.116 s = 44,801 msgs/s ( 22.32 us each)
RSocket 50,000 msgs in 0.702 s = 71,175 msgs/s ( 14.05 us each)
WebSocket 50,000 msgs in 0.459 s = 109,022 msgs/s ( 9.17 us each)
```
Same ordering, wider gaps: WebSocket about 2.4x gRPC, RSocket about 1.6x.
Now read the WebSocket row again with its harness in view. The client's inbox is an
`ArrayBlockingQueue` sized to hold **all fifty thousand messages**, because there was nothing else
to do with them &mdash; a raw WebSocket consumer that falls behind can buffer or drop, and those
are the only two options. WebSocket "won" this benchmark by buffering the entire stream in the
client's heap.
That is not a rhetorical point. It is the next chapter.
---
Previous: [1. Three protocols](01-three-protocols.md) &middot; Next: [3. Payload](03-payload.md)
protocol-comparison/docs/03-payload.md
-56
View File
@@ -1,56 +0,0 @@
# 3. Payload: the one comparison that needs no benchmark
Previous: [2. Benchmarks](02-benchmarks.md) &middot; Next: [4. Back-pressure](04-backpressure.md)
---
Five fields &mdash; a symbol, a sequence number, two doubles and a timestamp &mdash; encoded three
ways ([`payload-sizes.txt`](output/payload-sizes.txt)):
```
protobuf : 35 bytes 0a044141504c102a197b14ae47e11a5940215c8fc2f5281c5940288080abb4fef39203
JSON : 83 bytes {"symbol":"AAPL","seq":42,"bid":100.42,"ask":100.44,"epochMicros":1772000000000000}
CBOR : 67 bytes bf6673796d626f6c644141504c63736571182a63626964fb40591ae147ae147b6361736bfb40591c28f5c28f5c6b65706f63684d6963726f731b00064b9fe68ac000ff
JSON is 2.37x protobuf; CBOR is 1.91x protobuf
```
This is a property of the formats, not of a machine, so it is the one row of the comparison that
transfers to your hardware unchanged.
**Protobuf is 35 bytes because field names are integers.** `0a04 41 41 50 4c` is field 1, length
4, `AAPL`. There is no `"symbol"` on the wire at all &mdash; the name lives in the `.proto` file
that both sides compiled against. That is where the size comes from, and it is also the whole
argument about schemas: the saving and the coupling are the same fact.
**CBOR is 67 bytes and still writes the field names.** `6673796d626f6c` is a 6-character text
string, `symbol`. CBOR is binary JSON, not a schema format: it saves the punctuation and encodes
numbers compactly, and it keeps every key. So its advantage over JSON is real but modest, and it
does not require the coupling.
Concretely, on a stream of a million quotes: 35&nbsp;MB, 67&nbsp;MB, 83&nbsp;MB. On a loopback
that difference disappears into memory bandwidth &mdash; which is exactly why the benchmarks in
[chapter 2](02-benchmarks.md) do not reward it and a real link would.
Two practical notes:
- **Spring's RSocket default is CBOR**, not JSON, which is why
`spring-boot-starter-rsocket` brings `jackson-dataformat-cbor`. If you are debugging with
`tcpdump` and expecting to read your payloads, that is why you cannot. The mechanism is codec
ordering, printed by
[`RSocketDefaultsTest`](../src/test/java/com/ankurm/protocols/RSocketDefaultsTest.java):
```
encoders : [CharSequenceEncoder, ByteBufferEncoder, ByteArrayEncoder, DataBufferEncoder, JacksonCborEncoder, JacksonJsonEncoder]
bare RSocketStrategies.create() : [CharSequenceEncoder, ByteBufferEncoder, ByteArrayEncoder, DataBufferEncoder]
```
CBOR sits *ahead of* JSON in the Boot-configured list, which is the whole of why it wins. Note
also that a bare `RSocketStrategies.create()` carries neither — an `RSocketRequester` built
without injecting Boot's strategies cannot encode your objects at all. And the class names
follow the Boot 4 rule: `JacksonCborEncoder`, not `Jackson2…`.
- **You can put protobuf on RSocket.** The encoding and the protocol are independent choices;
the row above is what each stack does *by default*, not what it is capable of.
---
Previous: [2. Benchmarks](02-benchmarks.md) &middot; Next: [4. Back-pressure](04-backpressure.md)
protocol-comparison/docs/04-backpressure.md
-112
View File
@@ -1,112 +0,0 @@
# 4. Back-pressure: the measurement that decides it
Previous: [3. Payload](03-payload.md) &middot; Next: [5. Choosing](05-choosing.md)
---
Every protocol comparison eventually reduces to one question, and it is not throughput. It is:
> **When the consumer stops keeping up, what happens?**
[`BackPressureTest`](../src/test/java/com/ankurm/protocols/BackPressureTest.java) asks it the
simplest way there is. Each server offers an unbounded stream. Each client takes a hundred
messages and then does nothing. How many did the server produce?
The sweep over quiet periods is what makes the answer conclusive. A number that stays flat as the
wait grows means something bounded the producer. A number that keeps climbing means nothing did.
[`backpressure.txt`](output/backpressure.txt):
```
protocol quiet 0.5s quiet 1s quiet 2s
RSocket 100 100 100
gRPC 103,425 449,318 1,208,219
WebSocket 19,664 143,870 320,255
```
RSocket produced **exactly one hundred**, at every quiet period. gRPC produced over a million in
two seconds. Raw WebSocket, three hundred thousand. These are re-measured on every run and the
absolute counts move by tens of percent; the three orders of magnitude between the rows do not.
(The growth is faster than linear because the loop is still being JIT-compiled during the shortest
run. The shape is the finding, not the exponent.)
## Why RSocket stops
`request(n)` is a **frame on the wire**. When the client's subscriber calls `subscription.request(100)`,
a REQUEST_N frame carries that number to the server, and Reactor's `Flux.generate` is called
exactly a hundred times and then not again. Nothing blocks; there is no buffer to size and no
thread parked. The producer simply is not invoked.
That is what "Reactive Streams over the network" means, and it is the entire reason RSocket
exists. The same property is why RSocket's streaming throughput in
[chapter 2](02-benchmarks.md) is respectable rather than the highest: it is doing real work per
batch that the others are not doing at all.
## Why the other two do not
**Raw WebSocket has no mechanism.** The frame protocol has no notion of demand. The server writes
until something underneath refuses &mdash; a socket buffer, Spring's `sendBufferSizeLimit`, the
heap. Every one of those 320&nbsp;255 messages was serialised and written for a client that had
asked for a hundred.
**gRPC has HTTP/2 flow control, and it did not help.** This is the result I did not expect.
`StreamObserver.onNext` on a gRPC server **does not block**, so a plain server-streaming loop keeps
producing regardless of what the client is doing.
The documented answer is `ServerCallStreamObserver.isReady()`, and this module tests it rather than
citing it ([`grpc-isready.txt`](output/grpc-isready.txt)):
```
handler quiet 0.5s quiet 2s
plain onNext loop 258,145 1,421,778
isReady() checked before write 667,259 1,044,018
```
**It did not bound the producer.** The `isReady()` row is not lower than the plain one in any
useful sense &mdash; it produced 667&nbsp;thousand messages in half a second for a client that
wanted a hundred &mdash; and across runs it has come out both far above and far below the plain
loop, which is its own kind of answer. I
have not established whether the surplus accumulates in the server's outbound queue, in the client
transport, or is discarded after the deframer, and I am not going to assert a mechanism I have not
read the source for.
What the numbers do establish is narrower and still worth knowing: **`isReady()` is not a demand
signal**, and on a fast link with a stalled consumer it does not behave like one. If you are
streaming from gRPC to a consumer that can fall behind, the bound has to come from your own
design &mdash; a paged API, an explicit "send me more" message on a bidirectional stream, or a
`count` in the request as `StreamQuotes` uses.
## The honest caveat
A loopback connection is the worst case for this test in the fairest possible way: the wire drains
instantly, so nothing downstream ever pushes back. On a slow real link, TCP and the HTTP/2 window
*would* eventually stop the gRPC and WebSocket producers &mdash; by blocking a thread or by
filling a buffer, but they would stop.
That does not rescue them, because the failure mode you actually meet is not "the network is
slow", it is "one consumer of forty is slow", and a shared server discovers that as latency for
everyone else or as a heap that will not shrink. The difference RSocket makes is that the slow
consumer's slowness is *expressed*, in a number, that the producer can act on before anything
fills up.
## The other thing gRPC does not do for you
Abandoning a gRPC stream produced this, repeatedly, in the server log:
```
java.lang.IllegalStateException: Failed to close the call
Caused by: java.lang.IllegalStateException: call already closed
```
That is the cancellation story from
[the Spring gRPC article](https://ankurm.com/spring-grpc-spring-boot-4/) showing up in practice:
gRPC does not interrupt your thread when a client leaves, it sets a flag on the `Context`, and a
handler that touches an already-closed stream throws. The loop in
[`GrpcQuoteService`](../src/main/java/com/ankurm/protocols/GrpcQuoteService.java) checks
`Context.current().isCancelled()` and returns without calling `onCompleted` for exactly this
reason.
---
Previous: [3. Payload](03-payload.md) &middot; Next: [5. Choosing](05-choosing.md)
protocol-comparison/docs/05-choosing.md
-63
View File
@@ -1,63 +0,0 @@
# 5. Choosing
Previous: [4. Back-pressure](04-backpressure.md)
---
## The decision table
| If | Pick | Because |
|---|---|---|
| Service-to-service RPC, polyglot team, you want a schema | **gRPC** | The `.proto` is a contract other languages generate from. Deadlines are absolute and propagate. Boot 4 makes it a first-class starter. |
| A stream whose consumer can fall behind | **RSocket** | `request(n)` is the only demand signal that crosses the wire. 100 vs 1.3 million is not a tuning difference. |
| A browser is one of the peers | **WebSocket**, with STOMP on it | gRPC needs grpc-web and a proxy. RSocket over WebSocket works but the client ecosystem is thin. |
| Server pushes, client never speaks | **none of these** — use SSE | See [`../sse-websocket`](../../sse-websocket/README.md). |
| You want the lowest possible median latency and control both ends | **raw WebSocket** | It is fastest here because it does the least. Read the p99 before deciding that is what you want. |
| Long-lived bidirectional exchange with independent streams | **RSocket channel** or **gRPC bidi** | Both give you two independent streams; only RSocket gives each of them demand. |
## What each one really costs
**gRPC** costs a build step, a schema you must version, and an operational story for HTTP/2 through
your proxies. In exchange you get the best-supported cross-language RPC there is, and deadlines
&mdash; which is a bigger deal than it sounds, because an absolute deadline that propagates across
hops is the single most effective defence against one slow dependency taking down a system.
**RSocket** costs ecosystem. It is the least deployed of the three, the client libraries outside
Java and JavaScript are thin, and if you are not already reactive, `Mono` and `Flux` in your
service signatures is a real commitment. In exchange you get the only demand signalling in the
list, plus resumption and leasing.
**Raw WebSocket** costs everything you then invent: routing, correlation, error shape, versioning,
and the demand mechanism you will eventually need. That is the cost STOMP exists to prevent, and
if you find yourself designing a text protocol with a command word at the front &mdash; as
[`WebSocketQuoteHandler`](../src/main/java/com/ankurm/protocols/WebSocketQuoteHandler.java)
deliberately does &mdash; that is the signal to stop and use one.
## Should you use any of them?
Three cases where the answer is no:
- **Request/response between two services that both speak HTTP already.** A REST endpoint with a
`RestClient` and a timeout is fine, and everything in your organisation already knows how to
observe, cache, proxy and debug it. Reach for gRPC when the schema or the deadline propagation
is the thing you need, not because it benchmarks faster.
- **Streaming that is really batching.** If the consumer processes a page at a time anyway,
paginated HTTP has back-pressure for free: the consumer asks for the next page when it is ready.
That is `request(n)` with a URL.
- **Events that outlive the connection.** None of these three persist anything. A dropped
connection loses the messages in flight. If that matters, the answer is a broker &mdash;
[`../kafka-basics`](../../kafka-basics/README.md),
[`../rabbitmq`](../../rabbitmq/README.md), and the comparison in
[`../broker-comparison`](../../broker-comparison/README.md).
## And the benchmark caveat, one more time
Every number in this module was measured with client and server in one JVM on one machine. That
deletes the network, which is where protobuf's 35 bytes would pay and where the tail latencies
would look completely different. The measurement that does transfer is
[chapter 4](04-backpressure.md), because "how many did the server produce for a consumer that
wanted a hundred" is a property of the protocol and not of the link.
---
Previous: [4. Back-pressure](04-backpressure.md)
protocol-comparison/docs/output/backpressure.txt
-10
View File
@@ -1,10 +0,0 @@
=== unbounded stream; client takes 100, then goes quiet ===
protocol quiet 0.5s quiet 1s quiet 2s
RSocket 100 100 100
gRPC 103,425 449,318 1,208,219
WebSocket 19,664 143,870 320,255
RSocket is flat: the server produced exactly what request(n) asked for.
gRPC grows 11.7x between 0.5 s and 2 s; WebSocket grows 16.3x.
Growth that tracks the wait, rather than settling at a buffer size,
means nothing bounded the producer. (Growth is faster than linear
because the loop is still being JIT-compiled during the first run.)
protocol-comparison/docs/output/grpc-isready.txt
-4
View File
@@ -1,4 +0,0 @@
=== gRPC server streaming, client takes 100 then stops ===
handler quiet 0.5s quiet 2s
plain onNext loop 258,145 1,421,778
isReady() checked before write 667,259 1,044,018
protocol-comparison/docs/output/payload-sizes.txt
-5
View File
@@ -1,5 +0,0 @@
=== one Quote, five fields, three encodings ===
protobuf : 35 bytes 0a044141504c102a197b14ae47e11a5940215c8fc2f5281c5940288080abb4fef39203
JSON : 83 bytes {"symbol":"AAPL","seq":42,"bid":100.42,"ask":100.44,"epochMicros":1772000000000000}
CBOR : 67 bytes bf6673796d626f6c644141504c63736571182a63626964fb40591ae147ae147b6361736bfb40591c28f5c28f5c6b65706f63684d6963726f731b00064b9fe68ac000ff
JSON is 2.37x protobuf; CBOR is 1.91x protobuf
protocol-comparison/docs/output/request-response.txt
-4
View File
@@ -1,4 +0,0 @@
=== request/response, 5000 sequential calls after 2000 warm-up, loopback, JDK 25.0.4.1 ===
gRPC n=5000 p50= 302.0 us p99= 994.7 us mean= 335.1 us ~2,984 calls/s
RSocket n=5000 p50= 265.9 us p99= 1166.2 us mean= 311.9 us ~3,206 calls/s
WebSocket n=5000 p50= 122.8 us p99= 1893.9 us mean= 184.8 us ~5,410 calls/s
protocol-comparison/docs/output/rsocket-codecs.txt
-4
View File
@@ -1,4 +0,0 @@
=== Boot-configured RSocketStrategies ===
encoders : [CharSequenceEncoder, ByteBufferEncoder, ByteArrayEncoder, DataBufferEncoder, JacksonCborEncoder, JacksonJsonEncoder]
decoders : [StringDecoder, ByteBufferDecoder, ByteArrayDecoder, DataBufferDecoder, JacksonCborDecoder, JacksonJsonDecoder]
bare RSocketStrategies.create() encoders : [CharSequenceEncoder, ByteBufferEncoder, ByteArrayEncoder, DataBufferEncoder]
protocol-comparison/docs/output/stream-throughput.txt
-4
View File
@@ -1,4 +0,0 @@
=== server streaming, 50000 messages on one connection, loopback ===
gRPC 50,000 msgs in 1.341 s = 37,285 msgs/s ( 26.82 us each)
RSocket 50,000 msgs in 0.900 s = 55,580 msgs/s ( 17.99 us each)
WebSocket 50,000 msgs in 0.839 s = 59,630 msgs/s ( 16.77 us each)
protocol-comparison/docs/output/tests.txt
-6
View File
@@ -1,6 +0,0 @@
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.321 s -- in com.ankurm.protocols.PayloadSizeTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.76 s -- in com.ankurm.protocols.RequestResponseBenchmarkTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.29 s -- in com.ankurm.protocols.BackPressureTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.409 s -- in com.ankurm.protocols.GrpcIsReadyTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.474 s -- in com.ankurm.protocols.RSocketDefaultsTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.244 s -- in com.ankurm.protocols.StreamThroughputBenchmarkTest
protocol-comparison/pom.xml
-95
View File
@@ -1,95 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>protocol-comparison</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<!-- One application speaks all three protocols, so the comparison is not confounded by
different JVMs, different heaps or different warm-up states. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webmvc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<!-- RSocket. The starter drags in reactor-netty and jackson-dataformat-cbor; CBOR is not
an accident, it is RSocket's default data mime type in Spring. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-rsocket</artifactId>
</dependency>
<!-- gRPC. Two starters, both Boot-managed. See ankurm.com/spring-grpc-spring-boot-4/ -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-grpc-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-grpc-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jackson</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- protoc and the gRPC codegen plugin resolve as Maven artifacts for this OS and
architecture. Nothing to install locally. Both versions come from the Boot BOM;
overriding them independently is how you get a NoSuchMethodError between grpc-java
and its shaded Netty. -->
<plugin>
<groupId>io.github.ascopes</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>3.9.0</version>
<configuration>
<protocVersion>${protobuf-java.version}</protocVersion>
<binaryMavenPlugins>
<binaryMavenPlugin>
<groupId>io.grpc</groupId>
<artifactId>protoc-gen-grpc-java</artifactId>
<version>${grpc-java.version}</version>
</binaryMavenPlugin>
</binaryMavenPlugins>
</configuration>
<executions>
<execution><goals><goal>generate</goal></goals></execution>
</executions>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
protocol-comparison/scripts/run-all.sh
-40
View File
@@ -1,40 +0,0 @@
#!/usr/bin/env bash
# Regenerate every file under docs/output/. Needs a JDK and nothing else: protoc and the gRPC
# codegen plugin resolve as Maven artifacts, and all three servers start inside the test JVM.
#
# The benchmark numbers are NOT deterministic. They are re-measured on every run and the ratios
# between the three protocols are what the documentation quotes.
set -eu
cd "$(dirname "$0")/.."
LOG=/tmp/protocol-comparison-test.log
# Everything, for the test count and the non-timing transcripts.
MAVEN_OPTS=${MAVEN_OPTS:--Xmx900m} mvn -B test > "$LOG" 2>&1
# The two timing benchmarks are then re-run ALONE. This is not tidiness: the back-pressure and
# isReady tests leave unbounded producers spinning, and on a two-core box that halves every
# throughput number measured after them. Benchmarks share a JVM with nothing.
BENCH=/tmp/protocol-comparison-bench.log
MAVEN_OPTS=${MAVEN_OPTS:--Xmx900m} mvn -B test \
-Dtest=RequestResponseBenchmarkTest,StreamThroughputBenchmarkTest > "$BENCH" 2>&1
# Interleaved log lines from other contexts have to come out, or the transcripts are unreadable.
clean() { grep -vE '^2026-|^\s*$|^WARNING|^[[:space:]]+at |^Caused by|^java\.|^org\.|Initializing|Completed initialization'; }
sed -n '/=== one Quote, five fields/,/^JSON is /p' "$LOG" | clean > docs/output/payload-sizes.txt
sed -n '/=== request\/response/,/^WebSocket /p' "$BENCH" | clean > docs/output/request-response.txt
sed -n '/=== server streaming, 50000/,/^WebSocket /p' "$BENCH" | clean > docs/output/stream-throughput.txt
{ grep -m1 '=== unbounded stream' "$LOG"; grep -m1 '^protocol ' "$LOG"
grep -m1 '^RSocket *[0-9]' "$LOG"; grep -m1 '^gRPC *[0-9]' "$LOG"
grep -m1 '^WebSocket *[0-9]' "$LOG"
sed -n '/^RSocket is flat/,/JIT-compiled/p' "$LOG"; } > docs/output/backpressure.txt
{ grep -m1 '=== gRPC server streaming' "$LOG"; grep -m1 '^handler ' "$LOG"
grep -m1 '^plain onNext loop' "$LOG"; grep -m1 '^isReady() checked' "$LOG"; } \
> docs/output/grpc-isready.txt
sed -n '/=== Boot-configured RSocketStrategies/,/^bare RSocketStrategies/p' "$LOG" | clean \
> docs/output/rsocket-codecs.txt
grep -E 'Tests run:.*in com\.ankurm' "$LOG" | sed 's/^\[INFO\] //' > docs/output/tests.txt
echo "regenerated:"; ls -1 docs/output/
protocol-comparison/src/main/java/com/ankurm/protocols/GrpcQuoteService.java
-121
View File
@@ -1,121 +0,0 @@
package com.ankurm.protocols;
import com.ankurm.protocols.grpc.Quote;
import com.ankurm.protocols.grpc.QuoteRequest;
import com.ankurm.protocols.grpc.QuoteServiceGrpc;
import com.ankurm.protocols.grpc.StreamRequest;
import io.grpc.Context;
import io.grpc.stub.ServerCallStreamObserver;
import io.grpc.stub.StreamObserver;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.stereotype.Service;
/**
* The gRPC side. No registration code: the generated {@code ImplBase} is a
* {@code BindableService} and Boot registers every such bean with the server.
*
* <p>The cancellation check in {@link #streamQuotes} is not decoration. gRPC does not interrupt
* your thread when a client goes away &mdash; it sets a flag on the {@link Context} &mdash; so a
* loop that never looks keeps producing into a stream nobody is reading. That is covered at
* length in <a href="https://ankurm.com/spring-grpc-spring-boot-4/">the Spring gRPC article</a>;
* here it matters because the back-pressure benchmark deliberately abandons a stream.
*
* @see <a href="../../../../../docs/04-backpressure.md">docs/04-backpressure.md</a>
*/
@Service
public class GrpcQuoteService extends QuoteServiceGrpc.QuoteServiceImplBase {
/** Counts what the SERVER produced, comparable with the RSocket and WebSocket counters. */
static final AtomicLong PRODUCED = new AtomicLong();
private final QuoteSource source;
GrpcQuoteService(QuoteSource source) {
this.source = source;
}
@Override
public void getQuote(QuoteRequest request, StreamObserver<Quote> observer) {
observer.onNext(toProto(source.at(request.getSymbol(), 0)));
observer.onCompleted();
}
/**
* An effectively unbounded stream, for the back-pressure comparison. A consumer that stops
* calling {@code next()} closes the HTTP/2 flow-control window, and this loop blocks inside
* {@code onNext} &mdash; so the server stops producing, but it stops by blocking a thread
* rather than by being asked to stop.
*/
@Override
public void streamUnbounded(QuoteRequest request, StreamObserver<Quote> observer) {
PRODUCED.set(0);
while (!Context.current().isCancelled()) {
observer.onNext(toProto(source.at(request.getSymbol(), PRODUCED.get())));
PRODUCED.incrementAndGet();
}
}
/**
* The fix, and the reason the unfixed version is worth measuring.
*
* <p>{@code StreamObserver.onNext} on a gRPC server <strong>never blocks</strong>. If the
* client is not reading, the message is queued in the server's outbound buffer and the loop
* keeps going &mdash; HTTP/2 flow control governs the wire, not your code. The only thing
* that connects the two is {@link ServerCallStreamObserver#isReady()}, and using it means
* restructuring the handler around {@code setOnReadyHandler} rather than writing a loop.
*
* <p>The spin here is deliberately the simplest possible demonstration rather than the
* shape you would ship; a real implementation registers an on-ready handler and returns.
*/
@Override
public void streamUnboundedReady(QuoteRequest request, StreamObserver<Quote> observer) {
PRODUCED.set(0);
ServerCallStreamObserver<Quote> ready = (ServerCallStreamObserver<Quote>) observer;
while (!ready.isCancelled()) {
if (!ready.isReady()) {
Thread.onSpinWait();
continue;
}
ready.onNext(toProto(source.at(request.getSymbol(), PRODUCED.get())));
PRODUCED.incrementAndGet();
}
}
@Override
public void produced(QuoteRequest request, StreamObserver<com.ankurm.protocols.grpc.ProducedCount> observer) {
observer.onNext(com.ankurm.protocols.grpc.ProducedCount.newBuilder()
.setCount(PRODUCED.get()).build());
observer.onCompleted();
}
@Override
public void streamQuotes(StreamRequest request, StreamObserver<Quote> observer) {
for (int i = 0; i < request.getCount(); i++) {
if (Context.current().isCancelled()) {
// Do NOT call onCompleted/onError here: the stream is already closed and
// touching it throws IllegalStateException. Just return.
return;
}
observer.onNext(toProto(source.at(request.getSymbol(), i)));
if (request.getDelayMs() > 0) {
try {
Thread.sleep(request.getDelayMs());
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
return;
}
}
}
observer.onCompleted();
}
static Quote toProto(com.ankurm.protocols.Quote q) {
return Quote.newBuilder()
.setSymbol(q.symbol()).setSeq(q.seq())
.setBid(q.bid()).setAsk(q.ask())
.setEpochMicros(q.epochMicros())
.build();
}
}
protocol-comparison/src/main/java/com/ankurm/protocols/ProtocolComparisonApplication.java
-22
View File
@@ -1,22 +0,0 @@
package com.ankurm.protocols;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* One application, three protocols, one set of data.
*
* <p>gRPC on 9090, RSocket on 7000, WebSocket on the servlet port. Serving the same two
* operations from the same JVM is the only way to make a benchmark mean anything: the heap, the
* JIT state, the CPU and the data generator are shared, so a difference in the numbers is a
* difference in the transports.
*
* @see <a href="../../../../../docs/01-three-protocols.md">docs/01-three-protocols.md</a>
*/
@SpringBootApplication
public class ProtocolComparisonApplication {
public static void main(String[] args) {
SpringApplication.run(ProtocolComparisonApplication.class, args);
}
}
protocol-comparison/src/main/java/com/ankurm/protocols/Quote.java
-8
View File
@@ -1,8 +0,0 @@
package com.ankurm.protocols;
/**
* The JSON/CBOR shape, field-for-field identical to the protobuf {@code Quote} message, so the
* encoded sizes in {@code docs/output/payload-sizes.txt} compare like with like.
*/
public record Quote(String symbol, long seq, double bid, double ask, long epochMicros) {
}
protocol-comparison/src/main/java/com/ankurm/protocols/QuoteSource.java
-22
View File
@@ -1,22 +0,0 @@
package com.ankurm.protocols;
import java.time.Instant;
import org.springframework.stereotype.Component;
/**
* Deterministic quote generation, shared by all three protocol adapters.
*
* <p>Deliberately cheap and deliberately allocation-free apart from the record: a benchmark of
* three transports must not spend its time in the thing behind them.
*/
@Component
public class QuoteSource {
public Quote at(String symbol, long seq) {
double base = 100.0 + (seq % 97) * 0.01;
Instant now = Instant.now();
return new Quote(symbol, seq, base, base + 0.02,
now.getEpochSecond() * 1_000_000L + now.getNano() / 1_000L);
}
}
protocol-comparison/src/main/java/com/ankurm/protocols/RSocketQuoteController.java
-72
View File
@@ -1,72 +0,0 @@
package com.ankurm.protocols;
import java.time.Duration;
import java.util.concurrent.atomic.AtomicLong;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.messaging.handler.annotation.MessageMapping;
import org.springframework.stereotype.Controller;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
/**
* The RSocket side. The same two operations, and one extra that exists only to make
* back-pressure visible.
*
* <p>Note what the signatures say that the other two protocols' cannot: {@code Mono} means
* request/response, {@code Flux} means request/stream, and the choice of interaction model is
* the return type rather than a separate declaration. RSocket has four (fire-and-forget,
* request/response, request/stream, channel) and Spring picks from the method shape.
*
* @see <a href="../../../../../docs/04-backpressure.md">docs/04-backpressure.md</a>
*/
@Controller
public class RSocketQuoteController {
private static final Logger log = LoggerFactory.getLogger(RSocketQuoteController.class);
/** Counts what the SERVER produced, which is the number the back-pressure test cares about. */
static final AtomicLong PRODUCED = new AtomicLong();
private final QuoteSource source;
RSocketQuoteController(QuoteSource source) {
this.source = source;
}
@MessageMapping("quote")
public Mono<Quote> quote(String symbol) {
return Mono.just(source.at(symbol, 0));
}
@MessageMapping("quotes")
public Flux<Quote> quotes(StreamSpec spec) {
Flux<Quote> flux = Flux.range(0, spec.count()).map(i -> source.at(spec.symbol(), i));
return spec.delayMs() > 0 ? flux.delayElements(Duration.ofMillis(spec.delayMs())) : flux;
}
/**
* An unbounded generator. It emits only when the transport asks, so the count it reaches is
* a direct read-out of how much the consumer requested &mdash; which is the whole argument
* for RSocket.
*/
@MessageMapping("quotes.unbounded")
public Flux<Quote> unbounded(String symbol) {
PRODUCED.set(0);
return Flux.generate(() -> 0L, (seq, sink) -> {
sink.next(source.at(symbol, seq));
PRODUCED.incrementAndGet();
return seq + 1;
});
}
@MessageMapping("produced")
public Mono<Long> produced() {
return Mono.just(PRODUCED.get());
}
/** Request payload for {@code quotes}. */
public record StreamSpec(String symbol, int count, int delayMs) {
}
}
protocol-comparison/src/main/java/com/ankurm/protocols/WebSocketConfig.java
-27
View File
@@ -1,27 +0,0 @@
package com.ankurm.protocols;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.socket.config.annotation.EnableWebSocket;
import org.springframework.web.socket.config.annotation.WebSocketConfigurer;
import org.springframework.web.socket.config.annotation.WebSocketHandlerRegistry;
/**
* {@code @EnableWebSocket}, not {@code @EnableWebSocketMessageBroker}: this module wants the raw
* transport, with no broker in the path, so the benchmark measures a socket rather than a routing
* layer. The {@code sse-websocket} module in this repository is the STOMP version.
*/
@Configuration
@EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {
private final WebSocketQuoteHandler handler;
WebSocketConfig(WebSocketQuoteHandler handler) {
this.handler = handler;
}
@Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(handler, "/quotes").setAllowedOriginPatterns("*");
}
}
protocol-comparison/src/main/java/com/ankurm/protocols/WebSocketQuoteHandler.java
-94
View File
@@ -1,94 +0,0 @@
package com.ankurm.protocols;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.stereotype.Component;
import org.springframework.web.socket.CloseStatus;
import org.springframework.web.socket.TextMessage;
import org.springframework.web.socket.WebSocketSession;
import org.springframework.web.socket.handler.TextWebSocketHandler;
import tools.jackson.databind.ObjectMapper;
/**
* The WebSocket side, with no STOMP and no framework on top: a two-word text protocol, so the
* benchmark measures the transport and not a broker.
*
* <pre>
* QUOTE &lt;symbol&gt; -&gt; one JSON quote
* STREAM &lt;symbol&gt; &lt;count&gt; &lt;delayMs&gt; -&gt; count JSON quotes, then "END"
* UNBOUNDED &lt;symbol&gt; -&gt; quotes until the session dies
* </pre>
*
* <p>The absence worth noticing is in {@code UNBOUNDED}: there is nowhere for a consumer to say
* how many it wants. {@code sendMessage} either succeeds, blocks, or eventually throws when
* Spring's send buffer limit is exceeded. That is not a gap in this handler &mdash; it is what a
* raw WebSocket offers.
*
* @see <a href="../../../../../docs/04-backpressure.md">docs/04-backpressure.md</a>
*/
@Component
public class WebSocketQuoteHandler extends TextWebSocketHandler {
/** Counts what the SERVER wrote, comparable with RSocket's PRODUCED. */
static final AtomicLong PRODUCED = new AtomicLong();
private final QuoteSource source;
private final ObjectMapper mapper;
WebSocketQuoteHandler(QuoteSource source, ObjectMapper mapper) {
this.source = source;
this.mapper = mapper;
}
@Override
protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
String[] parts = message.getPayload().trim().split("\\s+");
switch (parts[0]) {
case "QUOTE" -> send(session, source.at(parts[1], 0));
case "STREAM" -> {
int count = Integer.parseInt(parts[2]);
int delay = parts.length > 3 ? Integer.parseInt(parts[3]) : 0;
for (int i = 0; i < count; i++) {
send(session, source.at(parts[1], i));
if (delay > 0) {
Thread.sleep(delay);
}
}
session.sendMessage(new TextMessage("END"));
}
case "UNBOUNDED" -> {
PRODUCED.set(0);
// No requestN. Nothing here can be told to slow down: the loop writes until the
// socket, the send buffer or the send-time limit stops it. Catching the write
// failure keeps an abandoned stream from filling the log with one stack trace
// per run -- and note that catching it is the ONLY notification that arrives.
try {
while (session.isOpen()) {
send(session, source.at(parts[1], PRODUCED.get()));
PRODUCED.incrementAndGet();
}
}
catch (Exception ex) {
// Broken pipe. The client left; there was no other way to find out.
}
}
case "PRODUCED" -> session.sendMessage(new TextMessage(Long.toString(PRODUCED.get())));
default -> session.sendMessage(new TextMessage("ERR unknown command"));
}
}
@Override
public void handleTransportError(WebSocketSession session, Throwable exception) {
// Where an over-buffered slow consumer surfaces: as a transport error on the writer,
// long after the consumer stopped reading.
PRODUCED.addAndGet(0);
}
@Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
}
private void send(WebSocketSession session, Quote quote) throws Exception {
session.sendMessage(new TextMessage(mapper.writeValueAsString(quote)));
}
}
protocol-comparison/src/main/proto/quotes.proto
-45
View File
@@ -1,45 +0,0 @@
syntax = "proto3";
package quotes;
option java_multiple_files = true;
option java_package = "com.ankurm.protocols.grpc";
// The same two operations the RSocket and WebSocket sides expose, so the comparison is between
// transports rather than between designs.
service QuoteService {
// Request/response.
rpc GetQuote (QuoteRequest) returns (Quote);
// Server streaming: count quotes, as fast as the transport allows.
rpc StreamQuotes (StreamRequest) returns (stream Quote);
// Never completes on its own. Used to show what stops a gRPC server producing.
rpc StreamUnbounded (QuoteRequest) returns (stream Quote);
// The same unbounded stream, but the server checks ServerCallStreamObserver.isReady()
// before every write. Same API, different memory profile.
rpc StreamUnboundedReady (QuoteRequest) returns (stream Quote);
// How many messages the server has produced on the current unbounded stream.
rpc Produced (QuoteRequest) returns (ProducedCount);
}
message ProducedCount {
int64 count = 1;
}
message QuoteRequest {
string symbol = 1;
}
message StreamRequest {
string symbol = 1;
int32 count = 2;
// Milliseconds to sleep between emissions; 0 means "as fast as possible".
int32 delay_ms = 3;
}
message Quote {
string symbol = 1;
int64 seq = 2;
double bid = 3;
double ask = 4;
int64 epoch_micros = 5;
}
protocol-comparison/src/main/resources/application.yaml
-22
View File
@@ -1,22 +0,0 @@
spring:
application:
name: protocol-comparison
rsocket:
server:
# RSocket gets its own port and its own Reactor Netty server. TCP here; "websocket" is the
# other transport and is what you use when the client is a browser.
port: 7000
transport: tcp
grpc:
server:
# spring.grpc.server.port -- note the spring. prefix and that "port: -1" means in-process
# only. See the property table in ankurm.com/spring-grpc-spring-boot-4/
port: 9090
server:
port: 8080
logging:
level:
io.grpc: WARN
io.rsocket: WARN
protocol-comparison/src/test/java/com/ankurm/protocols/BackPressureTest.java
-156
View File
@@ -1,156 +0,0 @@
package com.ankurm.protocols;
import java.util.Iterator;
import java.util.concurrent.atomic.AtomicLong;
import com.ankurm.protocols.grpc.QuoteRequest;
import com.ankurm.protocols.grpc.QuoteServiceGrpc;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import org.junit.jupiter.api.Test;
import org.reactivestreams.Subscription;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.server.LocalServerPort;
import org.springframework.messaging.rsocket.RSocketRequester;
import org.springframework.messaging.rsocket.RSocketStrategies;
import reactor.core.publisher.BaseSubscriber;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The measurement that decides the article.
*
* <p>Each protocol serves an unbounded stream. Each client takes a hundred messages and then does
* nothing for a while. The question is: <strong>how many did the server produce?</strong>
*
* <p>The sweep over quiet periods is what makes the answer conclusive. A number that stays flat
* as the quiet period grows means something bounded the producer. A number that grows in
* proportion to the wait means <em>nothing did</em> &mdash; the producer is running flat out and
* the only limit is the clock.
*/
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {"spring.rsocket.server.port=7003", "spring.grpc.server.port=9093"})
class BackPressureTest {
private static final int WANTED = 100;
private static final long[] QUIET_MS = {500, 1_000, 2_000};
@LocalServerPort
int httpPort;
@Autowired
RSocketStrategies strategies;
@Test
void howManyDidTheServerProduce() throws Exception {
System.out.println("=== unbounded stream; client takes " + WANTED + ", then goes quiet ===");
System.out.printf("%-14s %10s %10s %10s%n", "protocol", "quiet 0.5s", "quiet 1s", "quiet 2s");
long[] rs = new long[QUIET_MS.length];
long[] gr = new long[QUIET_MS.length];
long[] ws = new long[QUIET_MS.length];
for (int i = 0; i < QUIET_MS.length; i++) {
rs[i] = rsocket(QUIET_MS[i]);
gr[i] = grpc(QUIET_MS[i]);
ws[i] = websocket(QUIET_MS[i]);
}
row("RSocket", rs);
row("gRPC", gr);
row("WebSocket", ws);
System.out.println();
System.out.println("RSocket is flat: the server produced exactly what request(n) asked for.");
System.out.printf("gRPC grows %.1fx between 0.5 s and 2 s; WebSocket grows %.1fx.%n",
gr[2] / (double) gr[0], ws[2] / (double) ws[0]);
System.out.println("Growth that tracks the wait, rather than settling at a buffer size,");
System.out.println("means nothing bounded the producer. (Growth is faster than linear");
System.out.println("because the loop is still being JIT-compiled during the first run.)");
// RSocket delivered exactly the demand, at every quiet period.
assertThat(rs).containsOnly(WANTED);
// The other two grew with the wait rather than settling at a buffer size.
assertThat(gr[2]).isGreaterThan(gr[0] * 2);
assertThat(ws[2]).isGreaterThan(ws[0] * 2);
}
private void row(String label, long[] v) {
System.out.printf("%-14s %,10d %,10d %,10d%n", label, v[0], v[1], v[2]);
}
/**
* RSocket: {@code request(n)} is a frame on the wire. The server is told the number, produces
* that many, and stops. Nothing blocks and nothing buffers &mdash; the producer is simply not
* called again.
*/
private long rsocket(long quietMs) throws Exception {
RSocketRequester requester = RSocketRequester.builder()
.rsocketStrategies(strategies).tcp("localhost", 7003);
try {
AtomicLong received = new AtomicLong();
BaseSubscriber<Quote> subscriber = new BaseSubscriber<>() {
@Override
protected void hookOnSubscribe(Subscription subscription) {
subscription.request(WANTED); // and never again
}
@Override
protected void hookOnNext(Quote value) {
received.incrementAndGet();
}
};
requester.route("quotes.unbounded").data("AAPL")
.retrieveFlux(Quote.class).subscribe(subscriber);
Thread.sleep(quietMs);
long produced = requester.route("produced").retrieveMono(Long.class).block();
subscriber.dispose();
assertThat(received.get()).isEqualTo(WANTED);
return produced;
}
finally {
requester.dispose();
}
}
/**
* gRPC: there is no application-level demand signal. The blocking stub's {@code Iterator}
* requests one message per {@code next()}, and the handler writes with
* {@code StreamObserver.onNext}, which never blocks.
*/
private long grpc(long quietMs) throws Exception {
ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 9093)
.usePlaintext().build();
try {
var stub = QuoteServiceGrpc.newBlockingStub(channel);
QuoteRequest req = QuoteRequest.newBuilder().setSymbol("AAPL").build();
Iterator<com.ankurm.protocols.grpc.Quote> it = stub.streamUnbounded(req);
long received = 0;
while (received < WANTED && it.hasNext()) {
it.next();
received++;
}
Thread.sleep(quietMs);
return stub.produced(req).getCount();
}
finally {
channel.shutdownNow();
}
}
/** Raw WebSocket: the frame protocol has no notion of demand at all. */
private long websocket(long quietMs) throws Exception {
try (WsClient client = new WsClient(httpPort, WANTED)) {
client.send("UNBOUNDED AAPL");
long received = 0;
while (received < WANTED && client.take(5_000) != null) {
received++;
}
Thread.sleep(quietMs);
// Ask on a second connection: the first one's server-side thread is busy writing.
try (WsClient asker = new WsClient(httpPort, 8)) {
asker.send("PRODUCED");
return Long.parseLong(asker.take(5_000));
}
}
}
}
protocol-comparison/src/test/java/com/ankurm/protocols/Bench.java
-29
View File
@@ -1,29 +0,0 @@
package com.ankurm.protocols;
import java.util.Arrays;
/** Latency bookkeeping, kept in one place so the three protocols are measured identically. */
final class Bench {
private final long[] nanos;
private int i;
Bench(int n) {
this.nanos = new long[n];
}
void record(long ns) {
nanos[i++] = ns;
}
/** p50, p99 and mean in microseconds, plus calls per second derived from the mean. */
String summary(String label) {
long[] sorted = Arrays.copyOf(nanos, i);
Arrays.sort(sorted);
double p50 = sorted[(int) (sorted.length * 0.50)] / 1000.0;
double p99 = sorted[(int) (sorted.length * 0.99)] / 1000.0;
double mean = Arrays.stream(sorted).average().orElse(0) / 1000.0;
return String.format("%-12s n=%-6d p50=%8.1f us p99=%9.1f us mean=%8.1f us ~%,.0f calls/s",
label, sorted.length, p50, p99, mean, 1_000_000.0 / mean);
}
}

Some files were not shown because too many files have changed in this diff Show More