Compare commits
3 Commits
2919282a5c
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| e1f8aa7402 | |||
| f4b0112e2c | |||
| de9fc5ce4c |
14
README.md
14
README.md
@@ -8,6 +8,16 @@ module's `scripts/run-all.sh`, never typed by hand.
|
||||
| Module | Article | What it demonstrates |
|
||||
|---|---|---|
|
||||
| [`kafka-basics/`](kafka-basics/README.md) | [Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/) | The on-ramp: what the starter gives you, the two Jackson serializer families, where a key lands and why, and which defaults are Kafka's rather than Spring's |
|
||||
| [`kafka-error-handling/`](kafka-error-handling/README.md) | [Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/) | What the default error handler really does, why a poison pill stops a partition, and what blocking retries cost that retry topics do not |
|
||||
| [`rabbitmq/`](rabbitmq/README.md) | [Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/) | All four exchange types against a real broker, manual acknowledgement, and a dead-letter path exercised through both rejection and TTL expiry |
|
||||
| [`broker-comparison/`](broker-comparison/README.md) | [Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with Benchmarks](https://ankurm.com/kafka-vs-rabbitmq-vs-pulsar-java-decision-framework/) | The three brokers measured side by side on ordering, replay, consumer scaling and operational footprint, ending in a decision table where every row has a transcript behind it |
|
||||
|
||||
The brokers make an instructive set. Kafka's consumer holds an offset and the broker remembers
|
||||
nothing about individual records; RabbitMQ's broker owns the message until it is acknowledged and
|
||||
can route, expire and dead-letter it on its own; Pulsar keeps the log but lets each subscriber
|
||||
choose how it is read. Almost every difference in how you handle failure follows from those three
|
||||
sentences — which is why Kafka needs a retry topic to do what RabbitMQ does with a queue
|
||||
argument, and why `broker-comparison` is mostly an argument about where a message lives.
|
||||
|
||||
They are meant to be read in order. `kafka-basics` establishes that the default acknowledgement
|
||||
mode is `BATCH` and therefore that delivery is at-least-once; everything the error-handling
|
||||
@@ -21,8 +31,8 @@ Central and from Boot's own `spring-boot-dependencies` POM, rather than from rel
|
||||
announcements.
|
||||
|
||||
Every module runs its broker without Docker, so the transcripts can be regenerated on any
|
||||
machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ via a
|
||||
real broker started by the module's own scripts. Each module also ships a Testcontainers
|
||||
machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ and
|
||||
Pulsar via real brokers started by the modules' own scripts. Each module also ships a Testcontainers
|
||||
configuration for the cases where the deployed image is what you need to test against.
|
||||
|
||||
## Licence
|
||||
|
||||
80
broker-comparison/README.md
Normal file
80
broker-comparison/README.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# broker-comparison
|
||||
|
||||
Companion project for **Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with
|
||||
Benchmarks** on [ankurm.com](https://ankurm.com).
|
||||
|
||||
Three brokers, three questions, one set of measurements. Ordering, replay and consumer scaling are
|
||||
asked identically of all three by driving the client libraries directly, so that what is compared
|
||||
is the brokers' delivery models rather than three sets of Spring defaults. What each Spring
|
||||
integration adds on top is [chapter 6](docs/06-what-spring-adds.md).
|
||||
|
||||
## Verified stack
|
||||
|
||||
| Component | Version | Source of the number |
|
||||
|---|---|---|
|
||||
| JDK (tests) | 25.0.4.1+1 (Temurin) | `java -version` |
|
||||
| JDK (brokers) | 21.0.12.1+1 (Temurin) | `java -version` |
|
||||
| Spring Boot | 4.1.1 | `maven-metadata.xml` on Maven Central |
|
||||
| kafka-clients | 4.2.1 | `spring-boot-dependencies-4.1.1.pom` |
|
||||
| amqp-client | 5.30.0 | `spring-boot-dependencies-4.1.1.pom` |
|
||||
| pulsar-client | 4.2.4 | `spring-boot-dependencies-4.1.1.pom` |
|
||||
| Kafka broker | 4.2.1 (KRaft) | `kafka_2.13-4.2.1.tgz` |
|
||||
| RabbitMQ broker | 3.10.25 | generic-unix tarball |
|
||||
| Pulsar broker | 4.2.4 standalone | `apache-pulsar-4.2.4-bin.tar.gz` |
|
||||
|
||||
RabbitMQ is pinned at 3.10.25 because the box that produced these transcripts has Erlang 24;
|
||||
3.11 and later need Erlang 25. The AMQP 0-9-1 semantics measured here are unchanged in 4.x.
|
||||
|
||||
## Quickstart
|
||||
|
||||
The Kafka measurements need no external broker — `spring-kafka-test` starts a real KRaft broker
|
||||
in-process. The other two need a broker each, and the scripts start both without Docker and
|
||||
without root.
|
||||
|
||||
```bash
|
||||
mvn -Dgroups=kafka test
|
||||
|
||||
ERL_ROOT=... RABBITMQ_HOME=... scripts/rabbit-broker.sh
|
||||
mvn -Dgroups=rabbit test
|
||||
|
||||
PULSAR_HOME=... scripts/pulsar-broker.sh
|
||||
mvn -Dgroups=pulsar test
|
||||
|
||||
scripts/footprint.sh kafka|rabbit|pulsar # the operational numbers
|
||||
scripts/run-all.sh # regenerates every docs/output/ file
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
| Chapter | What it settles |
|
||||
|---|---|
|
||||
| [01 Three models](docs/01-three-models.md) | Where a message lives, and why everything else follows |
|
||||
| [02 Ordering](docs/02-ordering.md) | What survives the second consumer, in all three |
|
||||
| [03 Replay](docs/03-replay.md) | Reading it twice, and the one broker that cannot |
|
||||
| [04 Consumer scaling](docs/04-consumer-scaling.md) | The partition ceiling, and the client buffer that defeats fan-out in the other two |
|
||||
| [05 Operational footprint](docs/05-operational-footprint.md) | Startup, memory, ports, processes, configuration surface |
|
||||
| [06 What Spring adds](docs/06-what-spring-adds.md) | The three integrations, and the Boot 4 starter trap |
|
||||
| [07 The decision table](docs/07-the-decision-table.md) | Every row backed by a measurement, and how to choose |
|
||||
|
||||
## Captured output
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| [`kafka-ordering.txt`](docs/output/kafka-ordering.txt) | Per-key order kept, global order not — and why A, B and C were the wrong keys to demonstrate it |
|
||||
| [`kafka-replay.txt`](docs/output/kafka-replay.txt) | The same 12 records read three times |
|
||||
| [`kafka-consumer-scaling.txt`](docs/output/kafka-consumer-scaling.txt) | 5 consumers, 3 partitions, 2 idle |
|
||||
| [`rabbit-ordering.txt`](docs/output/rabbit-ordering.txt) | Message 1 finishing tenth |
|
||||
| [`rabbit-replay.txt`](docs/output/rabbit-replay.txt) | 12 messages, then 0, then an empty queue |
|
||||
| [`rabbit-consumer-scaling.txt`](docs/output/rabbit-consumer-scaling.txt) | 40/0/0/0/0 without `basicQos`, 8 each with it |
|
||||
| [`pulsar-ordering.txt`](docs/output/pulsar-ordering.txt) | `Shared` spreading a key, `Key_Shared` pinning it |
|
||||
| [`pulsar-replay.txt`](docs/output/pulsar-replay.txt) | `seek(earliest)` and a new subscription |
|
||||
| [`pulsar-consumer-scaling.txt`](docs/output/pulsar-consumer-scaling.txt) | The 1000-message receiver queue defeating fan-out |
|
||||
| [`footprint.txt`](docs/output/footprint.txt) | Startup, memory, ports and config surface for all three |
|
||||
| [`tests.txt`](docs/output/tests.txt) | The nine tests behind all of the above |
|
||||
|
||||
## What is not measured here
|
||||
|
||||
Throughput. A messages-per-second number from one 2-core container says nothing about any of
|
||||
these brokers, and publishing one would be worse than publishing nothing. The measurements above
|
||||
are structural: they hold on any hardware, because they are properties of the delivery models
|
||||
rather than of the machine.
|
||||
49
broker-comparison/docs/01-three-models.md
Normal file
49
broker-comparison/docs/01-three-models.md
Normal file
@@ -0,0 +1,49 @@
|
||||
[README](../README.md) · next: [Ordering](02-ordering.md)
|
||||
|
||||
# 1. Three storage models, and everything that follows from them
|
||||
|
||||
Almost every difference between these brokers follows from one sentence about where a message
|
||||
lives.
|
||||
|
||||
- **Kafka** is an append-only log, partitioned. The broker keeps every record for the retention
|
||||
period and remembers nothing about individual consumers except an offset. Reading does not
|
||||
remove anything.
|
||||
- **RabbitMQ** is a router with queues. The broker owns each message until a consumer
|
||||
acknowledges it, and then deletes it. It can route, expire, and dead-letter on its own.
|
||||
- **Pulsar** is a log too, but the cursor and the storage are separate services: brokers are
|
||||
stateless and BookKeeper holds the data. Acknowledgement moves a cursor, and a message is
|
||||
deleted once every subscription has passed it — unless a retention policy says otherwise.
|
||||
|
||||
Read those three sentences again before reading any comparison table, including the one in
|
||||
[chapter 7](07-the-decision-table.md). They predict most of it.
|
||||
|
||||
| | Kafka | RabbitMQ | Pulsar |
|
||||
|---|---|---|---|
|
||||
| unit of parallelism | partition | queue | subscription |
|
||||
| set when | topic is created | queue is declared | consumer subscribes |
|
||||
| consuming | moves an offset | deletes the message | moves a cursor |
|
||||
| a second reader | new consumer group | another queue, bound in advance | new subscription |
|
||||
|
||||
The last row is the one people underestimate. In Kafka and Pulsar you can add a reader that sees
|
||||
history you have already processed. In RabbitMQ you cannot add one after the fact at all — the
|
||||
messages are gone — so the decision to have a second consumer has to be made *before* the
|
||||
messages arrive.
|
||||
|
||||
## What was measured, and how
|
||||
|
||||
Three questions, asked identically of all three brokers by driving the client libraries directly
|
||||
rather than through three different Spring abstractions:
|
||||
|
||||
| Question | Chapter |
|
||||
|---|---|
|
||||
| What ordering survives when you add a second consumer? | [02](02-ordering.md) |
|
||||
| Can you read the same message twice? | [03](03-replay.md) |
|
||||
| How far do consumers scale, and what stops them? | [04](04-consumer-scaling.md) |
|
||||
| What does each one cost to run? | [05](05-operational-footprint.md) |
|
||||
|
||||
Versions are the ones a Spring Boot 4.1.1 application gets: kafka-clients 4.2.1, amqp-client
|
||||
5.30.0, pulsar-client 4.2.4, all read from `spring-boot-dependencies-4.1.1.pom`. The brokers are
|
||||
Kafka 4.2.1, RabbitMQ 3.10.25 and Pulsar 4.2.4, each started from its own distribution by a script
|
||||
in `scripts/`.
|
||||
|
||||
next: [Ordering](02-ordering.md)
|
||||
66
broker-comparison/docs/02-ordering.md
Normal file
66
broker-comparison/docs/02-ordering.md
Normal file
@@ -0,0 +1,66 @@
|
||||
prev: [Three models](01-three-models.md) · [README](../README.md) · next: [Replay](03-replay.md)
|
||||
|
||||
# 2. Ordering
|
||||
|
||||
Every broker here is FIFO with one producer and one consumer. The question that matters is what
|
||||
survives the second consumer, because that is the first thing you add.
|
||||
|
||||
## Kafka: ordered within a partition, and a key picks the partition
|
||||
|
||||
Twelve records, three keys, three partitions
|
||||
([`docs/output/kafka-ordering.txt`](output/kafka-ordering.txt)):
|
||||
|
||||
```
|
||||
F -> partition [2] values [2, 5, 8, 11]
|
||||
A -> partition [1] values [1, 4, 7, 10]
|
||||
D -> partition [0] values [3, 6, 9, 12]
|
||||
```
|
||||
|
||||
Each key's values come back in the order they were produced. The global sequence does not: the
|
||||
consumer drains one partition's buffer before moving to the next, so the delivered order is
|
||||
2, 5, 8, 11, 1, 4, 7, 10, 3, 6, 9, 12.
|
||||
|
||||
That is the whole Kafka ordering guarantee, and it is usually enough, because "in order" nearly
|
||||
always means "in order per customer / per account / per device" rather than globally.
|
||||
|
||||
**A detail worth stealing.** The keys in that transcript are D, A and F, not A, B and C. With
|
||||
three partitions, murmur2 sends A, B *and* C all to partition 1. Picking three obvious keys to
|
||||
demonstrate partitioning would have produced a transcript in which global order was accidentally
|
||||
preserved. Before you conclude that your keys spread evenly, compute
|
||||
`Utils.toPositive(Utils.murmur2(key)) % partitions` for the ones you actually use.
|
||||
|
||||
## RabbitMQ: FIFO per queue, and no key at all
|
||||
|
||||
One consumer sees the queue in order. Two consumers do not
|
||||
([`docs/output/rabbit-ordering.txt`](output/rabbit-ordering.txt)):
|
||||
|
||||
```
|
||||
one queue, one consumer : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
|
||||
one queue, two consumers : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
|
||||
```
|
||||
|
||||
Message 1 went to the slower consumer and finished tenth. There is no key and no partition, so
|
||||
there is nothing to pin related messages to one consumer. Ordering across related messages in
|
||||
RabbitMQ means one queue and one consumer — which means no horizontal scaling of that queue —
|
||||
or the consistent-hash exchange, which is a plugin and effectively reintroduces partitions by
|
||||
hand.
|
||||
|
||||
## Pulsar: the subscription type decides, per subscriber
|
||||
|
||||
Two consumers on a `Shared` subscription and two on `Key_Shared`, same topic, same messages
|
||||
([`docs/output/pulsar-ordering.txt`](output/pulsar-ordering.txt)):
|
||||
|
||||
```
|
||||
Shared A -> [consumer-1, consumer-2] B -> [consumer-1, consumer-2] C -> [consumer-1, consumer-2]
|
||||
Key_Shared A -> [consumer-2] B -> [consumer-1] C -> [consumer-2]
|
||||
```
|
||||
|
||||
Under `Shared`, every key was handled by both consumers, so two messages with the same key can be
|
||||
in flight at once and no ordering claim survives. Under `Key_Shared`, each key went to exactly one
|
||||
consumer — Kafka's guarantee, but chosen by the subscriber rather than fixed by a partition count
|
||||
someone picked when the topic was created, and recomputed as consumers join and leave.
|
||||
|
||||
That last property is the strongest single argument for Pulsar: two teams reading the same topic
|
||||
can make different ordering-versus-throughput trades without negotiating.
|
||||
|
||||
next: [Replay](03-replay.md)
|
||||
59
broker-comparison/docs/03-replay.md
Normal file
59
broker-comparison/docs/03-replay.md
Normal file
@@ -0,0 +1,59 @@
|
||||
prev: [Ordering](02-ordering.md) · [README](../README.md) · next: [Consumer scaling](04-consumer-scaling.md)
|
||||
|
||||
# 3. Replay
|
||||
|
||||
"Can I read that again?" is asked after a bad deploy, and by then it is too late to change
|
||||
brokers.
|
||||
|
||||
## Kafka
|
||||
|
||||
```
|
||||
group replay-group-1, first read : 12 records
|
||||
group replay-group-2, brand new group : 12 records
|
||||
group replay-group-1, after seekToBeginning: 12 records
|
||||
```
|
||||
|
||||
([`docs/output/kafka-replay.txt`](output/kafka-replay.txt)) Consuming moves an offset and removes
|
||||
nothing. A new group, a reset offset and a `seekToBeginning` all read the same records, for as
|
||||
long as retention keeps them.
|
||||
|
||||
## RabbitMQ
|
||||
|
||||
```
|
||||
first drain of the queue : 12 messages
|
||||
second drain of the queue : 0 messages
|
||||
queue depth afterwards : 0
|
||||
```
|
||||
|
||||
([`docs/output/rabbit-replay.txt`](output/rabbit-replay.txt)) Acknowledging deletes. There is no
|
||||
offset to rewind and no second reader that can see what the first consumed. Reading a message
|
||||
twice has to be arranged in advance — a second queue bound to the same exchange, or a copy written
|
||||
somewhere else — and it cannot be arranged afterwards.
|
||||
|
||||
This is not a defect. A router that deletes what it has delivered is much cheaper to operate than
|
||||
a log, and most work queues genuinely do not need history. It only becomes a defect at the moment
|
||||
you need history and do not have it.
|
||||
|
||||
RabbitMQ streams (3.9+) are a separate, log-shaped feature that does support replay. They are a
|
||||
different thing living in the same broker, with their own client protocol and their own semantics
|
||||
— worth knowing about, and not what you get from `queueDeclare`.
|
||||
|
||||
## Pulsar
|
||||
|
||||
```
|
||||
subscription replay-sub, first read : 12 messages
|
||||
subscription replay-sub, after seek(earliest): 12 messages
|
||||
subscription replay-sub-2, brand new : 12 messages
|
||||
```
|
||||
|
||||
([`docs/output/pulsar-replay.txt`](output/pulsar-replay.txt)) `seek(MessageId)` and
|
||||
`seek(timestamp)` rewind a live subscription; a new subscription starting from `Earliest` reads
|
||||
everything still stored.
|
||||
|
||||
The difference from Kafka is the **default**, and it is the one that surprises people: Kafka keeps
|
||||
a record for the retention period regardless of who read it, while Pulsar deletes a message once
|
||||
every subscription has acknowledged it, unless a retention policy on the namespace says otherwise.
|
||||
A Pulsar namespace with the default retention and one well-behaved subscription keeps nothing —
|
||||
so the replay you are counting on requires a policy you have to set.
|
||||
|
||||
next: [Consumer scaling](04-consumer-scaling.md)
|
||||
80
broker-comparison/docs/04-consumer-scaling.md
Normal file
80
broker-comparison/docs/04-consumer-scaling.md
Normal file
@@ -0,0 +1,80 @@
|
||||
prev: [Replay](03-replay.md) · [README](../README.md) · next: [Operational footprint](05-operational-footprint.md)
|
||||
|
||||
# 4. Consumer scaling
|
||||
|
||||
The question is what happens when you add the fifth consumer.
|
||||
|
||||
## Kafka: partitions are a hard ceiling
|
||||
|
||||
Five consumers in one group on a three-partition topic
|
||||
([`docs/output/kafka-consumer-scaling.txt`](output/kafka-consumer-scaling.txt)):
|
||||
|
||||
```
|
||||
consumer-1 -> partitions [0]
|
||||
consumer-2 -> partitions [1]
|
||||
consumer-3 -> partitions [2]
|
||||
consumer-4 -> no partitions (idle)
|
||||
consumer-5 -> no partitions (idle)
|
||||
```
|
||||
|
||||
A partition belongs to at most one consumer in a group, so partition count is the ceiling. Two of
|
||||
the five processes are running, connected, healthy, and doing nothing at all — and they will keep
|
||||
reporting healthy forever.
|
||||
|
||||
Raising the partition count later is possible and is not free: it changes which partition a key
|
||||
maps to, so the per-key ordering guarantee is broken across the change for every key that moves.
|
||||
In practice the partition count is a capacity decision made at design time, on incomplete
|
||||
information, that you then live with.
|
||||
|
||||
## RabbitMQ: no ceiling, but prefetch decides whether it is real
|
||||
|
||||
Five consumers on one queue, forty messages
|
||||
([`docs/output/rabbit-consumer-scaling.txt`](output/rabbit-consumer-scaling.txt)):
|
||||
|
||||
```
|
||||
no basicQos at all (unlimited prefetch, the AMQP default):
|
||||
consumer-1 -> 40 consumer-2..5 -> 0
|
||||
|
||||
basicQos(1):
|
||||
consumer-1..5 -> 8 each
|
||||
```
|
||||
|
||||
There is no structural ceiling — but with the AMQP default the broker pushes as many messages as
|
||||
a consumer will accept, so the first consumer to connect can be handed the entire backlog while
|
||||
four idle processes wait. `basicQos` is not a tuning knob to postpone; it is what makes the
|
||||
fan-out exist.
|
||||
|
||||
Spring AMQP sets it for you: `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` is 250.
|
||||
Better than unlimited, and still enough to concentrate any backlog smaller than 250 messages on
|
||||
one consumer.
|
||||
|
||||
## Pulsar: no ceiling either, and the same trap under a different name
|
||||
|
||||
Five consumers, one *non-partitioned* topic, forty messages
|
||||
([`docs/output/pulsar-consumer-scaling.txt`](output/pulsar-consumer-scaling.txt)):
|
||||
|
||||
```
|
||||
receiverQueueSize default (1000):
|
||||
one or two consumers take everything; the rest receive nothing
|
||||
|
||||
receiverQueueSize(1):
|
||||
consumer-1..5 -> 8 each
|
||||
```
|
||||
|
||||
The topic has no partitions and five consumers still share the work, which in Kafka would require
|
||||
at least five partitions decided in advance. But the default receiver queue is a thousand
|
||||
messages, so the first consumers to connect pull the whole backlog into their own buffers first.
|
||||
Same failure as RabbitMQ's unlimited prefetch, four times the default.
|
||||
|
||||
## The shape of the answer
|
||||
|
||||
- Kafka gives you ordering per key and pays for it with a fixed parallelism ceiling.
|
||||
- RabbitMQ gives you unlimited competing consumers and pays for it with no ordering across them.
|
||||
- Pulsar lets each subscription choose, which is genuinely the best of both — at the operational
|
||||
cost measured in [the next chapter](05-operational-footprint.md).
|
||||
|
||||
And all three brokers have a client-side buffer that will quietly defeat your fan-out if you leave
|
||||
it at the default and your backlog is small. That is the one finding here that applies to
|
||||
whichever you pick.
|
||||
|
||||
next: [Operational footprint](05-operational-footprint.md)
|
||||
55
broker-comparison/docs/05-operational-footprint.md
Normal file
55
broker-comparison/docs/05-operational-footprint.md
Normal file
@@ -0,0 +1,55 @@
|
||||
prev: [Consumer scaling](04-consumer-scaling.md) · [README](../README.md) · next: [What Spring adds](06-what-spring-adds.md)
|
||||
|
||||
# 5. Operational footprint
|
||||
|
||||
"Ops burden" cannot be benchmarked. The number of moving parts, the memory floor and the size of
|
||||
the configuration surface can be, and they are what people are actually asking about.
|
||||
|
||||
Each broker started from its shipped distribution with default configuration and a 700 MB heap
|
||||
cap, on one 2-core / 3.8 GB Linux box, Temurin JDK 21
|
||||
([`docs/output/footprint.txt`](output/footprint.txt)):
|
||||
|
||||
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 standalone |
|
||||
|---|---|---|---|
|
||||
| launch to first accepted connection | 10.0 s | 15.1 s | 20.8 s |
|
||||
| resident memory at idle | 333 MB | 115 MB | 610 MB |
|
||||
| server processes | 1 | 1 (+ `epmd`) | 1 |
|
||||
| listening ports | 9092, 9093 | 5672, 4369, 25672 | 6650, 8080, 2181 |
|
||||
| unpacked distribution | 135 MB | 26 MB | 344 MB |
|
||||
| settings in the shipped default config | 24 | **0** | 357 |
|
||||
|
||||
Read those numbers as orders of magnitude, not as a benchmark: one sample, one small machine.
|
||||
|
||||
Three things they say clearly.
|
||||
|
||||
**RabbitMQ ships no configuration file at all.** The boot log records `Config file(s): (none)`;
|
||||
`rabbitmq.conf.example` is entirely commented out. Everything works out of the box, and the
|
||||
smallest resident footprint here is the one that is not a JVM. Against that, `epmd` on 4369 and
|
||||
the inter-node port on 25672 are the Erlang distribution, which is also how clustering works and
|
||||
how clustering goes wrong.
|
||||
|
||||
**Pulsar standalone is three systems in one process.** Port 2181 is ZooKeeper and the on-disk
|
||||
data is BookKeeper's; the standalone distribution hides that behind one command. A real
|
||||
deployment does not: you operate ZooKeeper (or, from Pulsar 3.x, an alternative metadata store),
|
||||
BookKeeper bookies and Pulsar brokers as three tiers with three scaling stories. The 357-setting
|
||||
`standalone.conf` is the honest signal here — that is the configuration surface, and running it
|
||||
seriously means learning most of it.
|
||||
|
||||
**Kafka since KRaft is genuinely simpler than it was.** One process, two ports, 24 settings, no
|
||||
ZooKeeper. The old "Kafka means also running ZooKeeper" objection is a version behind; anyone
|
||||
comparing on that basis is comparing to 2022.
|
||||
|
||||
## What the numbers do not include
|
||||
|
||||
The footprint above is one node at idle. The thing that actually determines ops burden is what
|
||||
happens at three in the morning:
|
||||
|
||||
- **Kafka**: rebalances, consumer lag as the primary signal, partition-count decisions you cannot
|
||||
cleanly reverse, and a broad ecosystem of tools that assume Kafka.
|
||||
- **RabbitMQ**: the management plugin is genuinely good, queue depth is a direct and obvious
|
||||
signal, and the hard problem is network partitions in a cluster — which the Erlang distribution
|
||||
makes fast to detect and awkward to resolve.
|
||||
- **Pulsar**: the fewest people on your team will have run it. That is not a technical property
|
||||
and it is usually the deciding one.
|
||||
|
||||
next: [What Spring adds](06-what-spring-adds.md)
|
||||
52
broker-comparison/docs/06-what-spring-adds.md
Normal file
52
broker-comparison/docs/06-what-spring-adds.md
Normal file
@@ -0,0 +1,52 @@
|
||||
prev: [Operational footprint](05-operational-footprint.md) · [README](../README.md) · next: [The decision table](07-the-decision-table.md)
|
||||
|
||||
# 6. What Spring adds, and what Boot 4 no longer gives you for free
|
||||
|
||||
The measurements in this module drive the client libraries directly, so that what is being
|
||||
compared is the brokers rather than three sets of Spring defaults. In an application you would use
|
||||
the Spring integrations, and they are not equivalent to one another.
|
||||
|
||||
| | Kafka | RabbitMQ | Pulsar |
|
||||
|---|---|---|---|
|
||||
| project | Spring for Apache Kafka 4.1.1 | Spring AMQP 4.1.1 | Spring for Apache Pulsar 2.0.7 |
|
||||
| starter | `spring-boot-starter-kafka` | `spring-boot-starter-amqp` | `spring-boot-starter-pulsar` |
|
||||
| listener | `@KafkaListener` | `@RabbitListener` | `@PulsarListener` |
|
||||
| template | `KafkaTemplate` | `RabbitTemplate` | `PulsarTemplate` |
|
||||
| retry / DLQ | `DefaultErrorHandler`, `@RetryableTopic` | dead-letter exchange, container error handler | `DeadLetterPolicy` on the listener |
|
||||
|
||||
All three are Boot-managed at 4.1.1, so you do not pin their versions.
|
||||
|
||||
## The Boot 4 trap that applies to all three
|
||||
|
||||
**In Spring Boot 4, depending on a messaging library directly rather than through its Boot starter
|
||||
means you have no auto-configuration.** The auto-configuration classes moved out of
|
||||
`spring-boot-autoconfigure` into per-technology modules that only the starters bring:
|
||||
|
||||
| depending on | what you lose | how it presents |
|
||||
|---|---|---|
|
||||
| `org.springframework.kafka:spring-kafka` | `spring-boot-kafka` | `No qualifying bean of type KafkaTemplate<...>` — the context starts fine |
|
||||
| `org.springframework.amqp:spring-rabbit` | `spring-boot-amqp` | no `RabbitTemplate`, no `RabbitAdmin` |
|
||||
| `org.springframework.pulsar:spring-pulsar` | `spring-boot-pulsar` | no `PulsarTemplate` |
|
||||
|
||||
Every Boot 3 tutorial gets this wrong now, and the symptom is a missing bean rather than anything
|
||||
that names the cause. Use the starters.
|
||||
|
||||
## The Jackson fork, in both Kafka and RabbitMQ
|
||||
|
||||
Boot 4 moved to Jackson 3 (`tools.jackson`), and both messaging projects ship converters for
|
||||
both generations. **A `2` in the class name means the previous Jackson** — which is the opposite
|
||||
of the convention you would guess:
|
||||
|
||||
- Spring Kafka: `JsonSerializer`/`JsonDeserializer` are Jackson 2; `JacksonJsonSerializer`,
|
||||
`JacksonJsonDeserializer` and `JacksonJsonSerde` are Jackson 3.
|
||||
- Spring AMQP: `Jackson2JsonMessageConverter` is Jackson 2; `JacksonJsonMessageConverter` is
|
||||
Jackson 3.
|
||||
|
||||
Picking the wrong one gives you `SerializationException: Can't serialize data`, whose cause is
|
||||
`Java 8 date/time type java.time.Instant not supported by default` — a message about date types
|
||||
for what is really a wrong-library problem.
|
||||
|
||||
The [kafka-basics](../kafka-basics/README.md) and [rabbitmq](../rabbitmq/README.md) modules in
|
||||
this repository cover both stacks in detail.
|
||||
|
||||
next: [The decision table](07-the-decision-table.md)
|
||||
59
broker-comparison/docs/07-the-decision-table.md
Normal file
59
broker-comparison/docs/07-the-decision-table.md
Normal file
@@ -0,0 +1,59 @@
|
||||
prev: [What Spring adds](06-what-spring-adds.md) · [README](../README.md)
|
||||
|
||||
# 7. The decision table
|
||||
|
||||
Every row below is either a measurement in `docs/output/` or a statement about operating the
|
||||
thing. Nothing here is a vendor claim.
|
||||
|
||||
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 |
|
||||
|---|---|---|---|
|
||||
| ordering with one consumer | FIFO per partition | FIFO per queue | FIFO |
|
||||
| ordering with many consumers | per key, always | **none** | per key with `Key_Shared`, none with `Shared` |
|
||||
| who chooses that | topic design | plugin, or one consumer | each subscription, independently |
|
||||
| replay after the fact | yes, within retention | **no** | yes, if retention is configured |
|
||||
| second independent reader | new consumer group, any time | must be arranged in advance | new subscription, any time |
|
||||
| consumer parallelism ceiling | partition count | none | none |
|
||||
| changing that ceiling | repartition; breaks key→partition | nothing to change | nothing to change |
|
||||
| the client-side trap | none by default | unlimited prefetch (Spring: 250) | `receiverQueueSize` 1000 |
|
||||
| memory at idle | 333 MB | **115 MB** | 610 MB |
|
||||
| processes to operate | 1 (KRaft) | 1 + `epmd` | broker + BookKeeper + metadata store |
|
||||
| shipped default settings | 24 | **0** | 357 |
|
||||
| Spring project maturity | very high | very high | good, much smaller community |
|
||||
|
||||
## Choosing
|
||||
|
||||
**Choose RabbitMQ** when the work is a queue of tasks: each message is an instruction, order
|
||||
between instructions does not matter, and once it is done it is done. It is the smallest thing
|
||||
that works, it has the best out-of-the-box operability of the three, and queue depth is a metric
|
||||
anyone can interpret. Choose it in the knowledge that you are giving up replay permanently —
|
||||
adding it later is a broker migration, not a configuration change.
|
||||
|
||||
**Choose Kafka** when the messages are events other people will want to read: when more than one
|
||||
consumer will exist, when someone will need to reprocess history after a bug, or when ordering per
|
||||
entity is part of the contract. Pay for it with a partition count decided too early, consumers
|
||||
that scale only as far as that number, and an operational model where lag is the thing you watch.
|
||||
KRaft has removed the old ZooKeeper objection.
|
||||
|
||||
**Choose Pulsar** when you genuinely need what neither of the others gives you: per-subscription
|
||||
choice of ordering versus fan-out, consumer counts not bounded by a number chosen at topic
|
||||
creation, or multi-tenancy with real isolation. It is the most capable design here. It is also
|
||||
three systems, 357 settings, five times RabbitMQ's memory floor, and the one your team has least
|
||||
experience with — and that last point decides more incidents than the first three prevent.
|
||||
|
||||
**If the honest answer is "we do not know yet"**, that argues for Kafka, on grounds that have
|
||||
nothing to do with the technology: you can hire for it, your monitoring vendor supports it, and
|
||||
the failure modes are documented by thousands of people who hit them first. That is a real
|
||||
engineering argument and it is fine to make it out loud.
|
||||
|
||||
## The question that dissolves the choice
|
||||
|
||||
A surprising number of these decisions are made for a system that has one producer, one consumer
|
||||
and fewer than a hundred messages a second. At that volume all three brokers work, none of the
|
||||
measurements above will ever be reached, and the decision is entirely about what your team can
|
||||
operate at three in the morning.
|
||||
|
||||
The measurements matter when you can name which row you are relying on. If you cannot, you are
|
||||
picking an operational burden, not a broker — and the cheapest one to operate is the one in the
|
||||
`115 MB` cell.
|
||||
|
||||
[README](../README.md)
|
||||
40
broker-comparison/docs/output/footprint.txt
Normal file
40
broker-comparison/docs/output/footprint.txt
Normal file
@@ -0,0 +1,40 @@
|
||||
== Operational footprint, measured on one 2-core / 3.8 GB Linux box ==
|
||||
|
||||
Each broker started from its shipped distribution with default configuration and a
|
||||
700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as
|
||||
orders of magnitude, not as a benchmark.
|
||||
|
||||
broker : kafka
|
||||
time from launch to first
|
||||
accepted connection : 10035 ms
|
||||
resident memory at idle : 333 MB
|
||||
server processes : 1
|
||||
listening ports : 9092 (broker), 9093 (controller)
|
||||
unpacked distribution size : 135 MB
|
||||
settings in shipped default
|
||||
configuration file : 24 (server.properties)
|
||||
|
||||
broker : rabbit
|
||||
time from launch to first
|
||||
accepted connection : 15053 ms
|
||||
resident memory at idle : 115 MB
|
||||
server processes : 1
|
||||
listening ports : 5672 (AMQP), 4369 (epmd), 25672 (inter-node)
|
||||
unpacked distribution size : 26 MB
|
||||
settings in shipped default
|
||||
configuration file : 0 (rabbitmq.conf.example)
|
||||
epmd processes : 1
|
||||
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
|
||||
entirely commented out and the boot log records "Config file(s): (none)".
|
||||
Everything in that file is documentation, not a default.
|
||||
|
||||
broker : pulsar
|
||||
time from launch to first
|
||||
accepted connection : 20801 ms
|
||||
resident memory at idle : 610 MB
|
||||
server processes : 1
|
||||
listening ports : 6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)
|
||||
unpacked distribution size : 344 MB
|
||||
settings in shipped default
|
||||
configuration file : 357 (standalone.conf)
|
||||
|
||||
16
broker-comparison/docs/output/kafka-consumer-scaling.txt
Normal file
16
broker-comparison/docs/output/kafka-consumer-scaling.txt
Normal file
@@ -0,0 +1,16 @@
|
||||
== Kafka: partitions are the ceiling on consumer parallelism ==
|
||||
|
||||
topic 'orders-scaling', 3 partitions, 5 consumers in one group
|
||||
|
||||
consumer-1 -> partitions [0]
|
||||
consumer-2 -> partitions [1]
|
||||
consumer-3 -> partitions [2]
|
||||
consumer-4 -> no partitions (idle)
|
||||
consumer-5 -> no partitions (idle)
|
||||
|
||||
consumers with no partitions: 2
|
||||
|
||||
A partition is assigned to at most one consumer in a group, so the number
|
||||
of partitions is a hard ceiling on consumer parallelism. Adding consumers
|
||||
beyond it adds idle processes, not throughput.
|
||||
|
||||
30
broker-comparison/docs/output/kafka-ordering.txt
Normal file
30
broker-comparison/docs/output/kafka-ordering.txt
Normal file
@@ -0,0 +1,30 @@
|
||||
== Kafka: ordering is per partition ==
|
||||
|
||||
produced 12 records, keys D/A/F round-robin, values 1..12 in order
|
||||
(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C
|
||||
all hash to partition 1 with three partitions, which is worth knowing
|
||||
before you decide your keys are well distributed.)
|
||||
|
||||
consumed in this order:
|
||||
F=2@p2
|
||||
F=5@p2
|
||||
F=8@p2
|
||||
F=11@p2
|
||||
A=1@p1
|
||||
A=4@p1
|
||||
A=7@p1
|
||||
A=10@p1
|
||||
D=3@p0
|
||||
D=6@p0
|
||||
D=9@p0
|
||||
D=12@p0
|
||||
|
||||
per key:
|
||||
F -> partition [2] values [2, 5, 8, 11]
|
||||
A -> partition [1] values [1, 4, 7, 10]
|
||||
D -> partition [0] values [3, 6, 9, 12]
|
||||
|
||||
Order is preserved within each key because a key hashes to one partition.
|
||||
Across keys it is not: the values above are not 1..12 in order, because a
|
||||
consumer drains one partition's buffer before the next.
|
||||
|
||||
10
broker-comparison/docs/output/kafka-replay.txt
Normal file
10
broker-comparison/docs/output/kafka-replay.txt
Normal file
@@ -0,0 +1,10 @@
|
||||
== Kafka: the log is the storage ==
|
||||
|
||||
group replay-group-1, first read : 12 records
|
||||
group replay-group-2, brand new group : 12 records
|
||||
group replay-group-1, after seekToBeginning: 12 records
|
||||
|
||||
Consuming does not remove anything. A consumer group is a cursor over a log
|
||||
that the broker keeps until retention expires, so a new group, a reset
|
||||
offset or a seek all read the same records again.
|
||||
|
||||
30
broker-comparison/docs/output/pulsar-consumer-scaling.txt
Normal file
30
broker-comparison/docs/output/pulsar-consumer-scaling.txt
Normal file
@@ -0,0 +1,30 @@
|
||||
== Pulsar: no partition ceiling, but the receiver queue decides who gets the work ==
|
||||
|
||||
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
|
||||
|
||||
receiverQueueSize left at the default (1000):
|
||||
consumer-1 -> 40 messages
|
||||
consumer-2 -> 0 messages
|
||||
consumer-3 -> 0 messages
|
||||
consumer-4 -> 0 messages
|
||||
consumer-5 -> 0 messages
|
||||
consumers that received nothing : 4
|
||||
|
||||
receiverQueueSize(1):
|
||||
consumer-1 -> 8 messages
|
||||
consumer-2 -> 8 messages
|
||||
consumer-3 -> 8 messages
|
||||
consumer-4 -> 8 messages
|
||||
consumer-5 -> 8 messages
|
||||
consumers that received nothing : 0
|
||||
|
||||
The topic has no partitions and five consumers can still share the work --
|
||||
in Kafka the same shape needs at least five partitions, chosen when the
|
||||
topic was created. But the default receiver queue is 1000 messages, so the
|
||||
first consumers to connect pull the whole 40-message backlog into their own
|
||||
buffers before the rest ask for anything, and the subscription looks
|
||||
broken. Which consumers win is a race and varies between runs -- one
|
||||
consumer taking all forty, or two taking twenty each -- but the consumers
|
||||
that lose it see nothing at all. This is the same trap as RabbitMQ's
|
||||
unbounded prefetch, with a different name and a much larger default.
|
||||
|
||||
24
broker-comparison/docs/output/pulsar-ordering.txt
Normal file
24
broker-comparison/docs/output/pulsar-ordering.txt
Normal file
@@ -0,0 +1,24 @@
|
||||
== Pulsar: the subscription type decides ==
|
||||
|
||||
12 messages, keys A/B/C, values 1..12 in order, two consumers per
|
||||
subscription, receiverQueueSize 1 so the first consumer cannot take the
|
||||
whole backlog.
|
||||
|
||||
Shared subscription, which consumers saw each key:
|
||||
A -> [consumer-1, consumer-2]
|
||||
B -> [consumer-1, consumer-2]
|
||||
C -> [consumer-1, consumer-2]
|
||||
|
||||
Key_Shared subscription, which consumers saw each key:
|
||||
A -> [consumer-1]
|
||||
B -> [consumer-1]
|
||||
C -> [consumer-1]
|
||||
|
||||
A Shared subscription round-robins individual messages, so messages with
|
||||
the same key end up on different consumers and can be processed at the
|
||||
same time: there is no per-key order left to speak of. Key_Shared hashes
|
||||
the key to one consumer, which is Kafka's guarantee -- except that the
|
||||
assignment belongs to the subscription and is recomputed as consumers come
|
||||
and go, rather than being fixed by a partition count chosen when the topic
|
||||
was created.
|
||||
|
||||
14
broker-comparison/docs/output/pulsar-replay.txt
Normal file
14
broker-comparison/docs/output/pulsar-replay.txt
Normal file
@@ -0,0 +1,14 @@
|
||||
== Pulsar: acknowledged, but still there ==
|
||||
|
||||
subscription replay-sub, first read : 12 messages
|
||||
subscription replay-sub, after seek(earliest): 12 messages
|
||||
subscription replay-sub-2, brand new : 12 messages
|
||||
|
||||
Acknowledgement moves a cursor; the message itself lives in the managed
|
||||
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
|
||||
which Kafka can also do by resetting offsets. What differs is the default:
|
||||
Pulsar deletes a message once every subscription has acknowledged it,
|
||||
unless a retention policy on the namespace says otherwise, whereas Kafka
|
||||
keeps it for the retention period regardless of who read it. A Pulsar
|
||||
topic with no retention policy and no subscriptions keeps nothing.
|
||||
|
||||
34
broker-comparison/docs/output/rabbit-consumer-scaling.txt
Normal file
34
broker-comparison/docs/output/rabbit-consumer-scaling.txt
Normal file
@@ -0,0 +1,34 @@
|
||||
== RabbitMQ: consumers scale, and prefetch decides whether they actually do ==
|
||||
|
||||
one queue, 40 messages, 5 consumers, each taking 10 ms per message
|
||||
|
||||
no basicQos at all (unlimited prefetch, the AMQP default):
|
||||
consumer-1 -> 40 messages
|
||||
consumer-2 -> 0 messages
|
||||
consumer-3 -> 0 messages
|
||||
consumer-4 -> 0 messages
|
||||
consumer-5 -> 0 messages
|
||||
consumers that received nothing : 4
|
||||
|
||||
basicQos(1):
|
||||
consumer-1 -> 8 messages
|
||||
consumer-2 -> 9 messages
|
||||
consumer-3 -> 8 messages
|
||||
consumer-4 -> 8 messages
|
||||
consumer-5 -> 7 messages
|
||||
consumers that received nothing : 0
|
||||
|
||||
Every consumer on a queue competes for the same messages, so adding
|
||||
consumers adds throughput and there is no structural ceiling of the kind
|
||||
Kafka's partition count imposes. But with the AMQP default the broker
|
||||
pushes as many messages as a consumer will take, so whichever consumer
|
||||
connects first can be handed the entire backlog while the others sit idle.
|
||||
basicQos is not a tuning knob you get to postpone; it is what makes the
|
||||
fan-out real. Spring AMQP sets it for you --
|
||||
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
|
||||
better than unlimited and still large enough to concentrate a small
|
||||
backlog on one consumer.
|
||||
|
||||
The price of all this is the ordering measurement: there is no key, so
|
||||
nothing constrains related messages to one consumer.
|
||||
|
||||
18
broker-comparison/docs/output/rabbit-ordering.txt
Normal file
18
broker-comparison/docs/output/rabbit-ordering.txt
Normal file
@@ -0,0 +1,18 @@
|
||||
== RabbitMQ: FIFO per queue, per consumer ==
|
||||
|
||||
one queue, one consumer, 12 messages
|
||||
completion order : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
|
||||
|
||||
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
|
||||
completion order : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
|
||||
out-of-order steps: 1
|
||||
|
||||
A queue is FIFO and a single consumer sees it that way. The moment a second
|
||||
consumer is added the broker hands the next message to whichever consumer
|
||||
is free, so the order in which work finishes is no longer the order in
|
||||
which it was published. There is no key: RabbitMQ has no notion of a
|
||||
partition to which related messages could be pinned. Ordering across
|
||||
related messages means one queue and one consumer, and therefore no
|
||||
horizontal scaling for that queue -- or a consistent-hash exchange, which
|
||||
is a plugin.
|
||||
|
||||
12
broker-comparison/docs/output/rabbit-replay.txt
Normal file
12
broker-comparison/docs/output/rabbit-replay.txt
Normal file
@@ -0,0 +1,12 @@
|
||||
== RabbitMQ: there is nothing to replay ==
|
||||
|
||||
first drain of the queue : 12 messages
|
||||
second drain of the queue : 0 messages
|
||||
queue depth afterwards : 0
|
||||
|
||||
Acknowledging a message deletes it. The broker is a router with buffers,
|
||||
not a log: there is no offset to rewind and no second reader that can see
|
||||
what the first one consumed. Reading the same message twice means
|
||||
arranging it in advance -- a second queue bound to the same exchange, or a
|
||||
copy written somewhere else -- and it cannot be arranged after the fact.
|
||||
|
||||
12
broker-comparison/docs/output/tests.txt
Normal file
12
broker-comparison/docs/output/tests.txt
Normal file
@@ -0,0 +1,12 @@
|
||||
[INFO] Running com.ankurm.brokers.KafkaComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.88 s -- in com.ankurm.brokers.KafkaComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] Running com.ankurm.brokers.RabbitComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.732 s -- in com.ankurm.brokers.RabbitComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] Running com.ankurm.brokers.PulsarComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.00 s -- in com.ankurm.brokers.PulsarComparisonTest
|
||||
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
|
||||
[INFO] BUILD SUCCESS
|
||||
66
broker-comparison/pom.xml
Normal file
66
broker-comparison/pom.xml
Normal file
@@ -0,0 +1,66 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<!-- Every client version below is managed by spring-boot-dependencies 4.1.1, so the three
|
||||
brokers are compared at the versions a Boot 4.1 application would actually get:
|
||||
kafka-clients 4.2.1, amqp-client 5.30.0, pulsar-client 4.2.4. Read from
|
||||
spring-boot-dependencies-4.1.1.pom, not from release announcements. -->
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>broker-comparison</artifactId>
|
||||
<version>1.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- The three client libraries, driven directly. The comparison is between the brokers'
|
||||
delivery models, and going through three different Spring abstractions would put three
|
||||
different sets of defaults between the measurement and the thing being measured.
|
||||
What each Boot starter adds on top is covered in docs/06-what-spring-adds.md. -->
|
||||
<dependency>
|
||||
<groupId>org.apache.kafka</groupId>
|
||||
<artifactId>kafka-clients</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.rabbitmq</groupId>
|
||||
<artifactId>amqp-client</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.pulsar</groupId>
|
||||
<artifactId>pulsar-client</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- Provides the in-process KRaft broker used by the Kafka measurements: a real Kafka
|
||||
broker, no Docker. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.kafka</groupId>
|
||||
<artifactId>spring-kafka-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<!-- Each broker is started by its own script and the matching tagged tests are run on
|
||||
their own; see scripts/run-all.sh, which passes -Dgroups=kafka|rabbit|pulsar. Running
|
||||
"mvn test" with no group runs all three and needs all three brokers up. -->
|
||||
|
||||
</project>
|
||||
73
broker-comparison/scripts/footprint.sh
Executable file
73
broker-comparison/scripts/footprint.sh
Executable file
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# Measures the operational footprint of one broker: how long it takes to accept a connection,
|
||||
# how much memory it holds at idle, how many processes and listening ports it opens, and how
|
||||
# large its shipped default configuration is.
|
||||
#
|
||||
# These are proxies, not a benchmark. "Ops burden" is not directly measurable, but the number of
|
||||
# moving parts, the memory floor and the size of the configuration surface are, and they are what
|
||||
# people are actually asking about when they ask which broker is heavier to run.
|
||||
#
|
||||
# scripts/footprint.sh kafka|rabbit|pulsar
|
||||
set -euo pipefail
|
||||
BROKER="${1:?kafka|rabbit|pulsar}"
|
||||
|
||||
extra=""
|
||||
port_open() { (echo > "/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
|
||||
rss_mb() { ps -eo rss,cmd | grep -F "$1" | grep -v grep | awk '{s+=$1} END {printf "%.0f", s/1024}'; }
|
||||
procs() { ps -eo cmd | grep -F "$1" | grep -cv grep; }
|
||||
settings() { grep -cE '^[a-zA-Z]' "$1" 2>/dev/null || echo 0; }
|
||||
|
||||
start=$(date +%s%3N)
|
||||
case "$BROKER" in
|
||||
kafka)
|
||||
: "${KAFKA_HOME:?set KAFKA_HOME}"
|
||||
# Wipe whatever log.dirs points at, so the format below is not refused with
|
||||
# "Invalid cluster.id ... Expected X, but read Y" from a previous run.
|
||||
KAFKA_LOG_DIRS=$(grep -E '^log.dirs=' "$KAFKA_HOME/config/server.properties" | cut -d= -f2)
|
||||
rm -rf ${KAFKA_LOG_DIRS//,/ }
|
||||
"$KAFKA_HOME/bin/kafka-storage.sh" format --standalone -t "$("$KAFKA_HOME/bin/kafka-storage.sh" random-uuid)" \
|
||||
-c "$KAFKA_HOME/config/server.properties" --ignore-formatted >/dev/null
|
||||
start=$(date +%s%3N) # after formatting: kafka-storage.sh is two more JVM starts and is a
|
||||
# one-off, not part of what a restart costs
|
||||
setsid nohup "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/server.properties" \
|
||||
> /tmp/kafka-start.log 2>&1 < /dev/null &
|
||||
for _ in $(seq 1 90); do port_open 9092 && break; sleep 1; done
|
||||
ready=$(( $(date +%s%3N) - start ))
|
||||
marker="kafka.Kafka"; ports="9092 (broker), 9093 (controller)"; conf="$KAFKA_HOME/config/server.properties"
|
||||
dist="$KAFKA_HOME"
|
||||
;;
|
||||
rabbit)
|
||||
: "${RABBITMQ_HOME:?set RABBITMQ_HOME}"; : "${ERL_ROOT:?set ERL_ROOT}"
|
||||
scripts/rabbit-broker.sh >/dev/null
|
||||
ready=$(( $(date +%s%3N) - start ))
|
||||
# The broker is an Erlang VM, so the process to measure is beam.smp, not anything named
|
||||
# "rabbitmq". epmd is a second, tiny process and is counted separately below.
|
||||
marker="beam.smp"; ports="5672 (AMQP), 4369 (epmd), 25672 (inter-node)"
|
||||
conf="$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf.example"; dist="$RABBITMQ_HOME"
|
||||
extra="epmd processes : $(procs epmd)
|
||||
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
|
||||
entirely commented out and the boot log records \"Config file(s): (none)\".
|
||||
Everything in that file is documentation, not a default."
|
||||
;;
|
||||
pulsar)
|
||||
: "${PULSAR_HOME:?set PULSAR_HOME}"
|
||||
scripts/pulsar-broker.sh >/dev/null
|
||||
ready=$(( $(date +%s%3N) - start ))
|
||||
marker="PulsarStandaloneStarter"; ports="6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)"
|
||||
conf="$PULSAR_HOME/conf/standalone.conf"; dist="$PULSAR_HOME"
|
||||
;;
|
||||
*) echo "unknown broker $BROKER" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
cat <<EOF
|
||||
broker : $BROKER
|
||||
time from launch to first
|
||||
accepted connection : ${ready} ms
|
||||
resident memory at idle : $(rss_mb "$marker") MB
|
||||
server processes : $(procs "$marker")
|
||||
listening ports : $ports
|
||||
unpacked distribution size : $(du -sm "$dist" | cut -f1) MB
|
||||
settings in shipped default
|
||||
configuration file : $(settings "$conf") ($(basename "$conf"))
|
||||
${extra:-}
|
||||
EOF
|
||||
25
broker-comparison/scripts/pulsar-broker.sh
Executable file
25
broker-comparison/scripts/pulsar-broker.sh
Executable file
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start a real Apache Pulsar broker with no Docker and no root.
|
||||
#
|
||||
# The standalone distribution bundles ZooKeeper, BookKeeper and the broker in one process, which
|
||||
# is why a single tarball is enough. -nss skips the stream storage (BookKeeper's table service,
|
||||
# needed only by stateful Pulsar Functions) and -nfw skips the functions worker; both save
|
||||
# several hundred megabytes of heap and a few seconds of startup.
|
||||
#
|
||||
# PULSAR_HOME must point at an unpacked apache-pulsar-<version>-bin directory. Pulsar 4.2
|
||||
# supports Java 17 and 21; the transcripts here were produced on Temurin 21.
|
||||
set -eu
|
||||
: "${PULSAR_HOME:?set PULSAR_HOME to an unpacked apache-pulsar-*-bin directory}"
|
||||
export PULSAR_MEM="${PULSAR_MEM:--Xms384m -Xmx700m -XX:MaxDirectMemorySize=384m}"
|
||||
|
||||
cd "$PULSAR_HOME"
|
||||
setsid nohup bin/pulsar standalone -nss -nfw > /tmp/pulsar-start.log 2>&1 < /dev/null &
|
||||
for _ in $(seq 1 90); do
|
||||
if [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/admin/v2/brokers/version)" = "200" ]; then
|
||||
echo "broker ready on 6650 (admin 8080), version $(curl -s http://127.0.0.1:8080/admin/v2/brokers/version)"
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "broker did not start; see /tmp/pulsar-start.log" >&2
|
||||
exit 1
|
||||
31
broker-comparison/scripts/rabbit-broker.sh
Executable file
31
broker-comparison/scripts/rabbit-broker.sh
Executable file
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start a real RabbitMQ broker with no Docker and no root.
|
||||
#
|
||||
# RabbitMQ is an Erlang application, so an Erlang runtime and epmd on the path are the whole
|
||||
# dependency. Point ERL_ROOT at an Erlang installation and RABBITMQ_HOME at an unpacked
|
||||
# rabbitmq-server-generic-unix tarball. Ubuntu 22.04 ships Erlang 24, whose newest compatible
|
||||
# broker is 3.10.25; 3.11 and later need Erlang 25.
|
||||
#
|
||||
# The one non-obvious step is starting epmd yourself: rabbitmq-server's own attempt to start it
|
||||
# fails in a container without a resolvable hostname, and the failure surfaces as a forty-line
|
||||
# Erlang crash dump ending in {'EXIT',nodistribution}.
|
||||
set -eu
|
||||
: "${ERL_ROOT:?set ERL_ROOT to an Erlang installation directory}"
|
||||
: "${RABBITMQ_HOME:?set RABBITMQ_HOME to an unpacked rabbitmq-server-generic-unix directory}"
|
||||
|
||||
export PATH="$ERL_ROOT/bin:$ERL_ROOT/erts-"*/bin":$PATH"
|
||||
export RABBITMQ_MNESIA_BASE="${RABBITMQ_MNESIA_BASE:-/tmp/rmq/data}"
|
||||
export RABBITMQ_LOG_BASE="${RABBITMQ_LOG_BASE:-/tmp/rmq/log}"
|
||||
export RABBITMQ_NODENAME="${RABBITMQ_NODENAME:-rabbit@localhost}"
|
||||
export HOME="${HOME:-/tmp/rmq}"
|
||||
mkdir -p "$RABBITMQ_MNESIA_BASE" "$RABBITMQ_LOG_BASE"
|
||||
|
||||
epmd -daemon 2>/dev/null || true
|
||||
sleep 1
|
||||
setsid nohup "$RABBITMQ_HOME/sbin/rabbitmq-server" > /tmp/rmq/boot.log 2>&1 < /dev/null &
|
||||
for _ in $(seq 1 60); do
|
||||
if (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; then echo "broker ready on 5672"; exit 0; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "broker did not start; see /tmp/rmq/boot.log" >&2
|
||||
exit 1
|
||||
37
broker-comparison/scripts/run-all.sh
Executable file
37
broker-comparison/scripts/run-all.sh
Executable file
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates every file under docs/output/. Each broker is started, measured and left running
|
||||
# only for its own group of tests, because three brokers do not fit comfortably in memory on a
|
||||
# small machine at the same time.
|
||||
#
|
||||
# Required environment:
|
||||
# KAFKA_HOME unpacked kafka_2.13-4.2.1
|
||||
# ERL_ROOT an Erlang 24 installation (RabbitMQ 3.10.x)
|
||||
# RABBITMQ_HOME unpacked rabbitmq-server-generic-unix-3.10.25
|
||||
# PULSAR_HOME unpacked apache-pulsar-4.2.4-bin
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p docs/output
|
||||
|
||||
# Kafka needs no external broker for the measurements: spring-kafka-test starts a real KRaft
|
||||
# broker in-process. KAFKA_HOME is only used by the footprint measurement.
|
||||
mvn -B -Dgroups=kafka test 2>&1 | grep -E 'Running |Tests run:|BUILD ' > docs/output/tests.txt
|
||||
|
||||
scripts/rabbit-broker.sh
|
||||
mvn -B -Dgroups=rabbit test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
|
||||
|
||||
scripts/pulsar-broker.sh
|
||||
mvn -B -Dgroups=pulsar test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
|
||||
|
||||
{
|
||||
echo "== Operational footprint, measured on one 2-core / 3.8 GB Linux box =="
|
||||
echo
|
||||
echo "Each broker started from its shipped distribution with default configuration and a"
|
||||
echo "700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as"
|
||||
echo "orders of magnitude, not as a benchmark."
|
||||
echo
|
||||
scripts/footprint.sh kafka; echo
|
||||
scripts/footprint.sh rabbit; echo
|
||||
scripts/footprint.sh pulsar
|
||||
} > docs/output/footprint.txt
|
||||
|
||||
echo "regenerated:"; ls -1 docs/output
|
||||
@@ -0,0 +1,15 @@
|
||||
package com.ankurm.brokers;
|
||||
|
||||
/**
|
||||
* There is deliberately no Spring application here. The measurements in src/test drive the three
|
||||
* client libraries directly so that the thing being compared is the broker's delivery model and
|
||||
* not three different sets of Spring defaults.
|
||||
*
|
||||
* <p>What each Spring Boot starter adds on top — and what it silently does not add, which
|
||||
* in Boot 4 is more than people expect — is in docs/06-what-spring-adds.md.
|
||||
*/
|
||||
public final class BrokerComparisonApplication {
|
||||
|
||||
private BrokerComparisonApplication() {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package com.ankurm.brokers;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
/** Writes a transcript under docs/output/. Nothing in the article is typed by hand. */
|
||||
public final class Capture {
|
||||
|
||||
private Capture() {
|
||||
}
|
||||
|
||||
public static void write(String fileName, String heading, String body) {
|
||||
Path dir = Path.of(System.getProperty("user.dir"), "docs", "output");
|
||||
try {
|
||||
Files.createDirectories(dir);
|
||||
Files.writeString(dir.resolve(fileName), "== " + heading + " ==\n\n" + body + "\n");
|
||||
}
|
||||
catch (IOException ex) {
|
||||
throw new IllegalStateException("could not write " + fileName, ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
package com.ankurm.brokers;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.kafka.clients.consumer.ConsumerConfig;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecords;
|
||||
import org.apache.kafka.clients.consumer.KafkaConsumer;
|
||||
import org.apache.kafka.clients.producer.KafkaProducer;
|
||||
import org.apache.kafka.clients.producer.ProducerConfig;
|
||||
import org.apache.kafka.clients.producer.ProducerRecord;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.apache.kafka.common.serialization.StringDeserializer;
|
||||
import org.apache.kafka.common.serialization.StringSerializer;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import org.springframework.kafka.test.EmbeddedKafkaKraftBroker;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* Kafka measured against a real broker: an in-process KRaft broker from spring-kafka-test, one
|
||||
* node, one topic, three partitions.
|
||||
*
|
||||
* <p>The three questions are the same for all three brokers: what ordering survives, what can be
|
||||
* replayed, and how far consumers scale.
|
||||
*/
|
||||
@Tag("kafka")
|
||||
class KafkaComparisonTest {
|
||||
|
||||
private static final String ORDERING = "orders-ordering";
|
||||
|
||||
private static final String REPLAY = "orders-replay";
|
||||
|
||||
private static final String SCALING = "orders-scaling";
|
||||
|
||||
private static EmbeddedKafkaKraftBroker broker;
|
||||
|
||||
@BeforeAll
|
||||
static void startBroker() {
|
||||
broker = new EmbeddedKafkaKraftBroker(1, 3, ORDERING, REPLAY, SCALING);
|
||||
broker.afterPropertiesSet();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stopBroker() {
|
||||
broker.destroy();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- ordering
|
||||
|
||||
@Test
|
||||
void orderingIsPerPartitionOnly() {
|
||||
produce(ORDERING, 12);
|
||||
|
||||
List<ConsumerRecord<String, String>> received = consumeAll(ORDERING, "ordering-group", 12);
|
||||
|
||||
Map<String, List<String>> perKey = new LinkedHashMap<>();
|
||||
Map<String, Set<Integer>> partitionsPerKey = new LinkedHashMap<>();
|
||||
List<String> globalOrder = new ArrayList<>();
|
||||
for (ConsumerRecord<String, String> record : received) {
|
||||
perKey.computeIfAbsent(record.key(), (k) -> new ArrayList<>()).add(record.value());
|
||||
partitionsPerKey.computeIfAbsent(record.key(), (k) -> new java.util.TreeSet<>())
|
||||
.add(record.partition());
|
||||
globalOrder.add(record.key() + "=" + record.value() + "@p" + record.partition());
|
||||
}
|
||||
|
||||
// Per key, the sequence numbers must come back in the order they were produced. Compared
|
||||
// as numbers: as strings, "10" sorts before "2" and the assertion is meaningless.
|
||||
perKey.forEach((key, values) -> assertThat(values.stream().map(Integer::parseInt).toList())
|
||||
.isSorted());
|
||||
// Each key went to exactly one partition, which is why that holds...
|
||||
partitionsPerKey.forEach((key, partitions) -> assertThat(partitions).hasSize(1));
|
||||
// ...and the three keys are spread over all three partitions, so the transcript below is
|
||||
// actually testing something.
|
||||
assertThat(partitionsPerKey.values().stream().flatMap(Set::stream).distinct().count())
|
||||
.isEqualTo(3);
|
||||
List<Integer> asDelivered = globalOrder.stream()
|
||||
.map((entry) -> Integer.parseInt(entry.split("=")[1].split("@")[0])).toList();
|
||||
assertThat(asDelivered).isNotEqualTo(
|
||||
java.util.stream.IntStream.rangeClosed(1, 12).boxed().toList());
|
||||
|
||||
StringBuilder body = new StringBuilder();
|
||||
body.append("produced 12 records, keys D/A/F round-robin, values 1..12 in order\n")
|
||||
.append("(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C\n")
|
||||
.append(" all hash to partition 1 with three partitions, which is worth knowing\n")
|
||||
.append(" before you decide your keys are well distributed.)\n\n");
|
||||
body.append("consumed in this order:\n ").append(String.join("\n ", globalOrder))
|
||||
.append("\n\nper key:\n");
|
||||
perKey.forEach((key, values) -> body.append(" ").append(key).append(" -> partition ")
|
||||
.append(partitionsPerKey.get(key)).append(" values ").append(values).append('\n'));
|
||||
body.append("\nOrder is preserved within each key because a key hashes to one partition.\n")
|
||||
.append("Across keys it is not: the values above are not 1..12 in order, because a\n")
|
||||
.append("consumer drains one partition's buffer before the next.\n");
|
||||
Capture.write("kafka-ordering.txt", "Kafka: ordering is per partition", body.toString());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- replay
|
||||
|
||||
@Test
|
||||
void anythingStillWithinRetentionCanBeReadAgain() {
|
||||
produce(REPLAY, 12);
|
||||
|
||||
int first = consumeAll(REPLAY, "replay-group-1", 12).size();
|
||||
int again = consumeAll(REPLAY, "replay-group-2", 12).size();
|
||||
|
||||
List<ConsumerRecord<String, String>> third;
|
||||
try (KafkaConsumer<String, String> consumer = consumer("replay-group-1")) {
|
||||
consumer.subscribe(List.of(REPLAY));
|
||||
consumer.poll(Duration.ofSeconds(2));
|
||||
consumer.seekToBeginning(consumer.assignment());
|
||||
third = drain(consumer, 12);
|
||||
}
|
||||
|
||||
assertThat(first).isEqualTo(12);
|
||||
assertThat(again).isEqualTo(12);
|
||||
assertThat(third).hasSize(12);
|
||||
|
||||
Capture.write("kafka-replay.txt", "Kafka: the log is the storage",
|
||||
"""
|
||||
group replay-group-1, first read : %d records
|
||||
group replay-group-2, brand new group : %d records
|
||||
group replay-group-1, after seekToBeginning: %d records
|
||||
|
||||
Consuming does not remove anything. A consumer group is a cursor over a log
|
||||
that the broker keeps until retention expires, so a new group, a reset
|
||||
offset or a seek all read the same records again.
|
||||
""".formatted(first, again, third.size()));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- consumer scaling
|
||||
|
||||
@Test
|
||||
void consumerParallelismIsCappedByPartitionCount() throws Exception {
|
||||
produce(SCALING, 12);
|
||||
|
||||
List<KafkaConsumer<String, String>> consumers = new ArrayList<>();
|
||||
Map<String, Set<Integer>> assignment = new LinkedHashMap<>();
|
||||
try {
|
||||
for (int i = 1; i <= 5; i++) {
|
||||
KafkaConsumer<String, String> consumer = consumer("scaling-group");
|
||||
consumer.subscribe(List.of(SCALING));
|
||||
consumers.add(consumer);
|
||||
}
|
||||
// Poll each consumer until the group has settled and every member knows its share.
|
||||
for (int round = 0; round < 8; round++) {
|
||||
for (KafkaConsumer<String, String> consumer : consumers) {
|
||||
consumer.poll(Duration.ofMillis(500));
|
||||
}
|
||||
}
|
||||
for (int i = 0; i < consumers.size(); i++) {
|
||||
Set<Integer> partitions = new java.util.TreeSet<>();
|
||||
for (TopicPartition tp : consumers.get(i).assignment()) {
|
||||
partitions.add(tp.partition());
|
||||
}
|
||||
assignment.put("consumer-" + (i + 1), partitions);
|
||||
}
|
||||
}
|
||||
finally {
|
||||
consumers.forEach(KafkaConsumer::close);
|
||||
}
|
||||
|
||||
long idle = assignment.values().stream().filter(Set::isEmpty).count();
|
||||
assertThat(assignment).hasSize(5);
|
||||
assertThat(idle).isEqualTo(2);
|
||||
|
||||
StringBuilder body = new StringBuilder("topic 'orders-scaling', 3 partitions, 5 consumers in one group\n\n");
|
||||
assignment.forEach((name, partitions) -> body.append(" ").append(name).append(" -> ")
|
||||
.append(partitions.isEmpty() ? "no partitions (idle)" : "partitions " + partitions)
|
||||
.append('\n'));
|
||||
body.append("\nconsumers with no partitions: ").append(idle).append('\n')
|
||||
.append("\nA partition is assigned to at most one consumer in a group, so the number\n")
|
||||
.append("of partitions is a hard ceiling on consumer parallelism. Adding consumers\n")
|
||||
.append("beyond it adds idle processes, not throughput.\n");
|
||||
Capture.write("kafka-consumer-scaling.txt",
|
||||
"Kafka: partitions are the ceiling on consumer parallelism", body.toString());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- helpers
|
||||
|
||||
private void produce(String topic, int count) {
|
||||
Properties props = new Properties();
|
||||
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
|
||||
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
|
||||
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
|
||||
// D, A and F, not A, B and C: with three partitions murmur2 maps A, B and C all to
|
||||
// partition 1, so the obvious choice of keys would have produced a transcript in which
|
||||
// global order happened to be preserved and proved nothing. D->0, A->1, F->2.
|
||||
String[] keys = { "D", "A", "F" };
|
||||
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
|
||||
for (int i = 1; i <= count; i++) {
|
||||
producer.send(new ProducerRecord<>(topic, keys[i % keys.length], String.valueOf(i)));
|
||||
}
|
||||
producer.flush();
|
||||
}
|
||||
}
|
||||
|
||||
private KafkaConsumer<String, String> consumer(String group) {
|
||||
Properties props = new Properties();
|
||||
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
|
||||
props.put(ConsumerConfig.GROUP_ID_CONFIG, group);
|
||||
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
|
||||
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
|
||||
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
|
||||
return new KafkaConsumer<>(props);
|
||||
}
|
||||
|
||||
private List<ConsumerRecord<String, String>> consumeAll(String topic, String group, int expected) {
|
||||
try (KafkaConsumer<String, String> consumer = consumer(group)) {
|
||||
consumer.subscribe(List.of(topic));
|
||||
return drain(consumer, expected);
|
||||
}
|
||||
}
|
||||
|
||||
private List<ConsumerRecord<String, String>> drain(KafkaConsumer<String, String> consumer, int expected) {
|
||||
List<ConsumerRecord<String, String>> received = new ArrayList<>();
|
||||
for (int i = 0; i < 20 && received.size() < expected; i++) {
|
||||
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
|
||||
records.forEach(received::add);
|
||||
}
|
||||
return received;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,298 @@
|
||||
package com.ankurm.brokers;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import org.apache.pulsar.client.api.Consumer;
|
||||
import org.apache.pulsar.client.api.Message;
|
||||
import org.apache.pulsar.client.api.MessageId;
|
||||
import org.apache.pulsar.client.api.Producer;
|
||||
import org.apache.pulsar.client.api.PulsarClient;
|
||||
import org.apache.pulsar.client.api.Schema;
|
||||
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
|
||||
import org.apache.pulsar.client.api.SubscriptionType;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* Pulsar measured against a real standalone broker on localhost:6650, started by
|
||||
* scripts/pulsar-broker.sh.
|
||||
*
|
||||
* <p>Pulsar is the interesting third option because it does not force the trade the other two
|
||||
* force. The subscription type is chosen per subscriber rather than being a property of the
|
||||
* topic, so ordering and consumer fan-out are separate decisions.
|
||||
*/
|
||||
@Tag("pulsar")
|
||||
class PulsarComparisonTest {
|
||||
|
||||
private static PulsarClient client;
|
||||
|
||||
@BeforeAll
|
||||
static void connect() throws Exception {
|
||||
client = PulsarClient.builder().serviceUrl("pulsar://127.0.0.1:6650").build();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void close() throws Exception {
|
||||
client.close();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- ordering
|
||||
|
||||
@Test
|
||||
void sharedSpreadsAKeyAndKeySharedPinsIt() throws Exception {
|
||||
String topic = "persistent://public/default/ordering-" + System.nanoTime();
|
||||
produce(topic, 12);
|
||||
|
||||
Map<String, Set<String>> shared = consumersPerKey(topic, "sub-shared", SubscriptionType.Shared, 12);
|
||||
Map<String, Set<String>> keyShared = consumersPerKey(topic, "sub-keyshared",
|
||||
SubscriptionType.Key_Shared, 12);
|
||||
|
||||
// Key_Shared: every key was handled by exactly one consumer, so per-key order is safe.
|
||||
keyShared.forEach((key, handlers) -> assertThat(handlers).hasSize(1));
|
||||
// Shared: at least one key was handled by both, so two messages with that key were in
|
||||
// flight on two consumers and nothing orders them.
|
||||
assertThat(shared.values().stream().anyMatch((handlers) -> handlers.size() > 1)).isTrue();
|
||||
|
||||
Capture.write("pulsar-ordering.txt", "Pulsar: the subscription type decides",
|
||||
"""
|
||||
12 messages, keys A/B/C, values 1..12 in order, two consumers per
|
||||
subscription, receiverQueueSize 1 so the first consumer cannot take the
|
||||
whole backlog.
|
||||
|
||||
Shared subscription, which consumers saw each key:
|
||||
%s
|
||||
Key_Shared subscription, which consumers saw each key:
|
||||
%s
|
||||
A Shared subscription round-robins individual messages, so messages with
|
||||
the same key end up on different consumers and can be processed at the
|
||||
same time: there is no per-key order left to speak of. Key_Shared hashes
|
||||
the key to one consumer, which is Kafka's guarantee -- except that the
|
||||
assignment belongs to the subscription and is recomputed as consumers come
|
||||
and go, rather than being fixed by a partition count chosen when the topic
|
||||
was created.
|
||||
""".formatted(render3(shared), render3(keyShared)));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- replay
|
||||
|
||||
@Test
|
||||
void aSubscriptionCanBeRewound() throws Exception {
|
||||
String topic = "persistent://public/default/replay-" + System.nanoTime();
|
||||
produce(topic, 12);
|
||||
|
||||
int first;
|
||||
int afterSeek;
|
||||
try (Consumer<String> consumer = consumer(topic, "replay-sub", SubscriptionType.Exclusive)) {
|
||||
first = drain(consumer, 12).size();
|
||||
consumer.seek(MessageId.earliest);
|
||||
afterSeek = drain(consumer, 12).size();
|
||||
}
|
||||
|
||||
int newSubscription;
|
||||
try (Consumer<String> consumer = consumer(topic, "replay-sub-2", SubscriptionType.Exclusive)) {
|
||||
newSubscription = drain(consumer, 12).size();
|
||||
}
|
||||
|
||||
assertThat(first).isEqualTo(12);
|
||||
assertThat(afterSeek).isEqualTo(12);
|
||||
assertThat(newSubscription).isEqualTo(12);
|
||||
|
||||
Capture.write("pulsar-replay.txt", "Pulsar: acknowledged, but still there",
|
||||
"""
|
||||
subscription replay-sub, first read : %d messages
|
||||
subscription replay-sub, after seek(earliest): %d messages
|
||||
subscription replay-sub-2, brand new : %d messages
|
||||
|
||||
Acknowledgement moves a cursor; the message itself lives in the managed
|
||||
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
|
||||
which Kafka can also do by resetting offsets. What differs is the default:
|
||||
Pulsar deletes a message once every subscription has acknowledged it,
|
||||
unless a retention policy on the namespace says otherwise, whereas Kafka
|
||||
keeps it for the retention period regardless of who read it. A Pulsar
|
||||
topic with no retention policy and no subscriptions keeps nothing.
|
||||
""".formatted(first, afterSeek, newSubscription));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- consumer scaling
|
||||
|
||||
@Test
|
||||
void sharedSubscriptionsHaveNoPartitionCeilingButHaveAReceiverQueue() throws Exception {
|
||||
String topicA = "persistent://public/default/scaling-default-" + System.nanoTime();
|
||||
produce(topicA, 40);
|
||||
Map<String, Integer> withDefaultQueue = drainAcross(topicA, "scaling-sub", 5, 40, 0);
|
||||
|
||||
String topicB = "persistent://public/default/scaling-q1-" + System.nanoTime();
|
||||
produce(topicB, 40);
|
||||
Map<String, Integer> withQueueOfOne = drainAcross(topicB, "scaling-sub", 5, 40, 1);
|
||||
|
||||
long idleDefault = withDefaultQueue.values().stream().filter((n) -> n == 0).count();
|
||||
long idleTuned = withQueueOfOne.values().stream().filter((n) -> n == 0).count();
|
||||
|
||||
assertThat(idleTuned).isZero();
|
||||
assertThat(idleDefault).isGreaterThan(0);
|
||||
|
||||
Capture.write("pulsar-consumer-scaling.txt",
|
||||
"Pulsar: no partition ceiling, but the receiver queue decides who gets the work",
|
||||
"""
|
||||
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
|
||||
|
||||
receiverQueueSize left at the default (1000):
|
||||
%s consumers that received nothing : %d
|
||||
|
||||
receiverQueueSize(1):
|
||||
%s consumers that received nothing : %d
|
||||
|
||||
The topic has no partitions and five consumers can still share the work --
|
||||
in Kafka the same shape needs at least five partitions, chosen when the
|
||||
topic was created. But the default receiver queue is 1000 messages, so the
|
||||
first consumers to connect pull the whole 40-message backlog into their own
|
||||
buffers before the rest ask for anything, and the subscription looks
|
||||
broken. Which consumers win is a race and varies between runs -- one
|
||||
consumer taking all forty, or two taking twenty each -- but the consumers
|
||||
that lose it see nothing at all. This is the same trap as RabbitMQ's
|
||||
unbounded prefetch, with a different name and a much larger default.
|
||||
""".formatted(render2(withDefaultQueue), idleDefault,
|
||||
render2(withQueueOfOne), idleTuned));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- helpers
|
||||
|
||||
private void produce(String topic, int count) throws Exception {
|
||||
String[] keys = { "A", "B", "C" };
|
||||
try (Producer<String> producer = client.newProducer(Schema.STRING).topic(topic).create()) {
|
||||
for (int i = 1; i <= count; i++) {
|
||||
producer.newMessage().key(keys[i % keys.length]).value(String.valueOf(i)).send();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type)
|
||||
throws Exception {
|
||||
return consumer(topic, subscription, type, 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param receiverQueueSize 0 leaves Pulsar's default of 1000 in place; anything else sets it.
|
||||
* The default matters more than it looks: it is how many messages a single consumer will pull
|
||||
* into its own buffer before another consumer on the same subscription gets a chance.
|
||||
*/
|
||||
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type,
|
||||
int receiverQueueSize) throws Exception {
|
||||
var builder = client.newConsumer(Schema.STRING)
|
||||
.topic(topic)
|
||||
.subscriptionName(subscription)
|
||||
.subscriptionType(type)
|
||||
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
|
||||
if (receiverQueueSize > 0) {
|
||||
builder = builder.receiverQueueSize(receiverQueueSize);
|
||||
}
|
||||
return builder.subscribe();
|
||||
}
|
||||
|
||||
private List<String> drain(Consumer<String> consumer, int expected) throws Exception {
|
||||
List<String> received = new ArrayList<>();
|
||||
while (received.size() < expected) {
|
||||
Message<String> message = consumer.receive(2, TimeUnit.SECONDS);
|
||||
if (message == null) {
|
||||
break;
|
||||
}
|
||||
received.add(message.getValue());
|
||||
consumer.acknowledge(message);
|
||||
}
|
||||
return received;
|
||||
}
|
||||
|
||||
/** Which consumers saw each key, with a receiver queue of one so nothing hoards. */
|
||||
private Map<String, Set<String>> consumersPerKey(String topic, String subscription,
|
||||
SubscriptionType type, int expected) throws Exception {
|
||||
Map<String, Set<String>> perKey = new LinkedHashMap<>();
|
||||
List<Consumer<String>> consumers = new ArrayList<>();
|
||||
try {
|
||||
consumers.add(consumer(topic, subscription, type, 1));
|
||||
consumers.add(consumer(topic, subscription, type, 1));
|
||||
int received = 0;
|
||||
while (received < expected) {
|
||||
boolean any = false;
|
||||
for (int i = 0; i < consumers.size(); i++) {
|
||||
Message<String> message = consumers.get(i).receive(500, TimeUnit.MILLISECONDS);
|
||||
if (message != null) {
|
||||
perKey.computeIfAbsent(message.getKey(), (k) -> new java.util.TreeSet<>())
|
||||
.add("consumer-" + (i + 1));
|
||||
consumers.get(i).acknowledge(message);
|
||||
received++;
|
||||
any = true;
|
||||
}
|
||||
}
|
||||
if (!any) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
finally {
|
||||
for (Consumer<String> consumer : consumers) {
|
||||
consumer.close();
|
||||
}
|
||||
}
|
||||
return perKey;
|
||||
}
|
||||
|
||||
/** Drains a Shared subscription across {@code n} consumers and reports the distribution. */
|
||||
private Map<String, Integer> drainAcross(String topic, String subscription, int n, int expected,
|
||||
int receiverQueueSize) throws Exception {
|
||||
Map<String, Integer> counts = new LinkedHashMap<>();
|
||||
List<Consumer<String>> consumers = new ArrayList<>();
|
||||
try {
|
||||
for (int i = 1; i <= n; i++) {
|
||||
consumers.add(consumer(topic, subscription, SubscriptionType.Shared, receiverQueueSize));
|
||||
counts.put("consumer-" + i, 0);
|
||||
}
|
||||
int received = 0;
|
||||
while (received < expected) {
|
||||
boolean any = false;
|
||||
for (int i = 0; i < consumers.size(); i++) {
|
||||
Message<String> message = consumers.get(i).receive(300, TimeUnit.MILLISECONDS);
|
||||
if (message != null) {
|
||||
consumers.get(i).acknowledge(message);
|
||||
counts.merge("consumer-" + (i + 1), 1, Integer::sum);
|
||||
received++;
|
||||
any = true;
|
||||
}
|
||||
}
|
||||
if (!any) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
finally {
|
||||
for (Consumer<String> consumer : consumers) {
|
||||
consumer.close();
|
||||
}
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
private static String render3(Map<String, Set<String>> perKey) {
|
||||
StringBuilder out = new StringBuilder();
|
||||
perKey.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
|
||||
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
|
||||
.append('\n'));
|
||||
return out.toString();
|
||||
}
|
||||
|
||||
private static String render2(Map<String, Integer> counts) {
|
||||
StringBuilder out = new StringBuilder();
|
||||
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
|
||||
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
|
||||
.append(" messages\n"));
|
||||
return out.toString();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,286 @@
|
||||
package com.ankurm.brokers;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import com.rabbitmq.client.AMQP;
|
||||
import com.rabbitmq.client.Channel;
|
||||
import com.rabbitmq.client.Connection;
|
||||
import com.rabbitmq.client.ConnectionFactory;
|
||||
import com.rabbitmq.client.DefaultConsumer;
|
||||
import com.rabbitmq.client.Envelope;
|
||||
import com.rabbitmq.client.GetResponse;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* RabbitMQ measured against a real broker on localhost:5672, started by
|
||||
* scripts/rabbit-broker.sh.
|
||||
*
|
||||
* <p>The same three questions as the Kafka measurements, against a broker that owns the message
|
||||
* until it is acknowledged rather than a log that keeps it.
|
||||
*/
|
||||
@Tag("rabbit")
|
||||
class RabbitComparisonTest {
|
||||
|
||||
private static Connection connection;
|
||||
|
||||
@BeforeAll
|
||||
static void connect() throws Exception {
|
||||
ConnectionFactory factory = new ConnectionFactory();
|
||||
factory.setHost("127.0.0.1");
|
||||
factory.setPort(5672);
|
||||
connection = factory.newConnection();
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void close() throws Exception {
|
||||
connection.close();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- ordering
|
||||
|
||||
@Test
|
||||
void oneConsumerIsFifoAndTwoConsumersAreNot() throws Exception {
|
||||
String single = declare("order-single");
|
||||
publish(single, 12);
|
||||
List<String> fifo = getAll(single, 12);
|
||||
|
||||
String shared = declare("order-shared");
|
||||
publish(shared, 12);
|
||||
List<String> completionOrder = consumeWithTwoConsumers(shared, 12);
|
||||
|
||||
assertThat(fifo).containsExactly("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12");
|
||||
|
||||
int inversions = 0;
|
||||
for (int i = 1; i < completionOrder.size(); i++) {
|
||||
if (Integer.parseInt(completionOrder.get(i)) < Integer.parseInt(completionOrder.get(i - 1))) {
|
||||
inversions++;
|
||||
}
|
||||
}
|
||||
|
||||
Capture.write("rabbit-ordering.txt", "RabbitMQ: FIFO per queue, per consumer",
|
||||
"""
|
||||
one queue, one consumer, 12 messages
|
||||
completion order : %s
|
||||
|
||||
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
|
||||
completion order : %s
|
||||
out-of-order steps: %d
|
||||
|
||||
A queue is FIFO and a single consumer sees it that way. The moment a second
|
||||
consumer is added the broker hands the next message to whichever consumer
|
||||
is free, so the order in which work finishes is no longer the order in
|
||||
which it was published. There is no key: RabbitMQ has no notion of a
|
||||
partition to which related messages could be pinned. Ordering across
|
||||
related messages means one queue and one consumer, and therefore no
|
||||
horizontal scaling for that queue -- or a consistent-hash exchange, which
|
||||
is a plugin.
|
||||
""".formatted(fifo, completionOrder, inversions));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- replay
|
||||
|
||||
@Test
|
||||
void anAcknowledgedMessageIsGone() throws Exception {
|
||||
String queue = declare("replay-probe");
|
||||
publish(queue, 12);
|
||||
|
||||
List<String> first = getAll(queue, 12);
|
||||
List<String> second = getAll(queue, 12);
|
||||
|
||||
int depth = depth(queue);
|
||||
|
||||
assertThat(first).hasSize(12);
|
||||
assertThat(second).isEmpty();
|
||||
assertThat(depth).isZero();
|
||||
|
||||
Capture.write("rabbit-replay.txt", "RabbitMQ: there is nothing to replay",
|
||||
"""
|
||||
first drain of the queue : %d messages
|
||||
second drain of the queue : %d messages
|
||||
queue depth afterwards : %d
|
||||
|
||||
Acknowledging a message deletes it. The broker is a router with buffers,
|
||||
not a log: there is no offset to rewind and no second reader that can see
|
||||
what the first one consumed. Reading the same message twice means
|
||||
arranging it in advance -- a second queue bound to the same exchange, or a
|
||||
copy written somewhere else -- and it cannot be arranged after the fact.
|
||||
""".formatted(first.size(), second.size(), depth));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- consumer scaling
|
||||
|
||||
@Test
|
||||
void consumersScaleWithoutAPartitionCeilingButPrefetchDecidesWho() throws Exception {
|
||||
String unlimited = declare("scaling-unlimited");
|
||||
publish(unlimited, 40);
|
||||
Map<String, Integer> withDefaultPrefetch = drainAcross(unlimited, 5, 40, 0);
|
||||
|
||||
String throttled = declare("scaling-prefetch-1");
|
||||
publish(throttled, 40);
|
||||
Map<String, Integer> withPrefetchOne = drainAcross(throttled, 5, 40, 1);
|
||||
|
||||
long idleDefault = withDefaultPrefetch.values().stream().filter((n) -> n == 0).count();
|
||||
long idleTuned = withPrefetchOne.values().stream().filter((n) -> n == 0).count();
|
||||
|
||||
assertThat(idleTuned).isZero();
|
||||
|
||||
Capture.write("rabbit-consumer-scaling.txt",
|
||||
"RabbitMQ: consumers scale, and prefetch decides whether they actually do",
|
||||
"""
|
||||
one queue, 40 messages, 5 consumers, each taking 10 ms per message
|
||||
|
||||
no basicQos at all (unlimited prefetch, the AMQP default):
|
||||
%s consumers that received nothing : %d
|
||||
|
||||
basicQos(1):
|
||||
%s consumers that received nothing : %d
|
||||
|
||||
Every consumer on a queue competes for the same messages, so adding
|
||||
consumers adds throughput and there is no structural ceiling of the kind
|
||||
Kafka's partition count imposes. But with the AMQP default the broker
|
||||
pushes as many messages as a consumer will take, so whichever consumer
|
||||
connects first can be handed the entire backlog while the others sit idle.
|
||||
basicQos is not a tuning knob you get to postpone; it is what makes the
|
||||
fan-out real. Spring AMQP sets it for you --
|
||||
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
|
||||
better than unlimited and still large enough to concentrate a small
|
||||
backlog on one consumer.
|
||||
|
||||
The price of all this is the ordering measurement: there is no key, so
|
||||
nothing constrains related messages to one consumer.
|
||||
""".formatted(render(withDefaultPrefetch), idleDefault,
|
||||
render(withPrefetchOne), idleTuned));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- helpers
|
||||
|
||||
private String declare(String name) throws Exception {
|
||||
try (Channel channel = connection.createChannel()) {
|
||||
channel.queueDelete(name);
|
||||
channel.queueDeclare(name, false, false, true, null);
|
||||
}
|
||||
return name;
|
||||
}
|
||||
|
||||
private void publish(String queue, int count) throws Exception {
|
||||
try (Channel channel = connection.createChannel()) {
|
||||
for (int i = 1; i <= count; i++) {
|
||||
channel.basicPublish("", queue, null, String.valueOf(i).getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private List<String> getAll(String queue, int max) throws Exception {
|
||||
List<String> received = new ArrayList<>();
|
||||
try (Channel channel = connection.createChannel()) {
|
||||
for (int i = 0; i < max; i++) {
|
||||
GetResponse response = channel.basicGet(queue, true);
|
||||
if (response == null) {
|
||||
break;
|
||||
}
|
||||
received.add(new String(response.getBody(), StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
return received;
|
||||
}
|
||||
|
||||
private int depth(String queue) throws Exception {
|
||||
try (Channel channel = connection.createChannel()) {
|
||||
return channel.queueDeclarePassive(queue).getMessageCount();
|
||||
}
|
||||
}
|
||||
|
||||
private List<String> consumeWithTwoConsumers(String queue, int count) throws Exception {
|
||||
List<String> completion = new CopyOnWriteArrayList<>();
|
||||
CountDownLatch done = new CountDownLatch(count);
|
||||
List<Channel> channels = new ArrayList<>();
|
||||
for (int i = 1; i <= 2; i++) {
|
||||
long delay = (i == 1) ? 60 : 5;
|
||||
Channel channel = connection.createChannel();
|
||||
channel.basicQos(1);
|
||||
channels.add(channel);
|
||||
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
|
||||
@Override
|
||||
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
|
||||
byte[] body) throws IOException {
|
||||
try {
|
||||
Thread.sleep(delay);
|
||||
}
|
||||
catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
completion.add(new String(body, StandardCharsets.UTF_8));
|
||||
getChannel().basicAck(envelope.getDeliveryTag(), false);
|
||||
done.countDown();
|
||||
}
|
||||
});
|
||||
}
|
||||
done.await(30, TimeUnit.SECONDS);
|
||||
for (Channel channel : channels) {
|
||||
channel.close();
|
||||
}
|
||||
return List.copyOf(completion);
|
||||
}
|
||||
|
||||
/** Drains a queue across {@code n} competing consumers. prefetch 0 means "do not set it". */
|
||||
private Map<String, Integer> drainAcross(String queue, int n, int expected, int prefetch)
|
||||
throws Exception {
|
||||
Map<String, AtomicInteger> perConsumer = new ConcurrentHashMap<>();
|
||||
CountDownLatch done = new CountDownLatch(expected);
|
||||
List<Channel> channels = new ArrayList<>();
|
||||
for (int i = 1; i <= n; i++) {
|
||||
String name = "consumer-" + i;
|
||||
Channel channel = connection.createChannel();
|
||||
if (prefetch > 0) {
|
||||
channel.basicQos(prefetch);
|
||||
}
|
||||
channels.add(channel);
|
||||
perConsumer.put(name, new AtomicInteger());
|
||||
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
|
||||
@Override
|
||||
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
|
||||
byte[] body) throws IOException {
|
||||
perConsumer.get(name).incrementAndGet();
|
||||
try {
|
||||
Thread.sleep(10);
|
||||
}
|
||||
catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
getChannel().basicAck(envelope.getDeliveryTag(), false);
|
||||
done.countDown();
|
||||
}
|
||||
});
|
||||
}
|
||||
done.await(30, TimeUnit.SECONDS);
|
||||
for (Channel channel : channels) {
|
||||
channel.close();
|
||||
}
|
||||
Map<String, Integer> counts = new LinkedHashMap<>();
|
||||
perConsumer.forEach((name, count) -> counts.put(name, count.get()));
|
||||
return counts;
|
||||
}
|
||||
|
||||
private static String render(Map<String, Integer> counts) {
|
||||
StringBuilder out = new StringBuilder();
|
||||
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
|
||||
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
|
||||
.append(" messages\n"));
|
||||
return out.toString();
|
||||
}
|
||||
}
|
||||
59
kafka-error-handling/README.md
Normal file
59
kafka-error-handling/README.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# `kafka-error-handling` — retries, DLT and poison pills
|
||||
|
||||
Companion project for
|
||||
[**Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills**](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/)
|
||||
on ankurm.com.
|
||||
|
||||
Six tests against a **real Kafka broker** started in-process in KRaft mode. No Docker, no local
|
||||
install. `./scripts/run-all.sh` regenerates everything under [`docs/output/`](docs/output/).
|
||||
|
||||
## Versions
|
||||
|
||||
| | Version |
|
||||
|---|---|
|
||||
| JDK | 25 (Temurin 25.0.4.1+1) |
|
||||
| Spring Boot | 4.1.1 |
|
||||
| Spring Kafka | 4.1.1 |
|
||||
| kafka-clients | 4.2.1 (Boot-managed) |
|
||||
|
||||
## Profiles
|
||||
|
||||
| Profile | What it wires |
|
||||
|---|---|
|
||||
| `dlt` | `DefaultErrorHandler` + `DeadLetterPublishingRecoverer`, `FixedBackOff(1000, 2)`, `PermanentFailure` classified non-retryable |
|
||||
| `dltbytes` | the same, with a `byte[]`-aware template map so poison pills keep their original bytes |
|
||||
| `retrytopic` | `@RetryableTopic` non-blocking retries with a `@DltHandler` |
|
||||
| `defaults` | the stock `DefaultErrorHandler`, for reading its behaviour |
|
||||
|
||||
## Documentation
|
||||
|
||||
1. [Two kinds of failure, and why they need different machinery](docs/01-two-kinds-of-failure.md)
|
||||
2. [What the default actually does](docs/02-default-error-handler.md)
|
||||
3. [Poison pills](docs/03-poison-pills.md)
|
||||
4. [The dead-letter topic](docs/04-the-dlt.md)
|
||||
5. [Non-blocking retries with `@RetryableTopic`](docs/05-retryable-topic.md)
|
||||
|
||||
## Captured output
|
||||
|
||||
| File | Shows |
|
||||
|---|---|
|
||||
| [`default-backoff.txt`](docs/output/default-backoff.txt) | ten deliveries, zero delay |
|
||||
| [`retry-and-dlt.txt`](docs/output/retry-and-dlt.txt) | measured back-off and the DLT headers |
|
||||
| [`poison-pill.txt`](docs/output/poison-pill.txt) | base64 payload, and the byte-aware fix |
|
||||
| [`retry-topics.txt`](docs/output/retry-topics.txt) | the non-blocking delivery trace |
|
||||
| [`tests.txt`](docs/output/tests.txt) | 6 tests |
|
||||
|
||||
## Six things this module exists to prove
|
||||
|
||||
1. **The default is ten deliveries, zero milliseconds apart, then the record is dropped.** Not
|
||||
"retry with backoff", and not "dead-letter".
|
||||
2. **The DLT suffix is `-dlt`, not `.DLT`.** Get it wrong and the recoverer logs a WARN and the
|
||||
record is lost — your safety net silently drops it.
|
||||
3. **`kafka_dlt-exception-fqcn` is always `ListenerExecutionFailedException`** for listener
|
||||
failures. The useful header is `-exception-cause-fqcn`.
|
||||
4. **A poison pill reaches the DLT base64-encoded**, because the recoverer reuses the JSON
|
||||
producer. A per-type template map fixes it; the module shows both transcripts.
|
||||
5. **`@RetryableTopic` names retry topics by delay** — `-retry-500`, `-retry-1000` — so changing
|
||||
the multiplier renames them.
|
||||
6. **`@Backoff` from spring-retry no longer exists here.** Spring Kafka 4 ships its own
|
||||
`@BackOff`, and the attribute is `backOff`.
|
||||
53
kafka-error-handling/docs/01-two-kinds-of-failure.md
Normal file
53
kafka-error-handling/docs/01-two-kinds-of-failure.md
Normal file
@@ -0,0 +1,53 @@
|
||||
[Module README](../README.md) · [DefaultErrorHandler →](02-default-error-handler.md)
|
||||
|
||||
# 1. Two kinds of failure, and why they need different machinery
|
||||
|
||||
Kafka delivery is at-least-once. The container commits offsets after your listener returns
|
||||
([the basics article](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/)
|
||||
covers why), so if the listener throws, the offset does not move and the record comes back.
|
||||
Everything in this module is about what happens next.
|
||||
|
||||
There are two failures, and conflating them is why a bad record can take a partition down for
|
||||
hours.
|
||||
|
||||
## Failure inside the listener
|
||||
|
||||
Your code threw. The record deserialized fine; the work failed. The container catches it, hands
|
||||
it to a `CommonErrorHandler`, and that decides whether to retry, how long to wait, and what to do
|
||||
when the attempts run out.
|
||||
|
||||
This splits again, and the split matters more than any back-off setting:
|
||||
|
||||
| | example | retrying it |
|
||||
|---|---|---|
|
||||
| **transient** | timeout, 503, deadlock, connection reset | may succeed |
|
||||
| **permanent** | validation failure, missing entity, malformed field | will fail identically |
|
||||
|
||||
Retrying a permanent failure ten times buys nothing and costs ten times the latency plus nine
|
||||
misleading log lines. Spring Kafka lets you say so:
|
||||
|
||||
```java
|
||||
handler.addNotRetryableExceptions(PermanentFailure.class);
|
||||
```
|
||||
|
||||
That single line is worth more than tuning the back-off, and it is the one most people skip.
|
||||
|
||||
## Failure before the listener
|
||||
|
||||
The bytes on the topic are not what the deserializer expects. Someone changed a schema, or
|
||||
published with a different serializer, or your `__TypeId__` header names a class you do not
|
||||
trust.
|
||||
|
||||
This one is nastier, because it happens **inside `poll()`**, before any listener exists to throw
|
||||
from. There is no error handler in the path. The consumer cannot advance past the offset,
|
||||
retries the same record on the next poll, fails again, and does that forever — at whatever rate
|
||||
the poll loop runs. The partition is stopped and the only symptom is a growing lag with a
|
||||
consumer that looks healthy.
|
||||
|
||||
That is a **poison pill**, and the cure is a different mechanism from the one above:
|
||||
`ErrorHandlingDeserializer`, covered in [chapter 3](03-poison-pills.md).
|
||||
|
||||
Keeping these two apart is the whole point of this module. One needs a retry policy; the other
|
||||
needs a wrapper around the deserializer. Neither fixes the other.
|
||||
|
||||
[DefaultErrorHandler →](02-default-error-handler.md)
|
||||
71
kafka-error-handling/docs/02-default-error-handler.md
Normal file
71
kafka-error-handling/docs/02-default-error-handler.md
Normal file
@@ -0,0 +1,71 @@
|
||||
[← Two kinds of failure](01-two-kinds-of-failure.md) · [Module README](../README.md) · [Poison pills →](03-poison-pills.md)
|
||||
|
||||
# 2. What the default actually does
|
||||
|
||||
If you configure nothing, the container factory installs a `DefaultErrorHandler` with
|
||||
`SeekUtils.DEFAULT_BACK_OFF` and a recoverer that logs. Run the back-off and read it off
|
||||
([`docs/output/default-backoff.txt`](output/default-backoff.txt)):
|
||||
|
||||
```
|
||||
=== DefaultErrorHandler default back-off ===
|
||||
interval 0 ms
|
||||
max attempts 9 retries
|
||||
SeekUtils.DEFAULT_MAX_FAILURES = 10
|
||||
retry intervals [0, 0, 0, 0, 0, 0, 0, 0, 0]
|
||||
total deliveries 10
|
||||
```
|
||||
|
||||
**Ten deliveries, zero milliseconds apart, and then the record is dropped.**
|
||||
|
||||
Both halves of that surprise people. It is not "retry with backoff" — it is ten immediate
|
||||
attempts as fast as the consumer thread can run them, which against a downstream that is
|
||||
overloaded is ten times the load at the worst moment. And "then dropped" means exactly that: the
|
||||
default recoverer logs the failure and the offset moves on. There is no dead-letter topic unless
|
||||
you make one.
|
||||
|
||||
## Giving it a back-off and a destination
|
||||
|
||||
```java
|
||||
@Bean
|
||||
DefaultErrorHandler errorHandler(KafkaOperations<String, Object> template) {
|
||||
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
|
||||
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
|
||||
handler.addNotRetryableExceptions(PermanentFailure.class);
|
||||
return handler;
|
||||
}
|
||||
```
|
||||
|
||||
`FixedBackOff(1000L, 2)` is one delivery plus two retries. Measured
|
||||
([`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt)):
|
||||
|
||||
```
|
||||
=== transient failure ===
|
||||
deliveries 3
|
||||
gap between 1&2 1007 ms (FixedBackOff interval 1000)
|
||||
```
|
||||
|
||||
and the classified permanent failure gets exactly one delivery before going to the DLT.
|
||||
|
||||
**`ExponentialBackOffWithMaxRetries` is usually the better choice** than `FixedBackOff` for a
|
||||
transient downstream, because a fixed interval synchronises every consumer in the group into
|
||||
retrying at the same instant.
|
||||
|
||||
## The cost of blocking retries
|
||||
|
||||
`DefaultErrorHandler` retries **on the consumer thread**. For the whole back-off, that partition
|
||||
processes nothing else. `FixedBackOff(1000L, 2)` is three seconds of a stalled partition per
|
||||
failing record — fine. A one-minute exponential back-off over five attempts is five minutes, and
|
||||
if failures are correlated you have a stalled consumer group, not a retry policy.
|
||||
|
||||
Two consequences worth planning for:
|
||||
|
||||
- **`max.poll.interval.ms` is your ceiling.** Default five minutes. Block longer than that
|
||||
between polls and the broker evicts the consumer from the group, triggering a rebalance —
|
||||
which usually makes things worse. A back-off schedule that can exceed it is a bug.
|
||||
- **Ordering is preserved**, which is the one thing blocking retries give you that
|
||||
[retry topics](05-retryable-topic.md) do not.
|
||||
|
||||
That trade — ordering versus throughput under failure — is the real decision, and it is covered
|
||||
in [chapter 5](05-retryable-topic.md).
|
||||
|
||||
[Poison pills →](03-poison-pills.md)
|
||||
95
kafka-error-handling/docs/03-poison-pills.md
Normal file
95
kafka-error-handling/docs/03-poison-pills.md
Normal file
@@ -0,0 +1,95 @@
|
||||
[← DefaultErrorHandler](02-default-error-handler.md) · [Module README](../README.md) · [The DLT →](04-the-dlt.md)
|
||||
|
||||
# 3. Poison pills
|
||||
|
||||
A record whose bytes cannot be deserialized fails inside `poll()`, before any listener exists.
|
||||
No error handler is in the path. The offset cannot advance, so the next poll fetches the same
|
||||
record and fails identically. Forever.
|
||||
|
||||
The consumer is up, the group is stable, no exception reaches your code, and lag grows. It is
|
||||
one of the few Kafka failures with no good symptom.
|
||||
|
||||
## `ErrorHandlingDeserializer`
|
||||
|
||||
```yaml
|
||||
spring:
|
||||
kafka:
|
||||
consumer:
|
||||
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
|
||||
properties:
|
||||
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
|
||||
```
|
||||
|
||||
It wraps the real deserializer, catches the failure, and returns a **null value with the
|
||||
exception in a header**. The record then flows normally into the container, the listener is
|
||||
skipped, and the error handler gets a record it can recover — which is to say, the poison pill
|
||||
becomes an ordinary failure.
|
||||
|
||||
Use `spring.deserializer.key.delegate.class` for keys. A malformed key is rarer and just as
|
||||
fatal.
|
||||
|
||||
## What arrives on the DLT
|
||||
|
||||
From [`docs/output/poison-pill.txt`](output/poison-pill.txt):
|
||||
|
||||
```
|
||||
kafka_dlt-exception-fqcn org.springframework.kafka.support.serializer.DeserializationException
|
||||
kafka_dlt-exception-cause-fqcn org.springframework.kafka.support.serializer.DeserializationException
|
||||
kafka_dlt-exception-message failed to deserialize
|
||||
kafka_dlt-original-topic payments
|
||||
kafka_dlt-original-partition 0x00000000
|
||||
kafka_dlt-original-offset 0x0000000000000000
|
||||
kafka_dlt-original-consumer-group payments
|
||||
__TypeId__ [B
|
||||
DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
|
||||
```
|
||||
|
||||
The listener was never invoked — asserted in the test — and the record is off the partition,
|
||||
which is the whole win.
|
||||
|
||||
Two details in that block are worth stopping on.
|
||||
|
||||
**`original-partition` and `original-offset` are binary.** They are big-endian `int` and `long`,
|
||||
not text. Printing them as a string gives you mojibake, which is why the transcript renders them
|
||||
as hex. A DLT tool that treats every header as UTF-8 will show garbage for exactly the three
|
||||
fields you need in order to find the original record.
|
||||
|
||||
**The payload is base64.** `"eyB0aGlzIGlzIG5vdCBqc29u"` decodes to `{ this is not json`. The
|
||||
recoverer publishes with the **application's** producer, whose value serializer is
|
||||
`JacksonJsonSerializer`; the failed value is a `byte[]`; Jackson writes a `byte[]` as a base64
|
||||
JSON string. So the DLT does not hold what arrived — it holds base64 of it, wrapped in quotes.
|
||||
|
||||
Replaying that topic naively republishes a quoted base64 string, which fails to deserialize, and
|
||||
now you have a poison pill in your poison-pill queue.
|
||||
|
||||
## The fix
|
||||
|
||||
Give the recoverer a template per value type:
|
||||
|
||||
```java
|
||||
Map<Class<?>, KafkaOperations<?, ?>> templates = new LinkedHashMap<>();
|
||||
templates.put(byte[].class, byteTemplate); // ByteArraySerializer
|
||||
templates.put(Object.class, jsonTemplate);
|
||||
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);
|
||||
```
|
||||
|
||||
```
|
||||
=== byte-aware DLT ===
|
||||
DLT payload -> { this is not json
|
||||
```
|
||||
|
||||
Byte for byte what was published. Replay is now a copy from one topic to another.
|
||||
|
||||
Three things had to be right to get there, and each failed first — they are commented in
|
||||
[`ErrorHandlerConfiguration`](../src/main/java/com/ankurm/kafkaerrors/ErrorHandlerConfiguration.java):
|
||||
|
||||
1. **`KafkaAutoConfiguration`'s template is `@ConditionalOnMissingBean(KafkaTemplate.class)`.**
|
||||
Declaring `byteTemplate` removed the auto-configured `KafkaTemplate` from the context
|
||||
entirely. Once you declare one template, you own all of them.
|
||||
2. With two templates present, `KafkaOperations<String, Object>` stops resolving, and
|
||||
`@Qualifier` alone does **not** rescue it — the generic check runs first. Use
|
||||
`KafkaOperations<?, ?>`, which is what the recoverer's constructor wants anyway.
|
||||
3. The map is `Map<Class<?>, KafkaOperations<?, ?>>`, matched by value type, with
|
||||
`Object.class` as the fallback.
|
||||
|
||||
[The DLT →](04-the-dlt.md)
|
||||
74
kafka-error-handling/docs/04-the-dlt.md
Normal file
74
kafka-error-handling/docs/04-the-dlt.md
Normal file
@@ -0,0 +1,74 @@
|
||||
[← Poison pills](03-poison-pills.md) · [Module README](../README.md) · [Retry topics →](05-retryable-topic.md)
|
||||
|
||||
# 4. The dead-letter topic
|
||||
|
||||
## The suffix is `-dlt`, not `.DLT`
|
||||
|
||||
```java
|
||||
public static final String RetryTopicConstants.DEFAULT_RETRY_SUFFIX = "-retry";
|
||||
public static final String RetryTopicConstants.DEFAULT_DLT_SUFFIX = "-dlt";
|
||||
```
|
||||
|
||||
Older Spring Kafka used `.DLT`, and most of the material online still says so. Getting it wrong
|
||||
is not an exception — it is this, at WARN, once per record:
|
||||
|
||||
```
|
||||
o.s.k.l.DeadLetterPublishingRecoverer : Destination resolver returned non-existent partition
|
||||
payments-dlt-0, KafkaProducer will determine partition to use for this topic
|
||||
[Producer] ... {payments-dlt=UNKNOWN_TOPIC_OR_PARTITION}
|
||||
```
|
||||
|
||||
and then, on a cluster with auto-topic-creation disabled, the record is **gone**. Your safety net
|
||||
dropped it and logged a warning. This module's tests were written against `payments.DLT` first
|
||||
and failed exactly this way.
|
||||
|
||||
Two things follow: pre-create your DLT topics as part of provisioning, and alert on that WARN.
|
||||
|
||||
## Same partition by default
|
||||
|
||||
`DeadLetterPublishingRecoverer` publishes to the **same partition number** as the original. If
|
||||
your DLT has fewer partitions than the source topic, records from the high-numbered partitions
|
||||
have nowhere to go. Either give the DLT the same partition count, or set
|
||||
|
||||
```java
|
||||
recoverer.setPartitionResolver((record, ex) -> null); // let the producer choose
|
||||
```
|
||||
|
||||
## The headers, and the one that will mislead you
|
||||
|
||||
From [`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt):
|
||||
|
||||
```
|
||||
kafka_dlt-exception-fqcn org.springframework.kafka.listener.ListenerExecutionFailedException
|
||||
kafka_dlt-exception-cause-fqcn com.ankurm.kafkaerrors.Failures$TransientFailure
|
||||
kafka_dlt-original-topic payments
|
||||
kafka_dlt-original-consumer-group payments
|
||||
```
|
||||
|
||||
**`kafka_dlt-exception-fqcn` is always the wrapper** for a listener failure. Build a DLT triage
|
||||
dashboard grouped by that header and every failure in the estate lands in one bucket called
|
||||
`ListenerExecutionFailedException`. The field you want is `kafka_dlt-exception-cause-fqcn`.
|
||||
|
||||
(For a deserialization failure there is no wrapper, so the two headers agree. That inconsistency
|
||||
is worth knowing if you are writing a tool over them.)
|
||||
|
||||
`kafka_dlt-original-consumer-group` is the one that saves you when several groups consume the
|
||||
same topic and share a DLT.
|
||||
|
||||
## Replay
|
||||
|
||||
A DLT is only useful if you can put records back. The mechanics are a copy:
|
||||
|
||||
1. read from `<topic>-dlt` with a **byte-array** deserializer — the payload may be the thing that
|
||||
could not be deserialized
|
||||
2. read `kafka_dlt-original-topic` and `kafka_dlt-original-consumer-group` to decide where it
|
||||
belongs and whether it is yours
|
||||
3. republish to the original topic, **stripping the `kafka_dlt-*` headers** so a second failure
|
||||
is not confused with the first
|
||||
4. do it deliberately, in bounded batches, after the cause is fixed
|
||||
|
||||
Automatic replay is almost always wrong: the records are on the DLT precisely because something
|
||||
was not transient, and a loop that moves them back on a timer is a slow-motion outage. A replay
|
||||
you run by hand, having read the failure, is the tool worth building.
|
||||
|
||||
[Retry topics →](05-retryable-topic.md)
|
||||
89
kafka-error-handling/docs/05-retryable-topic.md
Normal file
89
kafka-error-handling/docs/05-retryable-topic.md
Normal file
@@ -0,0 +1,89 @@
|
||||
[← The DLT](04-the-dlt.md) · [Module README](../README.md)
|
||||
|
||||
# 5. Non-blocking retries with `@RetryableTopic`
|
||||
|
||||
Blocking retries stall the partition. `@RetryableTopic` republishes the failed record to a
|
||||
separate topic and lets the main partition carry on.
|
||||
|
||||
```java
|
||||
@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
|
||||
sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
|
||||
exclude = Failures.PermanentFailure.class)
|
||||
@KafkaListener(topics = "invoices", groupId = "invoices")
|
||||
public void onInvoice(ConsumerRecord<String, Payment> record, ...) { ... }
|
||||
```
|
||||
|
||||
**Two API changes in Spring Kafka 4.x will stop older examples compiling:**
|
||||
|
||||
- the attribute is **`backOff`**, not `backoff`
|
||||
- the annotation is **`org.springframework.kafka.annotation.BackOff`**, not
|
||||
`org.springframework.retry.annotation.Backoff`. Spring Kafka 4 dropped the spring-retry
|
||||
dependency and brought its own.
|
||||
|
||||
The failure is `package org.springframework.retry.annotation does not exist`, which reads like a
|
||||
missing dependency and is not.
|
||||
|
||||
Also new in 4.1: `sameIntervalTopicReuseStrategy` defaults to `SINGLE_TOPIC` in
|
||||
`RetryTopicConfigurationBuilder`, aligning it with the annotation's default.
|
||||
|
||||
## What it actually does
|
||||
|
||||
From [`docs/output/retry-topics.txt`](output/retry-topics.txt) — a failing record and a good one
|
||||
published back to back on the same partition:
|
||||
|
||||
```
|
||||
=== @RetryableTopic delivery trace ===
|
||||
+0 ms invoices transient-1
|
||||
+531 ms invoices-retry-500 transient-1
|
||||
+550 ms invoices ok-1
|
||||
+1554 ms invoices-retry-1000 transient-1
|
||||
+3560 ms invoices-retry-2000 transient-1
|
||||
DLT: [transient-1 on invoices-dlt]
|
||||
```
|
||||
|
||||
Read the third line. `ok-1` was processed at +550 ms, while `transient-1` was still two retries
|
||||
from giving up. With a blocking handler it would have waited for the whole schedule.
|
||||
|
||||
**Retry topics are named by the delay, not the attempt number.** `invoices-retry-500`,
|
||||
`invoices-retry-1000`, `invoices-retry-2000` — that is
|
||||
`TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE`, the default. So provisioning topics ahead of
|
||||
time means knowing your whole back-off schedule in advance, and **changing the multiplier changes
|
||||
the topic names**, orphaning whatever is still sitting in the old ones. Deploy that change the
|
||||
way you would a rename.
|
||||
|
||||
## The cost
|
||||
|
||||
**Per-key ordering is gone for any record that fails.** That is not a side effect; it is the
|
||||
mechanism. If `invoice-7` fails and `invoice-7`'s next event succeeds, they are processed out of
|
||||
order, and no configuration prevents it.
|
||||
|
||||
So the decision is not "blocking or non-blocking", it is:
|
||||
|
||||
| | blocking (`DefaultErrorHandler`) | non-blocking (`@RetryableTopic`) |
|
||||
|---|---|---|
|
||||
| ordering under failure | preserved | lost for the failing key |
|
||||
| partition throughput under failure | stalled | unaffected |
|
||||
| topics to provision | 1 + DLT | 1 + one per distinct delay + DLT |
|
||||
| long back-offs | limited by `max.poll.interval.ms` | unlimited |
|
||||
|
||||
If your consumer is idempotent and order-insensitive — most notification, indexing and cache-warm
|
||||
consumers are — retry topics are strictly better. If it applies state transitions per key,
|
||||
blocking retries with a short schedule and a fast DLT are usually the safer answer.
|
||||
|
||||
Use `exclude` (or `include`) rather than retrying everything: a `PermanentFailure` here skips the
|
||||
retry topics entirely and goes straight to `invoices-dlt`.
|
||||
|
||||
## `@DltHandler`
|
||||
|
||||
```java
|
||||
@DltHandler
|
||||
public void onDlt(ConsumerRecord<String, Payment> record,
|
||||
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { ... }
|
||||
```
|
||||
|
||||
Without one, the framework still creates and populates the DLT — it just logs and moves on, and
|
||||
nothing in your application has looked at the record. A `@DltHandler` that increments a counter
|
||||
and writes a structured log line is the minimum worth having, because a DLT nobody watches is a
|
||||
queue that grows until someone notices the disk.
|
||||
|
||||
[Module README](../README.md)
|
||||
6
kafka-error-handling/docs/output/default-backoff.txt
Normal file
6
kafka-error-handling/docs/output/default-backoff.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
=== DefaultErrorHandler default back-off ===
|
||||
interval 0 ms
|
||||
max attempts 9 retries
|
||||
SeekUtils.DEFAULT_MAX_FAILURES = 10
|
||||
retry intervals [0, 0, 0, 0, 0, 0, 0, 0, 0]
|
||||
total deliveries 10
|
||||
15
kafka-error-handling/docs/output/poison-pill.txt
Normal file
15
kafka-error-handling/docs/output/poison-pill.txt
Normal file
@@ -0,0 +1,15 @@
|
||||
=== poison pill on the DLT ===
|
||||
kafka_dlt-exception-fqcn org.springframework.kafka.support.serializer.DeserializationException
|
||||
kafka_dlt-exception-cause-fqcn org.springframework.kafka.support.serializer.DeserializationException
|
||||
kafka_dlt-exception-message failed to deserialize
|
||||
kafka_dlt-exception-stacktrace org.springframework.kafka.support.serializer.DeserializationException: failed to deseriali...
|
||||
kafka_dlt-original-topic payments
|
||||
kafka_dlt-original-partition 0x00000000
|
||||
kafka_dlt-original-offset 0x0000000000000000
|
||||
kafka_dlt-original-timestamp 0x000001a04bd35409
|
||||
kafka_dlt-original-timestamp-type CreateTime
|
||||
kafka_dlt-original-consumer-group payments
|
||||
__TypeId__ [B
|
||||
DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
|
||||
=== byte-aware DLT ===
|
||||
DLT payload -> { this is not json
|
||||
15
kafka-error-handling/docs/output/retry-and-dlt.txt
Normal file
15
kafka-error-handling/docs/output/retry-and-dlt.txt
Normal file
@@ -0,0 +1,15 @@
|
||||
=== transient failure ===
|
||||
deliveries 3
|
||||
gap between 1&2 1007 ms (FixedBackOff interval 1000)
|
||||
=== transient failure on the DLT ===
|
||||
kafka_dlt-exception-fqcn org.springframework.kafka.listener.ListenerExecutionFailedException
|
||||
kafka_dlt-exception-cause-fqcn com.ankurm.kafkaerrors.Failures$TransientFailure
|
||||
kafka_dlt-exception-message Listener method 'public void com.ankurm.kafkaerrors.PaymentListener.onPayment(org.apache.k...
|
||||
kafka_dlt-exception-stacktrace org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'publ...
|
||||
kafka_dlt-original-topic payments
|
||||
kafka_dlt-original-partition 0x00000000
|
||||
kafka_dlt-original-offset 0x0000000000000002
|
||||
kafka_dlt-original-timestamp 0x000001a04bd35f33
|
||||
kafka_dlt-original-timestamp-type CreateTime
|
||||
kafka_dlt-original-consumer-group payments
|
||||
__TypeId__ com.ankurm.kafkaerrors.Payment
|
||||
8
kafka-error-handling/docs/output/retry-topics.txt
Normal file
8
kafka-error-handling/docs/output/retry-topics.txt
Normal file
@@ -0,0 +1,8 @@
|
||||
=== @RetryableTopic delivery trace ===
|
||||
+0 ms invoices transient-1
|
||||
+522 ms invoices-retry-500 transient-1
|
||||
+539 ms invoices ok-1
|
||||
+1548 ms invoices-retry-1000 transient-1
|
||||
+3554 ms invoices-retry-2000 transient-1
|
||||
DLT: [transient-1 on invoices-dlt]
|
||||
topics touched: [invoices, invoices-retry-1000, invoices-retry-2000, invoices-retry-500]
|
||||
4
kafka-error-handling/docs/output/tests.txt
Normal file
4
kafka-error-handling/docs/output/tests.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.009 s -- in com.ankurm.kafkaerrors.RetryableTopicTest
|
||||
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.809 s -- in com.ankurm.kafkaerrors.DeadLetterTopicTest
|
||||
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.ankurm.kafkaerrors.DefaultBackOffTest
|
||||
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.935 s -- in com.ankurm.kafkaerrors.ByteAwareDltTest
|
||||
84
kafka-error-handling/pom.xml
Normal file
84
kafka-error-handling/pom.xml
Normal file
@@ -0,0 +1,84 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<!-- Inheriting spring-boot-starter-parent so spring-kafka, kafka-clients, Jackson and the
|
||||
test stack are all Boot-managed. Boot 4.1.1 manages Spring Kafka 4.1.1 and
|
||||
kafka-clients 4.3.1; do not pin those yourself. -->
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>kafka-error-handling</artifactId>
|
||||
<version>1.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
<!-- spring-boot-starter-kafka, NOT a bare org.springframework.kafka:spring-kafka
|
||||
dependency. In Boot 4 the auto-configuration lives in the spring-boot-kafka module
|
||||
(package org.springframework.boot.kafka.autoconfigure), which the starter brings and
|
||||
spring-kafka does not. Depending on spring-kafka alone compiles, starts, and gives you
|
||||
no KafkaTemplate bean: "No qualifying bean of type KafkaTemplate<...>". Every Boot 3
|
||||
tutorial gets this wrong now. See docs/01-the-on-ramp.md. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-kafka</artifactId>
|
||||
</dependency>
|
||||
<!-- JsonSerializer/JsonDeserializer need a Jackson ObjectMapper. Boot 4 moved to
|
||||
Jackson 3 (tools.jackson), which changes the import in your own code. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-jackson</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- The broker used by the committed transcripts: a real Kafka broker in KRaft mode,
|
||||
started in-process. No Docker required, which is why docs/output/ can be regenerated
|
||||
anywhere. See docs/07-testing.md for the Testcontainers route. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.kafka</groupId>
|
||||
<artifactId>spring-kafka-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-testcontainers</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- NOT org.testcontainers:kafka. Testcontainers 2.x renamed every module with a
|
||||
"testcontainers-" prefix, and Boot 4.1.1 imports testcontainers-bom 2.0.5, which
|
||||
manages the new name only. The old coordinate stopped at 1.21.4 and fails the build
|
||||
with "'dependencies.dependency.version' ... is missing", which does not mention the
|
||||
rename. See docs/07-testing.md. -->
|
||||
<dependency>
|
||||
<groupId>org.testcontainers</groupId>
|
||||
<artifactId>testcontainers-kafka</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
14
kafka-error-handling/scripts/run-all.sh
Executable file
14
kafka-error-handling/scripts/run-all.sh
Executable file
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerate every file under docs/output/. A real Kafka broker starts in-process in KRaft mode;
|
||||
# no Docker and no local install.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
mvn -B test 2>&1 | tr -d '\000' | tee /tmp/eh-test.log > /dev/null
|
||||
sed -n '/=== DefaultErrorHandler default back-off ===/,/total deliveries/p' /tmp/eh-test.log > docs/output/default-backoff.txt
|
||||
sed -n '/=== transient failure ===/,/gap between/p' /tmp/eh-test.log > docs/output/retry-and-dlt.txt
|
||||
sed -n '/=== transient failure on the DLT ===/,/__TypeId__/p' /tmp/eh-test.log >> docs/output/retry-and-dlt.txt
|
||||
sed -n '/=== poison pill on the DLT ===/,/DLT payload/p' /tmp/eh-test.log > docs/output/poison-pill.txt
|
||||
sed -n '/=== byte-aware DLT ===/,/DLT payload/p' /tmp/eh-test.log >> docs/output/poison-pill.txt
|
||||
sed -n '/=== @RetryableTopic delivery trace ===/,/topics touched/p' /tmp/eh-test.log > docs/output/retry-topics.txt
|
||||
grep -aE 'Tests run:.*in com\.ankurm' /tmp/eh-test.log | sed 's/^\[INFO\] //' > docs/output/tests.txt
|
||||
echo "regenerated:"; ls -1 docs/output/
|
||||
@@ -0,0 +1,114 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Profile;
|
||||
import org.springframework.kafka.core.KafkaOperations;
|
||||
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
|
||||
import org.springframework.kafka.listener.DefaultErrorHandler;
|
||||
import org.springframework.boot.kafka.autoconfigure.KafkaProperties;
|
||||
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
|
||||
import org.springframework.kafka.core.KafkaTemplate;
|
||||
import org.apache.kafka.clients.producer.ProducerConfig;
|
||||
import org.apache.kafka.common.serialization.ByteArraySerializer;
|
||||
import org.springframework.util.backoff.FixedBackOff;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* The error handler, in three configurations selected by profile.
|
||||
*
|
||||
* <p>A {@code CommonErrorHandler} bean replaces the container factory's default one wholesale.
|
||||
* That default is a {@code DefaultErrorHandler} with {@code SeekUtils.DEFAULT_BACK_OFF} and a
|
||||
* recoverer that only logs — so out of the box a record is retried and then <b>dropped</b>,
|
||||
* which is the behaviour most people are surprised by.
|
||||
*
|
||||
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
|
||||
*/
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
public class ErrorHandlerConfiguration {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(ErrorHandlerConfiguration.class);
|
||||
|
||||
/**
|
||||
* Publishes the exhausted record to {@code <topic>.DLT}. Without a recoverer bean the record
|
||||
* is logged and discarded; this is the one line that turns "we lost it" into "it is on a
|
||||
* topic we can replay".
|
||||
*/
|
||||
@Bean
|
||||
@Profile("dlt")
|
||||
DefaultErrorHandler dltErrorHandler(KafkaOperations<String, Object> template) {
|
||||
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
|
||||
// Three attempts, one second apart, so the timing is visible in a transcript.
|
||||
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
|
||||
// Classification matters more than the back-off. A PermanentFailure retried ten times is
|
||||
// ten times the latency for the same answer, and nine extra log lines that look like an
|
||||
// outage.
|
||||
handler.addNotRetryableExceptions(Failures.PermanentFailure.class);
|
||||
handler.setRetryListeners((record, ex, deliveryAttempt) ->
|
||||
log.info("retry listener: attempt {} for offset {} ({})", deliveryAttempt,
|
||||
record.offset(), ex.getClass().getSimpleName()));
|
||||
return handler;
|
||||
}
|
||||
|
||||
/**
|
||||
* The same thing, publishing raw bytes correctly.
|
||||
*
|
||||
* <p>{@code DeadLetterPublishingRecoverer} takes a map from value type to template. Give it
|
||||
* a {@code byte[]} template backed by {@code ByteArraySerializer} and a deserialization
|
||||
* failure lands on the DLT as the exact bytes that arrived, instead of as base64 of them.
|
||||
* Without this, replaying a poison-pill DLT republishes a quoted base64 string.
|
||||
*/
|
||||
// Three things had to be right here, and each one failed first:
|
||||
//
|
||||
// 1. KafkaAutoConfiguration's template is @ConditionalOnMissingBean(KafkaTemplate.class),
|
||||
// so declaring byteTemplate below REMOVES the auto-configured KafkaTemplate from the
|
||||
// context entirely. Once you declare one, you own all of them - hence jsonTemplate.
|
||||
// 2. With two templates present, KafkaOperations<String, Object> no longer resolves; the
|
||||
// generic check runs before the qualifier, so @Qualifier alone does not rescue it.
|
||||
// 3. The recoverer's constructor takes Map<Class<?>, KafkaOperations<?, ?>>, so wildcards
|
||||
// are what it wants anyway.
|
||||
@Bean
|
||||
@Profile("dltbytes")
|
||||
DefaultErrorHandler byteAwareDltErrorHandler(
|
||||
@Qualifier("jsonTemplate") KafkaOperations<?, ?> jsonTemplate,
|
||||
@Qualifier("byteTemplate") KafkaOperations<?, ?> byteTemplate) {
|
||||
Map<Class<?>, KafkaOperations<?, ?>> templates = new java.util.LinkedHashMap<>();
|
||||
templates.put(byte[].class, byteTemplate);
|
||||
templates.put(Object.class, jsonTemplate);
|
||||
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);
|
||||
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 0));
|
||||
handler.addNotRetryableExceptions(Failures.PermanentFailure.class);
|
||||
return handler;
|
||||
}
|
||||
|
||||
/**
|
||||
* Replaces the auto-configured template, which backed off the moment {@code byteTemplate}
|
||||
* was declared.
|
||||
*/
|
||||
@Bean
|
||||
@Profile("dltbytes")
|
||||
KafkaTemplate<String, Object> jsonTemplate(KafkaProperties properties) {
|
||||
return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(properties.buildProducerProperties()));
|
||||
}
|
||||
|
||||
@Bean
|
||||
@Profile("dltbytes")
|
||||
KafkaTemplate<String, byte[]> byteTemplate(KafkaProperties properties) {
|
||||
Map<String, Object> config = new HashMap<>(properties.buildProducerProperties());
|
||||
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
|
||||
return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(config));
|
||||
}
|
||||
|
||||
/** The stock handler, declared explicitly so a test can read its behaviour. */
|
||||
@Bean
|
||||
@Profile("defaults")
|
||||
DefaultErrorHandler defaultErrorHandler() {
|
||||
return new DefaultErrorHandler();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
/**
|
||||
* What Spring Kafka does when your listener throws, and what it does when the record cannot even
|
||||
* be turned into an object.
|
||||
*
|
||||
* <p>Those are two different failures with two different cures, and conflating them is why a
|
||||
* poison pill takes a partition down for hours.
|
||||
*/
|
||||
@SpringBootApplication
|
||||
public class ErrorHandlingApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(ErrorHandlingApplication.class, args);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
/**
|
||||
* The two failure classes that Spring Kafka treats completely differently, made explicit so the
|
||||
* tests can be about the distinction rather than about a generic RuntimeException.
|
||||
*/
|
||||
public final class Failures {
|
||||
|
||||
/** Worth retrying: a timeout, a 503, a deadlock. The same input may succeed later. */
|
||||
public static class TransientFailure extends RuntimeException {
|
||||
|
||||
public TransientFailure(String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* Not worth retrying: a validation failure, a missing referenced entity, a malformed field.
|
||||
* The same input will fail identically ten times, and retrying it is pure latency.
|
||||
*/
|
||||
public static class PermanentFailure extends RuntimeException {
|
||||
|
||||
public PermanentFailure(String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private Failures() {
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
/** The payload. {@code paymentId} doubles as the record key. */
|
||||
public record Payment(String paymentId, String status) {
|
||||
|
||||
public static Payment of(String paymentId) {
|
||||
return new Payment(paymentId, "PENDING");
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.kafka.annotation.KafkaListener;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
|
||||
/**
|
||||
* A listener that fails on demand, so the retry machinery can be observed rather than described.
|
||||
*
|
||||
* <p>Payment ids beginning {@code transient-} throw a retryable exception; ids beginning
|
||||
* {@code permanent-} throw a non-retryable one; everything else succeeds.
|
||||
*
|
||||
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
|
||||
*/
|
||||
@Component
|
||||
public class PaymentListener {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(PaymentListener.class);
|
||||
|
||||
public static final String TOPIC = "payments";
|
||||
|
||||
/** Every delivery attempt, with the wall-clock time it happened. */
|
||||
public record Attempt(String paymentId, long atMillis, int partition, long offset) {
|
||||
}
|
||||
|
||||
private final List<Attempt> attempts = new CopyOnWriteArrayList<>();
|
||||
|
||||
@KafkaListener(topics = TOPIC, groupId = "payments")
|
||||
public void onPayment(ConsumerRecord<String, Payment> record) {
|
||||
Payment payment = record.value();
|
||||
this.attempts.add(new Attempt(payment.paymentId(), System.currentTimeMillis(),
|
||||
record.partition(), record.offset()));
|
||||
log.info("attempt {} for {}", this.attempts.size(), payment.paymentId());
|
||||
if (payment.paymentId().startsWith("transient-")) {
|
||||
throw new Failures.TransientFailure("downstream unavailable for " + payment.paymentId());
|
||||
}
|
||||
if (payment.paymentId().startsWith("permanent-")) {
|
||||
throw new Failures.PermanentFailure("invalid payment " + payment.paymentId());
|
||||
}
|
||||
}
|
||||
|
||||
public List<Attempt> attempts() {
|
||||
return this.attempts;
|
||||
}
|
||||
|
||||
public List<Attempt> attemptsFor(String paymentId) {
|
||||
return this.attempts.stream().filter((a) -> a.paymentId().equals(paymentId)).toList();
|
||||
}
|
||||
|
||||
public void clear() {
|
||||
this.attempts.clear();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.context.annotation.Profile;
|
||||
import org.springframework.kafka.annotation.BackOff;
|
||||
import org.springframework.kafka.annotation.DltHandler;
|
||||
import org.springframework.kafka.annotation.KafkaListener;
|
||||
import org.springframework.kafka.annotation.RetryableTopic;
|
||||
import org.springframework.kafka.retrytopic.SameIntervalTopicReuseStrategy;
|
||||
import org.springframework.kafka.support.KafkaHeaders;
|
||||
import org.springframework.messaging.handler.annotation.Header;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
|
||||
/**
|
||||
* Non-blocking retries. Instead of holding the consumer thread while it re-delivers,
|
||||
* {@code @RetryableTopic} republishes the failed record to a separate retry topic with a
|
||||
* timestamp header, and a container for that topic waits before processing it.
|
||||
*
|
||||
* <p>The consequence people miss: <b>the main partition advances immediately</b>. That is the
|
||||
* entire point — and it is also why per-key ordering is gone for any record that fails.
|
||||
* Record 2 for the same key is processed while record 1 is sitting in a retry topic.
|
||||
*
|
||||
* @see <a href="../../../../../docs/05-retryable-topic.md">docs/05-retryable-topic.md</a>
|
||||
*/
|
||||
@Component
|
||||
@Profile("retrytopic")
|
||||
public class RetryableTopicListener {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(RetryableTopicListener.class);
|
||||
|
||||
public static final String TOPIC = "invoices";
|
||||
|
||||
public record Delivery(String topic, String paymentId, long atMillis) {
|
||||
}
|
||||
|
||||
private final List<Delivery> deliveries = new CopyOnWriteArrayList<>();
|
||||
|
||||
private final List<Delivery> dead = new CopyOnWriteArrayList<>();
|
||||
|
||||
// backOff, not backoff - and org.springframework.kafka.annotation.BackOff, not
|
||||
// org.springframework.retry.annotation.Backoff. Spring Kafka 4.x dropped the spring-retry
|
||||
// dependency and brought its own annotation, so every pre-4.x @RetryableTopic example is a
|
||||
// compile error: "package org.springframework.retry.annotation does not exist".
|
||||
@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
|
||||
sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
|
||||
exclude = Failures.PermanentFailure.class)
|
||||
@KafkaListener(topics = TOPIC, groupId = "invoices")
|
||||
public void onInvoice(ConsumerRecord<String, Payment> record,
|
||||
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
|
||||
this.deliveries.add(new Delivery(topic, record.value().paymentId(), System.currentTimeMillis()));
|
||||
log.info("delivery {} on topic {}", this.deliveries.size(), topic);
|
||||
if (record.value().paymentId().startsWith("transient-")) {
|
||||
throw new Failures.TransientFailure("still failing");
|
||||
}
|
||||
if (record.value().paymentId().startsWith("permanent-")) {
|
||||
throw new Failures.PermanentFailure("never going to work");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a record lands once the attempts are exhausted. Without a {@code @DltHandler} the
|
||||
* framework logs it and moves on — the topic still exists and still has the record, but
|
||||
* nothing in your application has looked at it.
|
||||
*/
|
||||
@DltHandler
|
||||
public void onDlt(ConsumerRecord<String, Payment> record,
|
||||
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
|
||||
this.dead.add(new Delivery(topic, record.value().paymentId(), System.currentTimeMillis()));
|
||||
log.info("DLT handler: {} on {}", record.value().paymentId(), topic);
|
||||
}
|
||||
|
||||
public List<Delivery> deliveries() {
|
||||
return this.deliveries;
|
||||
}
|
||||
|
||||
public List<Delivery> dead() {
|
||||
return this.dead;
|
||||
}
|
||||
|
||||
}
|
||||
27
kafka-error-handling/src/main/resources/application.yaml
Normal file
27
kafka-error-handling/src/main/resources/application.yaml
Normal file
@@ -0,0 +1,27 @@
|
||||
spring:
|
||||
application:
|
||||
name: kafka-error-handling
|
||||
main:
|
||||
banner-mode: off
|
||||
kafka:
|
||||
bootstrap-servers: localhost:9092
|
||||
producer:
|
||||
key-serializer: org.apache.kafka.common.serialization.StringSerializer
|
||||
value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
|
||||
consumer:
|
||||
group-id: payments
|
||||
auto-offset-reset: earliest
|
||||
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
|
||||
# ErrorHandlingDeserializer wraps the real one. Without it a record that cannot be
|
||||
# deserialized is thrown BEFORE the listener exists, the container cannot advance past it,
|
||||
# and the same offset is retried forever. See docs/03-poison-pills.md.
|
||||
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
|
||||
properties:
|
||||
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
|
||||
spring.json.trusted.packages: com.ankurm.kafkaerrors
|
||||
spring.json.value.default.type: com.ankurm.kafkaerrors.Payment
|
||||
logging:
|
||||
level:
|
||||
root: WARN
|
||||
com.ankurm: INFO
|
||||
org.springframework.kafka.listener: WARN
|
||||
@@ -0,0 +1,83 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.apache.kafka.clients.consumer.Consumer;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecords;
|
||||
import org.apache.kafka.clients.consumer.KafkaConsumer;
|
||||
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
|
||||
import org.apache.kafka.common.serialization.ByteArraySerializer;
|
||||
import org.apache.kafka.common.serialization.StringDeserializer;
|
||||
import org.apache.kafka.common.serialization.StringSerializer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
|
||||
import org.springframework.kafka.core.KafkaTemplate;
|
||||
import org.springframework.kafka.test.EmbeddedKafkaBroker;
|
||||
import org.springframework.kafka.test.context.EmbeddedKafka;
|
||||
import org.springframework.kafka.test.utils.KafkaTestUtils;
|
||||
import org.springframework.test.context.ActiveProfiles;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* The fix for the base64 problem in {@link DeadLetterTopicTest}: a per-value-type template map
|
||||
* so that a {@code byte[]} is published with a {@code ByteArraySerializer}.
|
||||
*
|
||||
* @see <a href="../../../../../docs/03-poison-pills.md">docs/03-poison-pills.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
@ActiveProfiles({ "test", "dltbytes" })
|
||||
@EmbeddedKafka(topics = { "payments", "payments-dlt" }, partitions = 1)
|
||||
class ByteAwareDltTest {
|
||||
|
||||
@Autowired
|
||||
EmbeddedKafkaBroker broker;
|
||||
|
||||
@Test
|
||||
void aByteArrayTemplateKeepsTheOriginalBytesIntact() {
|
||||
Map<String, Object> producerProps = new HashMap<>(KafkaTestUtils.producerProps(this.broker.getBrokersAsString()));
|
||||
producerProps.put("key.serializer", StringSerializer.class);
|
||||
producerProps.put("value.serializer", ByteArraySerializer.class);
|
||||
DefaultKafkaProducerFactory<String, byte[]> factory = new DefaultKafkaProducerFactory<>(producerProps);
|
||||
try {
|
||||
new KafkaTemplate<>(factory).send(PaymentListener.TOPIC, "poison-2",
|
||||
"{ this is not json".getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
finally {
|
||||
factory.destroy();
|
||||
}
|
||||
|
||||
Map<String, Object> consumerProps = new HashMap<>(
|
||||
KafkaTestUtils.consumerProps(this.broker.getBrokersAsString(), "byte-dlt-reader", true));
|
||||
consumerProps.put("key.deserializer", StringDeserializer.class);
|
||||
consumerProps.put("value.deserializer", ByteArrayDeserializer.class);
|
||||
consumerProps.put("auto.offset.reset", "earliest");
|
||||
try (Consumer<String, byte[]> consumer = new KafkaConsumer<>(consumerProps)) {
|
||||
consumer.subscribe(List.of("payments-dlt"));
|
||||
long deadline = System.currentTimeMillis() + 30_000;
|
||||
while (System.currentTimeMillis() < deadline) {
|
||||
ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(500));
|
||||
for (ConsumerRecord<String, byte[]> record : records) {
|
||||
if ("poison-2".equals(record.key())) {
|
||||
String payload = new String(record.value(), StandardCharsets.UTF_8);
|
||||
System.out.println("=== byte-aware DLT ===");
|
||||
System.out.println(" DLT payload -> " + payload);
|
||||
// Byte for byte what was published. No base64, no quotes. Replay is now
|
||||
// a copy from one topic to another.
|
||||
assertThat(payload).isEqualTo("{ this is not json");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new AssertionError("nothing arrived on payments-dlt");
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.apache.kafka.clients.consumer.Consumer;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecords;
|
||||
import org.apache.kafka.common.serialization.ByteArraySerializer;
|
||||
import org.apache.kafka.common.serialization.StringSerializer;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
|
||||
import org.springframework.kafka.core.KafkaTemplate;
|
||||
import org.springframework.kafka.support.serializer.ErrorHandlingDeserializer;
|
||||
import org.springframework.kafka.test.EmbeddedKafkaBroker;
|
||||
import org.springframework.kafka.test.context.EmbeddedKafka;
|
||||
import org.springframework.kafka.test.utils.KafkaTestUtils;
|
||||
import org.springframework.test.context.ActiveProfiles;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
|
||||
/**
|
||||
* Retries, classification, and the two very different reasons a record ends up on the DLT.
|
||||
*
|
||||
* @see <a href="../../../../../docs/04-the-dlt.md">docs/04-the-dlt.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
@ActiveProfiles({ "test", "dlt" })
|
||||
@EmbeddedKafka(topics = { "payments", "payments-dlt" }, partitions = 1)
|
||||
class DeadLetterTopicTest {
|
||||
|
||||
@Autowired
|
||||
KafkaTemplate<String, Payment> template;
|
||||
|
||||
@Autowired
|
||||
PaymentListener listener;
|
||||
|
||||
@Autowired
|
||||
EmbeddedKafkaBroker broker;
|
||||
|
||||
@BeforeEach
|
||||
void clear() {
|
||||
this.listener.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the DLT as raw bytes, because a dead-lettered record may be exactly the thing that
|
||||
* could not be deserialized.
|
||||
*
|
||||
* <p>Note the topic name: {@code payments-dlt}. {@code RetryTopicConstants.DEFAULT_DLT_SUFFIX}
|
||||
* is {@code "-dlt"}, not {@code ".DLT"} — and if the topic does not exist, the
|
||||
* recoverer logs a WARN and the record is gone.
|
||||
*/
|
||||
private Consumer<String, byte[]> dltConsumer() {
|
||||
Map<String, Object> props = new HashMap<>(
|
||||
KafkaTestUtils.consumerProps(this.broker.getBrokersAsString(), "dlt-reader-" + System.nanoTime(), true));
|
||||
props.put("key.deserializer", org.apache.kafka.common.serialization.StringDeserializer.class);
|
||||
props.put("value.deserializer", org.apache.kafka.common.serialization.ByteArrayDeserializer.class);
|
||||
props.put("auto.offset.reset", "earliest");
|
||||
Consumer<String, byte[]> consumer = new org.apache.kafka.clients.consumer.KafkaConsumer<>(props);
|
||||
consumer.subscribe(java.util.List.of("payments-dlt"));
|
||||
return consumer;
|
||||
}
|
||||
|
||||
@Test
|
||||
void aRetryableFailureIsRetriedAndThenPublishedToTheDlt() {
|
||||
this.template.send(PaymentListener.TOPIC, "transient-1", Payment.of("transient-1"));
|
||||
|
||||
// FixedBackOff(1000, 2) = one delivery plus two retries.
|
||||
await().atMost(Duration.ofSeconds(30))
|
||||
.until(() -> this.listener.attemptsFor("transient-1").size() >= 3);
|
||||
assertThat(this.listener.attemptsFor("transient-1")).hasSize(3);
|
||||
|
||||
var attempts = this.listener.attemptsFor("transient-1");
|
||||
long gap = attempts.get(1).atMillis() - attempts.get(0).atMillis();
|
||||
System.out.println("=== transient failure ===");
|
||||
System.out.println(" deliveries " + attempts.size());
|
||||
System.out.println(" gap between 1&2 " + gap + " ms (FixedBackOff interval 1000)");
|
||||
assertThat(gap).isGreaterThanOrEqualTo(900);
|
||||
|
||||
try (Consumer<String, byte[]> consumer = dltConsumer()) {
|
||||
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "transient-1");
|
||||
assertThat(dead).isNotNull();
|
||||
printHeaders("transient failure on the DLT", dead);
|
||||
// The TOP-LEVEL exception header is always the wrapper. Group a DLT triage dashboard
|
||||
// by kafka_dlt-exception-fqcn and every listener failure in the estate lands in one
|
||||
// bucket called ListenerExecutionFailedException. The useful field is the cause.
|
||||
assertThat(header(dead, "kafka_dlt-exception-fqcn"))
|
||||
.isEqualTo("org.springframework.kafka.listener.ListenerExecutionFailedException");
|
||||
assertThat(header(dead, "kafka_dlt-exception-cause-fqcn"))
|
||||
.isEqualTo(Failures.TransientFailure.class.getName());
|
||||
assertThat(header(dead, "kafka_dlt-original-topic")).isEqualTo("payments");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aNonRetryableFailureGoesStraightToTheDltWithNoRetries() {
|
||||
this.template.send(PaymentListener.TOPIC, "permanent-1", Payment.of("permanent-1"));
|
||||
|
||||
try (Consumer<String, byte[]> consumer = dltConsumer()) {
|
||||
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "permanent-1");
|
||||
assertThat(dead).isNotNull();
|
||||
// addNotRetryableExceptions(PermanentFailure.class) means exactly one delivery.
|
||||
System.out.println("=== permanent failure ===");
|
||||
System.out.println(" deliveries " + this.listener.attemptsFor("permanent-1").size());
|
||||
assertThat(this.listener.attemptsFor("permanent-1")).hasSize(1);
|
||||
assertThat(header(dead, "kafka_dlt-exception-cause-fqcn"))
|
||||
.isEqualTo(Failures.PermanentFailure.class.getName());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPoisonPillNeverReachesTheListenerAndIsStillRecovered() {
|
||||
// Raw bytes that are not valid JSON, published with a byte[] serializer so nothing on
|
||||
// the producing side objects. This is what a schema change from another team looks like.
|
||||
Map<String, Object> props = new HashMap<>(KafkaTestUtils.producerProps(this.broker.getBrokersAsString()));
|
||||
props.put("key.serializer", StringSerializer.class);
|
||||
props.put("value.serializer", ByteArraySerializer.class);
|
||||
DefaultKafkaProducerFactory<String, byte[]> factory = new DefaultKafkaProducerFactory<>(props);
|
||||
try {
|
||||
new KafkaTemplate<>(factory).send(PaymentListener.TOPIC, "poison-1",
|
||||
"{ this is not json".getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
finally {
|
||||
factory.destroy();
|
||||
}
|
||||
|
||||
try (Consumer<String, byte[]> consumer = dltConsumer()) {
|
||||
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "poison-1");
|
||||
assertThat(dead).isNotNull();
|
||||
printHeaders("poison pill on the DLT", dead);
|
||||
|
||||
// The listener was never invoked - the failure happened in the deserializer, before
|
||||
// the listener existed. ErrorHandlingDeserializer is what turns that into a record
|
||||
// the error handler can recover instead of a poll that fails forever.
|
||||
assertThat(this.listener.attemptsFor("poison-1")).isEmpty();
|
||||
assertThat(header(dead, "kafka_dlt-exception-fqcn"))
|
||||
.contains("DeserializationException");
|
||||
// And here is the sting. The recoverer publishes with the APPLICATION's producer,
|
||||
// whose value serializer is JacksonJsonSerializer. The failed value is a byte[], and
|
||||
// Jackson writes a byte[] as a base64 JSON string. So the DLT does not hold the
|
||||
// original bytes - it holds base64 of them, wrapped in quotes.
|
||||
String payload = new String(dead.value(), StandardCharsets.UTF_8);
|
||||
System.out.println(" DLT payload -> " + payload);
|
||||
assertThat(payload).isEqualTo("\"eyB0aGlzIGlzIG5vdCBqc29u\"");
|
||||
assertThat(new String(java.util.Base64.getDecoder().decode(
|
||||
payload.replace("\"", "")), StandardCharsets.UTF_8)).isEqualTo("{ this is not json");
|
||||
}
|
||||
}
|
||||
|
||||
private ConsumerRecord<String, byte[]> pollFor(Consumer<String, byte[]> consumer, String key) {
|
||||
long deadline = System.currentTimeMillis() + 30_000;
|
||||
while (System.currentTimeMillis() < deadline) {
|
||||
ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(500));
|
||||
for (ConsumerRecord<String, byte[]> record : records) {
|
||||
if (key.equals(record.key())) {
|
||||
return record;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private static String header(ConsumerRecord<String, byte[]> record, String name) {
|
||||
var header = record.headers().lastHeader(name);
|
||||
return (header == null) ? null : new String(header.value(), StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private static void printHeaders(String title, ConsumerRecord<String, byte[]> record) {
|
||||
System.out.println("=== " + title + " ===");
|
||||
record.headers().forEach((h) -> {
|
||||
// Some of these headers are binary (the original partition, offset and timestamp are
|
||||
// big-endian ints and longs, not text), so render anything non-printable as hex.
|
||||
String value = new String(h.value(), StandardCharsets.UTF_8).replaceAll("\\s+", " ");
|
||||
if (!value.chars().allMatch((c) -> c >= 0x20 && c < 0x7f)) {
|
||||
StringBuilder hex = new StringBuilder("0x");
|
||||
for (byte b : h.value()) {
|
||||
hex.append("%02x".formatted(b));
|
||||
}
|
||||
value = hex.toString();
|
||||
}
|
||||
System.out.printf(" %-34s %s%n", h.key(), value.length() > 90 ? value.substring(0, 90) + "..." : value);
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.kafka.listener.SeekUtils;
|
||||
import org.springframework.util.backoff.BackOffExecution;
|
||||
import org.springframework.util.backoff.FixedBackOff;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* What the stock {@code DefaultErrorHandler} actually does, read out of its own back-off rather
|
||||
* than out of the documentation. No broker and no context: a {@code BackOff} is a state machine
|
||||
* you can just run.
|
||||
*
|
||||
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
|
||||
*/
|
||||
class DefaultBackOffTest {
|
||||
|
||||
@Test
|
||||
void theDefaultIsTenAttemptsWithNoDelayAtAll() {
|
||||
FixedBackOff backOff = SeekUtils.DEFAULT_BACK_OFF;
|
||||
BackOffExecution execution = backOff.start();
|
||||
|
||||
List<Long> intervals = new ArrayList<>();
|
||||
long next;
|
||||
while ((next = execution.nextBackOff()) != BackOffExecution.STOP) {
|
||||
intervals.add(next);
|
||||
}
|
||||
|
||||
System.out.println("=== DefaultErrorHandler default back-off ===");
|
||||
System.out.println(" interval " + backOff.getInterval() + " ms");
|
||||
System.out.println(" max attempts " + backOff.getMaxAttempts() + " retries");
|
||||
System.out.println(" SeekUtils.DEFAULT_MAX_FAILURES = " + SeekUtils.DEFAULT_MAX_FAILURES);
|
||||
System.out.println(" retry intervals " + intervals);
|
||||
System.out.println(" total deliveries " + (intervals.size() + 1));
|
||||
|
||||
// Nine retries after the first delivery = ten deliveries, matching DEFAULT_MAX_FAILURES.
|
||||
assertThat(intervals).hasSize(SeekUtils.DEFAULT_MAX_FAILURES - 1);
|
||||
// And every one of them is zero. The default is not "retry with backoff"; it is
|
||||
// "hammer the same record ten times as fast as the consumer thread can go, then give up".
|
||||
assertThat(intervals).containsOnly(0L);
|
||||
assertThat(backOff.getInterval()).isZero();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package com.ankurm.kafkaerrors;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.kafka.core.KafkaTemplate;
|
||||
import org.springframework.kafka.test.EmbeddedKafkaBroker;
|
||||
import org.springframework.kafka.test.context.EmbeddedKafka;
|
||||
import org.springframework.test.context.ActiveProfiles;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
|
||||
/**
|
||||
* Non-blocking retries, and the topics they create.
|
||||
*
|
||||
* @see <a href="../../../../../docs/05-retryable-topic.md">docs/05-retryable-topic.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
@ActiveProfiles({ "test", "retrytopic" })
|
||||
@EmbeddedKafka(topics = { "invoices" }, partitions = 1)
|
||||
class RetryableTopicTest {
|
||||
|
||||
@Autowired
|
||||
KafkaTemplate<String, Payment> template;
|
||||
|
||||
@Autowired
|
||||
RetryableTopicListener listener;
|
||||
|
||||
@Autowired
|
||||
EmbeddedKafkaBroker broker;
|
||||
|
||||
@Test
|
||||
void retriesHappenOnSeparateTopicsAndTheMainPartitionKeepsMoving() {
|
||||
this.template.send(RetryableTopicListener.TOPIC, "transient-1", Payment.of("transient-1"));
|
||||
// Sent immediately after the failing one, on the same partition. If retries were
|
||||
// blocking, this could not be processed until the first record gave up.
|
||||
this.template.send(RetryableTopicListener.TOPIC, "ok-1", Payment.of("ok-1"));
|
||||
|
||||
await().atMost(Duration.ofSeconds(60)).until(() -> !this.listener.dead().isEmpty());
|
||||
|
||||
List<RetryableTopicListener.Delivery> deliveries = this.listener.deliveries();
|
||||
System.out.println("=== @RetryableTopic delivery trace ===");
|
||||
long start = deliveries.get(0).atMillis();
|
||||
deliveries.forEach((d) -> System.out.printf(" +%-6d ms %-28s %s%n",
|
||||
d.atMillis() - start, d.topic(), d.paymentId()));
|
||||
System.out.println(" DLT: " + this.listener.dead().stream()
|
||||
.map((d) -> d.paymentId() + " on " + d.topic()).toList());
|
||||
System.out.println(" topics touched: " + deliveries.stream()
|
||||
.map(RetryableTopicListener.Delivery::topic).distinct().sorted().toList());
|
||||
|
||||
// attempts = "4" means the first delivery plus three retries.
|
||||
assertThat(deliveries).filteredOn((d) -> d.paymentId().equals("transient-1")).hasSize(4);
|
||||
|
||||
// Retries land on generated topics, not on the original one.
|
||||
// Retry topics are named by the DELAY, not by the attempt number. With a multiplier,
|
||||
// each distinct interval gets its own topic: invoices-retry-500, -1000, -2000. That is
|
||||
// TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE, the default. Provisioning a cluster
|
||||
// for this means knowing the whole back-off schedule in advance.
|
||||
assertThat(deliveries).filteredOn((d) -> d.paymentId().equals("transient-1"))
|
||||
.extracting(RetryableTopicListener.Delivery::topic)
|
||||
.containsExactly("invoices", "invoices-retry-500", "invoices-retry-1000",
|
||||
"invoices-retry-2000");
|
||||
assertThat(this.listener.dead()).extracting(RetryableTopicListener.Delivery::topic)
|
||||
.containsExactly("invoices-dlt");
|
||||
|
||||
// The good record was processed while the bad one was still being retried. That is the
|
||||
// whole benefit, and the whole cost: ordering on this partition is gone.
|
||||
long okAt = deliveries.stream().filter((d) -> d.paymentId().equals("ok-1"))
|
||||
.findFirst().orElseThrow().atMillis();
|
||||
long lastRetryAt = deliveries.stream().filter((d) -> d.paymentId().equals("transient-1"))
|
||||
.mapToLong(RetryableTopicListener.Delivery::atMillis).max().orElseThrow();
|
||||
assertThat(okAt).isLessThan(lastRetryAt);
|
||||
|
||||
assertThat(this.listener.dead()).extracting(RetryableTopicListener.Delivery::paymentId)
|
||||
.containsExactly("transient-1");
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
spring:
|
||||
kafka:
|
||||
bootstrap-servers: ${spring.embedded.kafka.brokers}
|
||||
80
rabbitmq/README.md
Normal file
80
rabbitmq/README.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# `rabbitmq` — exchanges, bindings and a dead-letter queue that works
|
||||
|
||||
Companion project for
|
||||
[**Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue**](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/)
|
||||
on ankurm.com.
|
||||
|
||||
Ten tests against a **real RabbitMQ broker**. All four exchange types, manual acknowledgement,
|
||||
a dead-letter path exercised through `basicNack` and through TTL expiry, and the three ways a
|
||||
topology loses work quietly.
|
||||
|
||||
## Versions
|
||||
|
||||
| | Version | Notes |
|
||||
|---|---|---|
|
||||
| JDK | 25 (Temurin 25.0.4.1+1) | current LTS |
|
||||
| Spring Boot | 4.1.1 | |
|
||||
| Spring AMQP / Spring Rabbit | 4.1.1 | Boot-managed |
|
||||
| `com.rabbitmq:amqp-client` | **5.30.0** | Boot-managed. Central has 5.35.0 |
|
||||
| Testcontainers | 2.0.5 | artifact is `testcontainers-rabbitmq`, not `rabbitmq` |
|
||||
| **Broker for the committed transcripts** | **RabbitMQ 3.10.25** | see the note below |
|
||||
|
||||
Client-side versions were read from `repo1.maven.org/.../maven-metadata.xml` and Boot's
|
||||
`spring-boot-dependencies` POM.
|
||||
|
||||
> **About the broker version.** The machine that regenerates `docs/output/` has no Docker daemon
|
||||
> and no root, so `scripts/broker.sh` starts a generic-unix RabbitMQ against an extracted Erlang
|
||||
> 24 runtime, and the newest release that pairs with Erlang 24 is 3.10.25. Everything exercised
|
||||
> here — the four exchange types, `x-dead-letter-exchange`, `x-dead-letter-routing-key`,
|
||||
> `x-message-ttl`, `x-death`, manual ack, `mandatory` returns, `PRECONDITION_FAILED` on
|
||||
> inequivalent arguments — is AMQP 0-9-1 behaviour that is unchanged in RabbitMQ 4.x. The one
|
||||
> broker-version difference worth knowing for 4.x is that **quorum queues are the default queue
|
||||
> type** and classic mirrored queues are gone; nothing in this module declares a queue type, so
|
||||
> the topology is valid on both. If you have Docker, run the same tests against
|
||||
> `rabbitmq:4.1-management` with
|
||||
> [`TestcontainersConfiguration`](src/test/java/com/ankurm/rabbit/TestcontainersConfiguration.java).
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# with Docker: point the tests at Testcontainers, or run your own broker on 5672
|
||||
mvn test
|
||||
|
||||
# without Docker:
|
||||
export ERL_ROOT=/path/to/erlang RABBITMQ_HOME=/path/to/rabbitmq_server-3.10.25
|
||||
./scripts/run-all.sh
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
1. [The on-ramp](docs/01-the-on-ramp.md)
|
||||
2. [Four exchange types](docs/02-exchanges.md)
|
||||
3. [The message that goes nowhere and says nothing](docs/03-the-silent-drop.md)
|
||||
4. [A dead-letter queue that actually works](docs/04-dead-lettering.md)
|
||||
5. [Acknowledgement](docs/05-acknowledgement.md)
|
||||
6. [Changing your mind about a queue](docs/06-changing-your-mind.md)
|
||||
|
||||
## Captured output
|
||||
|
||||
| File | Shows |
|
||||
|---|---|
|
||||
| [`topic-wildcards.txt`](docs/output/topic-wildcards.txt) | `*` vs `#` over five routing keys |
|
||||
| [`dead-letter.txt`](docs/output/dead-letter.txt) | `x-death` for `rejected` and for `expired` |
|
||||
| [`requeue-loop.txt`](docs/output/requeue-loop.txt) | 199 redeliveries, 0 dead-lettered |
|
||||
| [`unroutable.txt`](docs/output/unroutable.txt) | `312 NO_ROUTE` |
|
||||
| [`precondition-failed.txt`](docs/output/precondition-failed.txt) | `406` on an inequivalent argument |
|
||||
| [`tests.txt`](docs/output/tests.txt) | 10 tests |
|
||||
|
||||
## Five things this module exists to prove
|
||||
|
||||
1. **An unroutable message is discarded silently**, and finding out needs `publisher-returns`
|
||||
*and* `mandatory` — two settings in two different places. Setting only `mandatory` does
|
||||
nothing.
|
||||
2. **`requeue=true` never dead-letters.** 200 attempts, 199 redeliveries, an empty DLQ and a
|
||||
queue depth that stays at 1 the whole time.
|
||||
3. **`x-death.reason` distinguishes `rejected` from `expired`**, which is the difference between
|
||||
a consumer that refused the work and a consumer that never got to it.
|
||||
4. **Queue arguments are immutable** — `406 PRECONDITION_FAILED`, wrapped in an exception whose
|
||||
own message is the string `java.io.IOException`.
|
||||
5. **Boot auto-configures no JSON converter for RabbitMQ**, and the converter you want is
|
||||
`JacksonJsonMessageConverter`, not `Jackson2JsonMessageConverter`.
|
||||
59
rabbitmq/docs/01-the-on-ramp.md
Normal file
59
rabbitmq/docs/01-the-on-ramp.md
Normal file
@@ -0,0 +1,59 @@
|
||||
[Module README](../README.md) · [Exchanges →](02-exchanges.md)
|
||||
|
||||
# 1. The on-ramp
|
||||
|
||||
Two dependency decisions decide whether anything works, and neither produces a helpful error.
|
||||
|
||||
## Use the starter, not `spring-rabbit`
|
||||
|
||||
Boot 4 moved every auto-configuration out of `spring-boot-autoconfigure` into a per-technology
|
||||
module. RabbitMQ's lives in `spring-boot-amqp`, package
|
||||
`org.springframework.boot.amqp.autoconfigure`, and a bare `org.springframework.amqp:spring-rabbit`
|
||||
dependency does not bring it. You get no `RabbitTemplate`, no `RabbitAdmin`, no listener
|
||||
container factory — and a context that starts cleanly.
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-amqp</artifactId>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Boot does not give you a JSON converter
|
||||
|
||||
Spring Kafka defaults to serializers you configure. Spring AMQP defaults to
|
||||
`SimpleMessageConverter`, which handles `String`, `byte[]` and `Serializable` and refuses
|
||||
everything else:
|
||||
|
||||
```
|
||||
IllegalArgumentException: SimpleMessageConverter only supports String, byte[] and
|
||||
Serializable payloads, received: com.ankurm.rabbit.OrderMessage
|
||||
```
|
||||
|
||||
One bean fixes both directions, because the auto-configured `RabbitTemplate` and the listener
|
||||
container factory both look for a `MessageConverter`:
|
||||
|
||||
```java
|
||||
@Bean
|
||||
MessageConverter messageConverter() {
|
||||
return new JacksonJsonMessageConverter();
|
||||
}
|
||||
```
|
||||
|
||||
**Note the class name.** Spring AMQP 4.1 ships `Jackson2JsonMessageConverter` (Jackson 2) and
|
||||
`JacksonJsonMessageConverter` (Jackson 3) side by side — exactly as Spring Kafka ships
|
||||
`JsonSerializer` and `JacksonJsonSerializer`. Boot 4 is a Jackson 3 application. The rule across
|
||||
both stacks is the same: **if the class name contains a `2`, it belongs to the previous major
|
||||
version of Jackson.**
|
||||
|
||||
## Versions
|
||||
|
||||
Boot 4.1.1 manages Spring AMQP **4.1.1** and `com.rabbitmq:amqp-client` **5.30.0**. Maven Central
|
||||
has amqp-client 5.35.0; overriding the managed version to reach it is a change you should have a
|
||||
reason for.
|
||||
|
||||
One API change to know about: `RabbitAdmin.QUEUE_MESSAGE_COUNT` now holds a `Long`. Every older
|
||||
example casts it to `Integer`, which is a `ClassCastException` at runtime and not at compile
|
||||
time.
|
||||
|
||||
[Exchanges →](02-exchanges.md)
|
||||
71
rabbitmq/docs/02-exchanges.md
Normal file
71
rabbitmq/docs/02-exchanges.md
Normal file
@@ -0,0 +1,71 @@
|
||||
[← The on-ramp](01-the-on-ramp.md) · [Module README](../README.md) · [The silent drop →](03-the-silent-drop.md)
|
||||
|
||||
# 2. Four exchange types
|
||||
|
||||
A producer never publishes to a queue. It publishes to an **exchange** with a **routing key**,
|
||||
and bindings decide where that lands. The whole topology is in
|
||||
[`Topology.java`](../src/main/java/com/ankurm/rabbit/Topology.java) as beans; `RabbitAdmin`
|
||||
declares them when the connection opens.
|
||||
|
||||
## Direct — exact match
|
||||
|
||||
Binding key `new` receives routing key `new`. Nothing else. This is the workhorse: one queue per
|
||||
command type.
|
||||
|
||||
## Fanout — routing key ignored entirely
|
||||
|
||||
Every bound queue gets a copy. `audit.all` and `analytics.all` both receive it, and the routing
|
||||
key you passed is not consulted at all. Use it for broadcast; use it knowing that adding a queue
|
||||
adds a full copy of the traffic.
|
||||
|
||||
## Topic — wildcards over dot-separated words
|
||||
|
||||
`*` is **exactly one word**. `#` is **zero or more**. The distinction is the one people get
|
||||
wrong, so here it is against a real broker
|
||||
([`docs/output/topic-wildcards.txt`](output/topic-wildcards.txt)):
|
||||
|
||||
```
|
||||
routing key orders.eu orders.high note
|
||||
order.eu.high true true matches both
|
||||
order.eu.low true false matches order.eu.* only
|
||||
order.us.high false true matches order.#.high only
|
||||
order.eu.west.high false true matches order.#.high only - * is one word
|
||||
order.high false true matches order.#.high - # can be zero words
|
||||
```
|
||||
|
||||
Bindings are `order.eu.*` and `order.#.high`. Two rows are worth pausing on:
|
||||
|
||||
- `order.eu.west.high` does **not** match `order.eu.*`, because `*` matches one word and `west.high`
|
||||
is two. Regex intuition says otherwise.
|
||||
- `order.high` **does** match `order.#.high`, because `#` matches zero words. So a binding you
|
||||
wrote to mean "something in the middle" also matches "nothing in the middle".
|
||||
|
||||
Design routing keys most-general-to-most-specific (`order.eu.west.high`, not
|
||||
`high.order.eu.west`), because `#` and `*` work left to right and a hierarchy you can bind
|
||||
usefully is one that starts broad.
|
||||
|
||||
## Headers — match a map, ignore the routing key
|
||||
|
||||
`x-match=all` requires every named header to be present **and equal**. `x-match=any` requires
|
||||
one. From `RoutingTest`:
|
||||
|
||||
| Message headers | `x-match=all` queue | `x-match=any` queue |
|
||||
|---|---|---|
|
||||
| `priority=high, region=eu` | yes | yes |
|
||||
| `priority=high` | no | yes |
|
||||
| `priority=high, region=us` | no | yes |
|
||||
|
||||
The third row is the one to remember: `all` matches on **value**, not on presence. A header that
|
||||
is there with the wrong value fails the same way a missing one does.
|
||||
|
||||
Headers exchanges are slower than topic exchanges and much less common. Reach for them when the
|
||||
routing criteria are genuinely multi-dimensional and do not compose into a hierarchy.
|
||||
|
||||
## The default exchange
|
||||
|
||||
Publishing to the empty exchange name `""` routes by **queue name**, using the routing key as the
|
||||
queue name. Every queue is implicitly bound to it. That is how
|
||||
`convertAndSend("", "orders.work", message)` works, and it is the one piece of AMQP that behaves
|
||||
like a magic constant.
|
||||
|
||||
[The silent drop →](03-the-silent-drop.md)
|
||||
78
rabbitmq/docs/03-the-silent-drop.md
Normal file
78
rabbitmq/docs/03-the-silent-drop.md
Normal file
@@ -0,0 +1,78 @@
|
||||
[← Exchanges](02-exchanges.md) · [Module README](../README.md) · [Dead-lettering →](04-dead-lettering.md)
|
||||
|
||||
# 3. The message that goes nowhere and says nothing
|
||||
|
||||
Publish to `orders.direct` with routing key `amend`. No binding matches. The broker discards the
|
||||
message. `convertAndSend` returns normally. Nothing is logged, by the broker or by Spring, and
|
||||
the message is gone.
|
||||
|
||||
This is correct AMQP behaviour and it is the single most expensive default in RabbitMQ, because
|
||||
the symptom is "the consumer isn't running" and the cause is three services away.
|
||||
|
||||
**Two settings are required, and they live in different places:**
|
||||
|
||||
```yaml
|
||||
spring:
|
||||
rabbitmq:
|
||||
publisher-returns: true # on the connection factory
|
||||
template:
|
||||
mandatory: true # on the template
|
||||
```
|
||||
|
||||
`publisher-returns` puts the connection into a mode where returns are listened for at all.
|
||||
`mandatory` asks the broker to return *this* publish if it is unroutable. Setting only
|
||||
`mandatory` — which is what most examples show — changes nothing, and the test in this module
|
||||
was written that way first and failed.
|
||||
|
||||
Then register a callback:
|
||||
|
||||
```java
|
||||
template.setReturnsCallback((returned) -> log.error("unroutable: exchange={} key={} {} {}",
|
||||
returned.getExchange(), returned.getRoutingKey(),
|
||||
returned.getReplyCode(), returned.getReplyText()));
|
||||
```
|
||||
|
||||
From [`docs/output/unroutable.txt`](output/unroutable.txt):
|
||||
|
||||
```
|
||||
replyCode 312
|
||||
replyText NO_ROUTE
|
||||
exchange orders.direct
|
||||
routingKey amend
|
||||
```
|
||||
|
||||
`312 NO_ROUTE` is the code worth putting in an alert.
|
||||
|
||||
## What returns still do not give you
|
||||
|
||||
The callback fires **asynchronously, on the connection's thread**. `convertAndSend` had already
|
||||
returned successfully by then. So a return tells you a message was lost; it does not let you
|
||||
stop the caller from believing it succeeded.
|
||||
|
||||
If the caller must know, you need **publisher confirms** as well:
|
||||
|
||||
```yaml
|
||||
spring:
|
||||
rabbitmq:
|
||||
publisher-confirm-type: correlated
|
||||
```
|
||||
|
||||
Confirms and returns answer different questions, and you usually want both:
|
||||
|
||||
| | answers |
|
||||
|---|---|
|
||||
| **return** | the broker had nowhere to route this |
|
||||
| **confirm** | the broker took responsibility for this (`ack`), or refused it (`nack`) |
|
||||
|
||||
A message that is unroutable gets a **return followed by an `ack`** — the broker successfully did
|
||||
nothing with it. So a confirm alone will not tell you the message vanished, which is the trap
|
||||
inside the trap.
|
||||
|
||||
## The alternate exchange
|
||||
|
||||
The topology-level version of the same fix: give an exchange an `alternate-exchange` argument and
|
||||
unroutable messages go there instead of being dropped, with no publisher-side configuration at
|
||||
all. It costs one exchange and one queue, and it catches the publishes made by services that
|
||||
forgot to set `mandatory`.
|
||||
|
||||
[Dead-lettering →](04-dead-lettering.md)
|
||||
112
rabbitmq/docs/04-dead-lettering.md
Normal file
112
rabbitmq/docs/04-dead-lettering.md
Normal file
@@ -0,0 +1,112 @@
|
||||
[← The silent drop](03-the-silent-drop.md) · [Module README](../README.md) · [Acknowledgement →](05-acknowledgement.md)
|
||||
|
||||
# 4. A dead-letter queue that actually works
|
||||
|
||||
There is no "send to DLQ" operation in AMQP. A dead-letter exchange is a **queue argument**, and
|
||||
messages arrive there as a side effect of three specific events.
|
||||
|
||||
```java
|
||||
QueueBuilder.durable("orders.work")
|
||||
.deadLetterExchange("orders.dlx")
|
||||
.deadLetterRoutingKey("failed")
|
||||
.build();
|
||||
```
|
||||
|
||||
`deadLetterRoutingKey` matters more than it looks. Without it, the message is republished with
|
||||
its **original** routing key, so your DLQ binding has to anticipate every routing key the
|
||||
upstream might have used. Setting it to a constant means one binding catches everything.
|
||||
|
||||
## The three triggers
|
||||
|
||||
| Event | `x-death` reason |
|
||||
|---|---|
|
||||
| `basicNack` / `basicReject` with `requeue=false` | `rejected` |
|
||||
| message TTL expires (`x-message-ttl` or per-message) | `expired` |
|
||||
| queue length or byte limit exceeded (`x-max-length`) | `maxlen` |
|
||||
|
||||
And the one that is **not** a trigger: `requeue=true`. More on that below.
|
||||
|
||||
## What arrives
|
||||
|
||||
Both captured from real runs, in
|
||||
[`docs/output/dead-letter.txt`](output/dead-letter.txt):
|
||||
|
||||
```
|
||||
=== x-death after basicNack(requeue=false) ===
|
||||
reason rejected
|
||||
count 1
|
||||
exchange
|
||||
time Sat Aug 29 09:53:17 IST 2026
|
||||
routing-keys [orders.work]
|
||||
queue orders.work
|
||||
|
||||
=== x-death after x-message-ttl expiry ===
|
||||
reason expired
|
||||
count 1
|
||||
queue orders.ttl
|
||||
```
|
||||
|
||||
Same destination queue, different `reason`. That field is the whole diagnostic: `rejected` means
|
||||
a consumer looked at the work and refused it; `expired` means nobody got to it in time. Those are
|
||||
different incidents with different fixes, and they are indistinguishable without reading the
|
||||
header.
|
||||
|
||||
The third trigger has a detail worth knowing: when `x-max-length` is exceeded RabbitMQ drops from
|
||||
the **head**, so the message that gets dead-lettered is the **oldest** one already queued, not the
|
||||
one that just arrived:
|
||||
|
||||
```
|
||||
=== x-death after x-max-length overflow ===
|
||||
reason maxlen
|
||||
queue orders.bounded
|
||||
body {"orderId":"m-1","detail":"detail for m-1"}
|
||||
```
|
||||
|
||||
Three messages into a queue that holds two, and `m-1` is the one on the DLQ. A bounded queue
|
||||
under sustained overload therefore dead-letters your *backlog* while continuing to accept new
|
||||
work — which is usually what you want for telemetry and exactly wrong for orders.
|
||||
|
||||
`x-death` is a **list**, not a map — one entry per queue the message has been dead-lettered from,
|
||||
and `count` accumulates. That is how you build a retry limit: read
|
||||
`x-death[0].count`, and stop republishing past a threshold.
|
||||
|
||||
## The infinite loop
|
||||
|
||||
```java
|
||||
channel.basicNack(deliveryTag, false, true); // requeue = true
|
||||
```
|
||||
|
||||
From [`docs/output/requeue-loop.txt`](output/requeue-loop.txt), 200 delivery attempts:
|
||||
|
||||
```
|
||||
redelivered 199 times
|
||||
dead-lettered 0
|
||||
still on queue 1
|
||||
```
|
||||
|
||||
The dead-letter exchange is never consulted. `x-death` is never written. There is no counter to
|
||||
exhaust and no backoff. One poison message with an unconditional `requeue=true` handler will
|
||||
saturate a consumer indefinitely, and the queue depth stays at 1 the whole time — so a depth
|
||||
alarm will not fire either. The only visible signal is CPU.
|
||||
|
||||
**`requeue=true` is only correct when the failure is transient and you have a delay.** With no
|
||||
delay, a message that fails because a database is down comes straight back while the database is
|
||||
still down. The `redelivered` flag on the envelope is your one piece of state:
|
||||
|
||||
```java
|
||||
boolean retried = response.getEnvelope().isRedeliver();
|
||||
channel.basicNack(tag, false, !retried); // one requeue, then dead-letter
|
||||
```
|
||||
|
||||
That is a one-attempt retry with no extra infrastructure. For anything more, use a delay queue:
|
||||
a queue with a TTL and a DLX pointing back at the work exchange turns "expire after 30 seconds"
|
||||
into "retry in 30 seconds".
|
||||
|
||||
## Do not point the DLQ's DLX at its own source
|
||||
|
||||
If `orders.dlq` dead-letters to an exchange bound back to `orders.work`, a message that fails
|
||||
consistently cycles between the two forever, gaining an `x-death` entry each time until the
|
||||
header itself is the biggest thing in the message. The DLQ in this module deliberately has no
|
||||
dead-letter exchange at all.
|
||||
|
||||
[Acknowledgement →](05-acknowledgement.md)
|
||||
82
rabbitmq/docs/05-acknowledgement.md
Normal file
82
rabbitmq/docs/05-acknowledgement.md
Normal file
@@ -0,0 +1,82 @@
|
||||
[← Dead-lettering](04-dead-lettering.md) · [Module README](../README.md) · [Changing your mind →](06-changing-your-mind.md)
|
||||
|
||||
# 5. Acknowledgement
|
||||
|
||||
```yaml
|
||||
spring:
|
||||
rabbitmq:
|
||||
listener:
|
||||
simple:
|
||||
acknowledge-mode: manual
|
||||
prefetch: 1
|
||||
```
|
||||
|
||||
Three modes, and the default is not the one you want for work that matters.
|
||||
|
||||
| Mode | Behaviour |
|
||||
|---|---|
|
||||
| `auto` (default) | the container acks after the listener returns, nacks on exception |
|
||||
| `manual` | your code calls `basicAck` / `basicNack`; nothing else does |
|
||||
| `none` | the broker forgets the message the moment it is delivered |
|
||||
|
||||
(Boot publishes no default for `acknowledge-mode` either; the field initialiser in
|
||||
`AbstractMessageListenerContainer` is `AcknowledgeMode.AUTO`.)
|
||||
|
||||
`none` is genuinely fire-and-forget: no redelivery, ever, and no flow control either, because
|
||||
`prefetch` is meaningless without acknowledgement. It is the right choice for metrics and the
|
||||
wrong choice for everything else.
|
||||
|
||||
`auto` is a reasonable default and its name is misleading: it does not mean "auto-ack" in the
|
||||
AMQP sense (that is `none`). It means the container decides, and it decides by whether your
|
||||
listener threw.
|
||||
|
||||
`manual` is what you want when the acknowledgement must happen after some other durable side
|
||||
effect — a database commit, a downstream call. In `manual` mode the listener takes a `Channel`
|
||||
and a delivery tag:
|
||||
|
||||
```java
|
||||
@RabbitListener(queues = "orders.work")
|
||||
void onOrder(OrderMessage message, Channel channel,
|
||||
@Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
|
||||
try {
|
||||
process(message);
|
||||
channel.basicAck(tag, false);
|
||||
}
|
||||
catch (TransientFailure ex) {
|
||||
channel.basicNack(tag, false, !alreadyRedelivered);
|
||||
}
|
||||
catch (PermanentFailure ex) {
|
||||
channel.basicNack(tag, false, false); // dead-letter it
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The two catch blocks are the point. A single `catch (Exception)` that nacks with `requeue=true`
|
||||
is the loop from [chapter 4](04-dead-lettering.md); a single one that nacks with `requeue=false`
|
||||
dead-letters transient failures that would have succeeded on a retry. Deciding which kind of
|
||||
failure you had is work you cannot delegate to the container.
|
||||
|
||||
## `prefetch` is the flow-control knob
|
||||
|
||||
`prefetch` is how many un-acknowledged messages the broker will send a consumer. Spring Boot sets
|
||||
no default, so `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` applies, and that
|
||||
constant is **250** — throughput-oriented: a slow consumer holds 250 messages that no
|
||||
other consumer can take, so one stuck worker parks 250 messages behind it.
|
||||
|
||||
Set it to 1 when messages are expensive and unevenly sized — it gives you round-robin
|
||||
distribution across consumers. Leave it high when messages are small and uniform and you want
|
||||
throughput. This is the single most effective RabbitMQ tuning parameter and it is almost never
|
||||
touched.
|
||||
|
||||
## The forgotten ack
|
||||
|
||||
Not calling `basicAck` does not lose the message and does not throw. The message stays
|
||||
un-acknowledged, counting against `prefetch`, until the channel closes. With `prefetch: 1` the
|
||||
consumer stops after exactly one message; with the default it stops after 250. Both look like the
|
||||
consumer "just stopped", hours apart in wall-clock time depending on traffic.
|
||||
|
||||
`rabbitmqctl list_queues name messages_ready messages_unacknowledged` is the command that
|
||||
distinguishes them. A large `messages_unacknowledged` with a live consumer means somebody forgot
|
||||
an ack.
|
||||
|
||||
[Changing your mind →](06-changing-your-mind.md)
|
||||
58
rabbitmq/docs/06-changing-your-mind.md
Normal file
58
rabbitmq/docs/06-changing-your-mind.md
Normal file
@@ -0,0 +1,58 @@
|
||||
[← Acknowledgement](05-acknowledgement.md) · [Module README](../README.md)
|
||||
|
||||
# 6. Changing your mind about a queue
|
||||
|
||||
Queue arguments are immutable. There is no `ALTER QUEUE`. Redeclaring an existing queue with
|
||||
different arguments is a channel-level error
|
||||
([`docs/output/precondition-failed.txt`](output/precondition-failed.txt)):
|
||||
|
||||
```
|
||||
PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/':
|
||||
received '9999' but current is '1500', class-id=50, method-id=10
|
||||
```
|
||||
|
||||
Reply code **406**. The message is precise and helpful, and you will probably never see it,
|
||||
because of what Spring wraps it in:
|
||||
|
||||
```
|
||||
AmqpIOException: java.io.IOException
|
||||
caused by: null
|
||||
caused by: channel error; protocol method: #method<channel.close>(reply-code=406, ...)
|
||||
```
|
||||
|
||||
`AmqpIOException`'s own message is the string `java.io.IOException`. The next cause's message is
|
||||
`null`. The useful text is two levels down. This is why the failure gets reported as "some
|
||||
IOException from RabbitMQ" and why it is worth writing a log statement that walks the cause
|
||||
chain.
|
||||
|
||||
## What this means operationally
|
||||
|
||||
Changing a TTL, a max-length, a dead-letter exchange or a queue type on a live queue is not a
|
||||
config edit. It is a migration:
|
||||
|
||||
1. declare the new queue under a new name
|
||||
2. bind it alongside the old one
|
||||
3. move consumers over
|
||||
4. drain the old queue
|
||||
5. delete it
|
||||
|
||||
Plan the rename into the change from the start — `orders.work.v2` is not ugly, it is honest.
|
||||
|
||||
## Related failure modes
|
||||
|
||||
**A failed declaration stops the ones after it.** `RabbitAdmin` declares beans on connection, and
|
||||
a `PRECONDITION_FAILED` closes the channel. Declarations queued behind it on that channel do not
|
||||
happen. So one stale queue argument can leave half your topology undeclared, and the symptom is a
|
||||
*different* queue being missing.
|
||||
|
||||
**Declarations happen on connection, not on context refresh.** A broken topology does not fail
|
||||
startup; it fails the first publish. If you want it to fail at boot, force a connection early —
|
||||
`connectionFactory.createConnection()` in an `ApplicationRunner` is enough.
|
||||
|
||||
**`missing-queues-fatal` defaults differently for the two container types.** It is `true` for
|
||||
`spring.rabbitmq.listener.simple` and `false` for `spring.rabbitmq.listener.direct`. So the same
|
||||
missing queue kills a simple container and is quietly tolerated by a direct one, which makes
|
||||
"it works in that service" an unhelpful data point. Combined with the point above, a topology
|
||||
error can present as a listener startup problem in one service and as silence in another.
|
||||
|
||||
[Module README](../README.md)
|
||||
22
rabbitmq/docs/output/dead-letter.txt
Normal file
22
rabbitmq/docs/output/dead-letter.txt
Normal file
@@ -0,0 +1,22 @@
|
||||
=== x-death after basicNack(requeue=false) ===
|
||||
reason rejected
|
||||
count 1
|
||||
exchange
|
||||
time Sat Aug 29 10:30:04 IST 2026
|
||||
routing-keys [orders.work]
|
||||
queue orders.work
|
||||
=== x-death after x-message-ttl expiry ===
|
||||
reason expired
|
||||
count 1
|
||||
exchange
|
||||
time Sat Aug 29 10:30:06 IST 2026
|
||||
routing-keys [orders.ttl]
|
||||
queue orders.ttl
|
||||
=== x-death after x-max-length overflow ===
|
||||
reason maxlen
|
||||
count 1
|
||||
exchange orders.direct
|
||||
time Sat Aug 29 10:30:06 IST 2026
|
||||
routing-keys [bounded]
|
||||
queue orders.bounded
|
||||
body {"orderId":"m-1","detail":"detail for m-1"}
|
||||
2
rabbitmq/docs/output/precondition-failed.txt
Normal file
2
rabbitmq/docs/output/precondition-failed.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
=== redeclaring orders.ttl with x-message-ttl=9999 ===
|
||||
java.io.IOException | null | channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/': received '9999' but current is '1500', class-id=50, method-id=10) | channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/': received '9999' but current is '1500', class-id=50, method-id=10) |
|
||||
4
rabbitmq/docs/output/requeue-loop.txt
Normal file
4
rabbitmq/docs/output/requeue-loop.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
=== basicNack(requeue=true), 200 attempts ===
|
||||
redelivered 199 times
|
||||
dead-lettered 0
|
||||
still on queue 1
|
||||
4
rabbitmq/docs/output/tests.txt
Normal file
4
rabbitmq/docs/output/tests.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.132 s -- in com.ankurm.rabbit.TopologyTrapsTest
|
||||
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.714 s -- in com.ankurm.rabbit.DeadLetterTest
|
||||
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 s -- in com.ankurm.rabbit.MaxLengthTest
|
||||
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.476 s -- in com.ankurm.rabbit.RoutingTest
|
||||
7
rabbitmq/docs/output/topic-wildcards.txt
Normal file
7
rabbitmq/docs/output/topic-wildcards.txt
Normal file
@@ -0,0 +1,7 @@
|
||||
=== topic exchange ===
|
||||
routing key orders.eu orders.high note
|
||||
order.eu.high true true matches both
|
||||
order.eu.low true false matches order.eu.* only
|
||||
order.us.high false true matches order.#.high only
|
||||
order.eu.west.high false true matches order.#.high only - * is one word
|
||||
order.high false true matches order.#.high - # can be zero words
|
||||
5
rabbitmq/docs/output/unroutable.txt
Normal file
5
rabbitmq/docs/output/unroutable.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
=== returned message ===
|
||||
replyCode 312
|
||||
replyText NO_ROUTE
|
||||
exchange orders.direct
|
||||
routingKey amend
|
||||
67
rabbitmq/pom.xml
Normal file
67
rabbitmq/pom.xml
Normal file
@@ -0,0 +1,67 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>rabbitmq-exchanges-and-dlq</artifactId>
|
||||
<version>1.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<!-- spring-boot-starter-amqp, not a bare org.springframework.amqp:spring-rabbit. In Boot 4
|
||||
the auto-configuration lives in the spring-boot-amqp module (package
|
||||
org.springframework.boot.amqp.autoconfigure); depending on spring-rabbit alone gives
|
||||
you no RabbitTemplate and no listener container factory. Same trap as Kafka. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-amqp</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-jackson</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.amqp</groupId>
|
||||
<artifactId>spring-rabbit-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- NOT org.testcontainers:rabbitmq, which stopped at 1.21.4. Testcontainers 2.x prefixed
|
||||
every module artifact and Boot 4.1.1 imports testcontainers-bom 2.0.5. -->
|
||||
<dependency>
|
||||
<groupId>org.testcontainers</groupId>
|
||||
<artifactId>testcontainers-rabbitmq</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-testcontainers</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
28
rabbitmq/scripts/broker.sh
Executable file
28
rabbitmq/scripts/broker.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start a real RabbitMQ broker with no Docker and no root.
|
||||
#
|
||||
# RabbitMQ is an Erlang application, so it needs an Erlang runtime and epmd on the path - that
|
||||
# is the whole dependency. Point ERL_ROOT at an Erlang installation and RABBITMQ_HOME at an
|
||||
# unpacked rabbitmq-server-generic-unix tarball, and it starts in about fifteen seconds.
|
||||
#
|
||||
# If you have Docker, use Testcontainers instead: see src/test/java/.../TestcontainersConfiguration.java
|
||||
set -eu
|
||||
: "${ERL_ROOT:?set ERL_ROOT to an Erlang installation directory}"
|
||||
: "${RABBITMQ_HOME:?set RABBITMQ_HOME to an unpacked rabbitmq-server-generic-unix directory}"
|
||||
|
||||
export PATH="$ERL_ROOT/bin:$ERL_ROOT/erts-"*/bin":$PATH"
|
||||
export RABBITMQ_MNESIA_BASE="${RABBITMQ_MNESIA_BASE:-/tmp/rmq/data}"
|
||||
export RABBITMQ_LOG_BASE="${RABBITMQ_LOG_BASE:-/tmp/rmq/log}"
|
||||
export RABBITMQ_NODENAME="${RABBITMQ_NODENAME:-rabbit@localhost}"
|
||||
export HOME="${HOME:-/tmp/rmq}"
|
||||
mkdir -p "$RABBITMQ_MNESIA_BASE" "$RABBITMQ_LOG_BASE"
|
||||
|
||||
epmd -daemon 2>/dev/null || true
|
||||
sleep 1
|
||||
setsid nohup "$RABBITMQ_HOME/sbin/rabbitmq-server" > /tmp/rmq/boot.log 2>&1 < /dev/null &
|
||||
for _ in $(seq 1 60); do
|
||||
if (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; then echo "broker ready on 5672"; exit 0; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "broker did not start; see /tmp/rmq/boot.log" >&2
|
||||
exit 1
|
||||
19
rabbitmq/scripts/run-all.sh
Executable file
19
rabbitmq/scripts/run-all.sh
Executable file
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerate every file under docs/output/.
|
||||
#
|
||||
# Starts a real RabbitMQ broker first. Set ERL_ROOT and RABBITMQ_HOME (see scripts/broker.sh),
|
||||
# or point spring.rabbitmq.host at a broker you already have and skip the first line.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
./scripts/broker.sh
|
||||
mvn -B test 2>&1 | tee /tmp/rabbit-test.log > /dev/null
|
||||
|
||||
sed -n '/=== topic exchange ===/,/^order\.high/p' /tmp/rabbit-test.log > docs/output/topic-wildcards.txt
|
||||
sed -n '/=== x-death after basicNack/,/^ queue/p' /tmp/rabbit-test.log > docs/output/dead-letter.txt
|
||||
sed -n '/=== x-death after x-message-ttl/,/^ queue/p' /tmp/rabbit-test.log >> docs/output/dead-letter.txt
|
||||
sed -n '/=== x-death after x-max-length/,/^ body/p' /tmp/rabbit-test.log >> docs/output/dead-letter.txt
|
||||
sed -n '/=== basicNack(requeue=true)/,/still on queue/p' /tmp/rabbit-test.log > docs/output/requeue-loop.txt
|
||||
sed -n '/=== returned message ===/,/routingKey/p' /tmp/rabbit-test.log > docs/output/unroutable.txt
|
||||
sed -n '/=== redeclaring orders.ttl/,+1p' /tmp/rabbit-test.log > docs/output/precondition-failed.txt
|
||||
grep -E 'Tests run:.*in com\.ankurm' /tmp/rabbit-test.log | sed 's/^\[INFO\] //' > docs/output/tests.txt
|
||||
echo "regenerated:"; ls -1 docs/output/
|
||||
@@ -0,0 +1,36 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.springframework.amqp.support.converter.JacksonJsonMessageConverter;
|
||||
import org.springframework.amqp.support.converter.MessageConverter;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
/**
|
||||
* Spring Boot does <b>not</b> auto-configure a JSON message converter for RabbitMQ. The default
|
||||
* is {@code SimpleMessageConverter}, which handles {@code String}, {@code byte[]} and
|
||||
* {@code Serializable} and nothing else:
|
||||
*
|
||||
* <pre>
|
||||
* IllegalArgumentException: SimpleMessageConverter only supports String, byte[] and
|
||||
* Serializable payloads, received: com.ankurm.rabbit.OrderMessage
|
||||
* </pre>
|
||||
*
|
||||
* <p>Declaring one {@code MessageConverter} bean fixes both directions — the auto-configured
|
||||
* {@code RabbitTemplate} and the listener container factory both pick it up.
|
||||
*
|
||||
* <p>Note the class name. Spring AMQP 4.1 ships {@code Jackson2JsonMessageConverter} (Jackson 2)
|
||||
* and {@code JacksonJsonMessageConverter} (Jackson 3) side by side, exactly as Spring Kafka
|
||||
* ships {@code JsonSerializer} and {@code JacksonJsonSerializer}. Boot 4 is a Jackson 3
|
||||
* application; pick the one without the 2.
|
||||
*
|
||||
* @see <a href="../../../../../docs/01-the-on-ramp.md">docs/01-the-on-ramp.md</a>
|
||||
*/
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
public class ConverterConfiguration {
|
||||
|
||||
@Bean
|
||||
MessageConverter messageConverter() {
|
||||
return new JacksonJsonMessageConverter();
|
||||
}
|
||||
|
||||
}
|
||||
13
rabbitmq/src/main/java/com/ankurm/rabbit/OrderMessage.java
Normal file
13
rabbitmq/src/main/java/com/ankurm/rabbit/OrderMessage.java
Normal file
@@ -0,0 +1,13 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
/**
|
||||
* The payload. Kept to a String field so the tests are about routing and acknowledgement rather
|
||||
* than about converters.
|
||||
*/
|
||||
public record OrderMessage(String orderId, String detail) {
|
||||
|
||||
public static OrderMessage of(String orderId) {
|
||||
return new OrderMessage(orderId, "detail for " + orderId);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
/**
|
||||
* All four AMQP 0-9-1 exchange types, a working dead-letter path, and the traps that make
|
||||
* RabbitMQ feel unpredictable until you know them.
|
||||
*
|
||||
* <p>Everything here runs against a real broker. See the module README for how the scripts start
|
||||
* one without Docker.
|
||||
*/
|
||||
@SpringBootApplication
|
||||
public class RabbitDemoApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(RabbitDemoApplication.class, args);
|
||||
}
|
||||
|
||||
}
|
||||
217
rabbitmq/src/main/java/com/ankurm/rabbit/Topology.java
Normal file
217
rabbitmq/src/main/java/com/ankurm/rabbit/Topology.java
Normal file
@@ -0,0 +1,217 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.springframework.amqp.core.Binding;
|
||||
import org.springframework.amqp.core.BindingBuilder;
|
||||
import org.springframework.amqp.core.DirectExchange;
|
||||
import org.springframework.amqp.core.FanoutExchange;
|
||||
import org.springframework.amqp.core.HeadersExchange;
|
||||
import org.springframework.amqp.core.Queue;
|
||||
import org.springframework.amqp.core.QueueBuilder;
|
||||
import org.springframework.amqp.core.TopicExchange;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* The whole topology as beans. {@code RabbitAdmin} declares every {@code Exchange},
|
||||
* {@code Queue} and {@code Binding} bean when the connection is first opened.
|
||||
*
|
||||
* <p>Two things about that are worth knowing before you rely on it:
|
||||
* <ul>
|
||||
* <li>Declaration is <b>idempotent only if the arguments match</b>. Redeclaring an existing
|
||||
* queue with different arguments is a channel-level {@code PRECONDITION_FAILED} — see
|
||||
* docs/06-changing-your-mind.md.</li>
|
||||
* <li>Declarations happen on connection, not on context refresh, so a topology error surfaces
|
||||
* at the first publish rather than at startup.</li>
|
||||
* </ul>
|
||||
*
|
||||
* @see <a href="../../../../../docs/02-exchanges.md">docs/02-exchanges.md</a>
|
||||
*/
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
public class Topology {
|
||||
|
||||
public static final String DIRECT = "orders.direct";
|
||||
|
||||
public static final String FANOUT = "orders.fanout";
|
||||
|
||||
public static final String TOPIC = "orders.topic";
|
||||
|
||||
public static final String HEADERS = "orders.headers";
|
||||
|
||||
public static final String DLX = "orders.dlx";
|
||||
|
||||
public static final String Q_NEW = "orders.new";
|
||||
|
||||
public static final String Q_CANCEL = "orders.cancel";
|
||||
|
||||
public static final String Q_AUDIT = "audit.all";
|
||||
|
||||
public static final String Q_ANALYTICS = "analytics.all";
|
||||
|
||||
public static final String Q_EU = "orders.eu";
|
||||
|
||||
public static final String Q_HIGH = "orders.high";
|
||||
|
||||
public static final String Q_PRIORITY = "orders.priority";
|
||||
|
||||
public static final String Q_ANY = "orders.any";
|
||||
|
||||
public static final String Q_WORK = "orders.work";
|
||||
|
||||
public static final String Q_TTL = "orders.ttl";
|
||||
|
||||
public static final String Q_DLQ = "orders.dlq";
|
||||
|
||||
// --- direct: routing key must match the binding key exactly -----------------------------
|
||||
|
||||
@Bean
|
||||
DirectExchange directExchange() {
|
||||
return new DirectExchange(DIRECT);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue newQueue() {
|
||||
return QueueBuilder.durable(Q_NEW).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue cancelQueue() {
|
||||
return QueueBuilder.durable(Q_CANCEL).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindNew(DirectExchange directExchange, Queue newQueue) {
|
||||
return BindingBuilder.bind(newQueue).to(directExchange).with("new");
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindCancel(DirectExchange directExchange, Queue cancelQueue) {
|
||||
return BindingBuilder.bind(cancelQueue).to(directExchange).with("cancel");
|
||||
}
|
||||
|
||||
// --- fanout: routing key ignored entirely -----------------------------------------------
|
||||
|
||||
@Bean
|
||||
FanoutExchange fanoutExchange() {
|
||||
return new FanoutExchange(FANOUT);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue auditQueue() {
|
||||
return QueueBuilder.durable(Q_AUDIT).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue analyticsQueue() {
|
||||
return QueueBuilder.durable(Q_ANALYTICS).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindAudit(FanoutExchange fanoutExchange, Queue auditQueue) {
|
||||
return BindingBuilder.bind(auditQueue).to(fanoutExchange);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindAnalytics(FanoutExchange fanoutExchange, Queue analyticsQueue) {
|
||||
return BindingBuilder.bind(analyticsQueue).to(fanoutExchange);
|
||||
}
|
||||
|
||||
// --- topic: '*' is exactly one word, '#' is zero or more -------------------------------
|
||||
|
||||
@Bean
|
||||
TopicExchange topicExchange() {
|
||||
return new TopicExchange(TOPIC);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue euQueue() {
|
||||
return QueueBuilder.durable(Q_EU).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue highQueue() {
|
||||
return QueueBuilder.durable(Q_HIGH).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindEu(TopicExchange topicExchange, Queue euQueue) {
|
||||
return BindingBuilder.bind(euQueue).to(topicExchange).with("order.eu.*");
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindHigh(TopicExchange topicExchange, Queue highQueue) {
|
||||
return BindingBuilder.bind(highQueue).to(topicExchange).with("order.#.high");
|
||||
}
|
||||
|
||||
// --- headers: routing key ignored, header map matched ------------------------------------
|
||||
|
||||
@Bean
|
||||
HeadersExchange headersExchange() {
|
||||
return new HeadersExchange(HEADERS);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue priorityQueue() {
|
||||
return QueueBuilder.durable(Q_PRIORITY).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue anyQueue() {
|
||||
return QueueBuilder.durable(Q_ANY).build();
|
||||
}
|
||||
|
||||
/** x-match=all: every header must match. */
|
||||
@Bean
|
||||
Binding bindPriority(HeadersExchange headersExchange, Queue priorityQueue) {
|
||||
return BindingBuilder.bind(priorityQueue).to(headersExchange)
|
||||
.whereAll(Map.of("priority", "high", "region", "eu")).match();
|
||||
}
|
||||
|
||||
/** x-match=any: one is enough. */
|
||||
@Bean
|
||||
Binding bindAny(HeadersExchange headersExchange, Queue anyQueue) {
|
||||
return BindingBuilder.bind(anyQueue).to(headersExchange)
|
||||
.whereAny(Map.of("priority", "high", "region", "eu")).match();
|
||||
}
|
||||
|
||||
// --- the dead-letter path ----------------------------------------------------------------
|
||||
|
||||
@Bean
|
||||
DirectExchange deadLetterExchange() {
|
||||
return new DirectExchange(DLX);
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue deadLetterQueue() {
|
||||
// Deliberately NO dead-letter-exchange on the DLQ itself. Pointing a DLQ's DLX back at
|
||||
// the exchange that feeds it is the classic infinite loop.
|
||||
return QueueBuilder.durable(Q_DLQ).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Binding bindDlq(DirectExchange deadLetterExchange, Queue deadLetterQueue) {
|
||||
// The routing key used when dead-lettering is the message's ORIGINAL routing key unless
|
||||
// deadLetterRoutingKey() overrides it. Both work queues below set it explicitly to
|
||||
// "failed", so this one binding catches everything.
|
||||
return BindingBuilder.bind(deadLetterQueue).to(deadLetterExchange).with("failed");
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue workQueue() {
|
||||
return QueueBuilder.durable(Q_WORK)
|
||||
.deadLetterExchange(DLX)
|
||||
.deadLetterRoutingKey("failed")
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
Queue ttlQueue() {
|
||||
return QueueBuilder.durable(Q_TTL)
|
||||
.ttl(1500)
|
||||
.deadLetterExchange(DLX)
|
||||
.deadLetterRoutingKey("failed")
|
||||
.build();
|
||||
}
|
||||
|
||||
}
|
||||
31
rabbitmq/src/main/resources/application.yaml
Normal file
31
rabbitmq/src/main/resources/application.yaml
Normal file
@@ -0,0 +1,31 @@
|
||||
spring:
|
||||
application:
|
||||
name: rabbitmq-exchanges-and-dlq
|
||||
main:
|
||||
banner-mode: off
|
||||
rabbitmq:
|
||||
host: localhost
|
||||
port: 5672
|
||||
username: guest
|
||||
password: guest
|
||||
# BOTH of these are required to find out about an unroutable message, and they are set in
|
||||
# two different places. publisher-returns puts the connection factory into a mode where it
|
||||
# listens for returns at all; mandatory asks the broker to return this particular publish.
|
||||
# Setting only mandatory - which is what most examples show - changes nothing at all.
|
||||
# See docs/03-the-silent-drop.md.
|
||||
publisher-returns: true
|
||||
template:
|
||||
mandatory: true
|
||||
listener:
|
||||
simple:
|
||||
# MANUAL means nothing is removed from the queue until your code says so - and that
|
||||
# basicNack(requeue=false) is what feeds the dead-letter exchange.
|
||||
acknowledge-mode: manual
|
||||
prefetch: 1
|
||||
# Listener containers are not started automatically here; each test starts the one it
|
||||
# needs, so that a redelivery loop in one test cannot consume another test's messages.
|
||||
auto-startup: false
|
||||
logging:
|
||||
level:
|
||||
root: WARN
|
||||
com.ankurm: INFO
|
||||
143
rabbitmq/src/test/java/com/ankurm/rabbit/DeadLetterTest.java
Normal file
143
rabbitmq/src/test/java/com/ankurm/rabbit/DeadLetterTest.java
Normal file
@@ -0,0 +1,143 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import com.rabbitmq.client.GetResponse;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.amqp.rabbit.core.RabbitAdmin;
|
||||
import org.springframework.amqp.rabbit.core.RabbitTemplate;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* The dead-letter path, driven with the AMQP primitives directly rather than through a listener
|
||||
* container. {@code basicGet} + {@code basicNack} is what a container does underneath, and doing
|
||||
* it by hand removes all the timing from the test.
|
||||
*
|
||||
* @see <a href="../../../../../docs/04-dead-lettering.md">docs/04-dead-lettering.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
class DeadLetterTest {
|
||||
|
||||
@Autowired
|
||||
RabbitTemplate template;
|
||||
|
||||
@Autowired
|
||||
RabbitAdmin admin;
|
||||
|
||||
@BeforeEach
|
||||
void drain() {
|
||||
List.of(Topology.Q_WORK, Topology.Q_TTL, Topology.Q_DLQ)
|
||||
.forEach((q) -> this.admin.purgeQueue(q, false));
|
||||
}
|
||||
|
||||
private int depth(String queue) {
|
||||
Properties properties = this.admin.getQueueProperties(queue);
|
||||
return (properties == null) ? -1
|
||||
: ((Number) properties.get(RabbitAdmin.QUEUE_MESSAGE_COUNT)).intValue();
|
||||
}
|
||||
|
||||
private int settled(String queue, int expected) throws InterruptedException {
|
||||
for (int i = 0; i < 150; i++) {
|
||||
if (depth(queue) == expected) {
|
||||
return expected;
|
||||
}
|
||||
Thread.sleep(20);
|
||||
}
|
||||
return depth(queue);
|
||||
}
|
||||
|
||||
@Test
|
||||
void nackWithoutRequeueDeadLettersAndStampsTheReason() throws Exception {
|
||||
this.template.convertAndSend(Topology.DIRECT, "ignored", OrderMessage.of("o-1"),
|
||||
(m) -> m, null);
|
||||
// Publish straight to the work queue via the default exchange: the empty exchange name
|
||||
// routes by queue name, which is the one piece of AMQP that behaves like a magic
|
||||
// constant and is worth knowing.
|
||||
this.template.convertAndSend("", Topology.Q_WORK, OrderMessage.of("o-1"));
|
||||
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
|
||||
|
||||
this.template.execute((channel) -> {
|
||||
GetResponse response = channel.basicGet(Topology.Q_WORK, false);
|
||||
assertThat(response).isNotNull();
|
||||
// requeue=false is the entire dead-letter trigger. There is no separate "send to
|
||||
// DLQ" call in AMQP.
|
||||
channel.basicNack(response.getEnvelope().getDeliveryTag(), false, false);
|
||||
return null;
|
||||
});
|
||||
|
||||
assertThat(settled(Topology.Q_DLQ, 1)).isEqualTo(1);
|
||||
assertThat(depth(Topology.Q_WORK)).isZero();
|
||||
|
||||
Map<String, Object> death = firstDeath(Topology.Q_DLQ);
|
||||
System.out.println("=== x-death after basicNack(requeue=false) ===");
|
||||
death.forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
|
||||
assertThat(death).containsEntry("reason", "rejected");
|
||||
assertThat(death).containsEntry("queue", Topology.Q_WORK);
|
||||
assertThat(((Number) death.get("count")).intValue()).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void messageTtlExpiryAlsoDeadLettersButWithADifferentReason() throws Exception {
|
||||
this.template.convertAndSend("", Topology.Q_TTL, OrderMessage.of("o-ttl"));
|
||||
// orders.ttl carries x-message-ttl=1500. Nobody consumes it; the broker expires it.
|
||||
assertThat(settled(Topology.Q_DLQ, 1)).isEqualTo(1);
|
||||
|
||||
Map<String, Object> death = firstDeath(Topology.Q_DLQ);
|
||||
System.out.println("=== x-death after x-message-ttl expiry ===");
|
||||
death.forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
|
||||
// Same destination, different reason. This is the field that tells an operator whether
|
||||
// the consumer rejected the work or never got to it.
|
||||
assertThat(death).containsEntry("reason", "expired");
|
||||
assertThat(death).containsEntry("queue", Topology.Q_TTL);
|
||||
}
|
||||
|
||||
@Test
|
||||
void nackWithRequeueIsAnInfiniteLoopAndNeverDeadLetters() throws Exception {
|
||||
this.template.convertAndSend("", Topology.Q_WORK, OrderMessage.of("o-loop"));
|
||||
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
|
||||
|
||||
int redeliveries = this.template.execute((channel) -> {
|
||||
int count = 0;
|
||||
for (int i = 0; i < 200; i++) {
|
||||
GetResponse response = channel.basicGet(Topology.Q_WORK, false);
|
||||
if (response == null) {
|
||||
break;
|
||||
}
|
||||
if (response.getEnvelope().isRedeliver()) {
|
||||
count++;
|
||||
}
|
||||
// requeue=true puts it back. The dead-letter exchange is never consulted,
|
||||
// x-death is never written, and the loop has no counter to exhaust.
|
||||
channel.basicNack(response.getEnvelope().getDeliveryTag(), false, true);
|
||||
}
|
||||
return count;
|
||||
});
|
||||
|
||||
System.out.println("=== basicNack(requeue=true), 200 attempts ===");
|
||||
System.out.println(" redelivered " + redeliveries + " times");
|
||||
System.out.println(" dead-lettered " + depth(Topology.Q_DLQ));
|
||||
System.out.println(" still on queue " + depth(Topology.Q_WORK));
|
||||
|
||||
assertThat(redeliveries).isGreaterThan(150);
|
||||
// The message is exactly where it started, having been processed 200 times.
|
||||
assertThat(depth(Topology.Q_DLQ)).isZero();
|
||||
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private Map<String, Object> firstDeath(String queue) {
|
||||
var message = this.template.receive(queue, 5000);
|
||||
assertThat(message).isNotNull();
|
||||
List<Map<String, Object>> deaths =
|
||||
(List<Map<String, Object>>) message.getMessageProperties().getHeader("x-death");
|
||||
assertThat(deaths).isNotNull().isNotEmpty();
|
||||
return deaths.get(0);
|
||||
}
|
||||
|
||||
}
|
||||
73
rabbitmq/src/test/java/com/ankurm/rabbit/MaxLengthTest.java
Normal file
73
rabbitmq/src/test/java/com/ankurm/rabbit/MaxLengthTest.java
Normal file
@@ -0,0 +1,73 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import com.rabbitmq.client.GetResponse;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.amqp.core.Binding;
|
||||
import org.springframework.amqp.core.BindingBuilder;
|
||||
import org.springframework.amqp.core.DirectExchange;
|
||||
import org.springframework.amqp.core.Queue;
|
||||
import org.springframework.amqp.core.QueueBuilder;
|
||||
import org.springframework.amqp.rabbit.core.RabbitAdmin;
|
||||
import org.springframework.amqp.rabbit.core.RabbitTemplate;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* The third dead-letter trigger: a queue that is full.
|
||||
*
|
||||
* <p>Included because the article states all three triggers in one table, and this was the only
|
||||
* row not produced by a run. It is now.
|
||||
*
|
||||
* @see <a href="../../../../../docs/04-dead-lettering.md">docs/04-dead-lettering.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
class MaxLengthTest {
|
||||
|
||||
static final String Q_BOUNDED = "orders.bounded";
|
||||
|
||||
@Autowired
|
||||
RabbitTemplate template;
|
||||
|
||||
@Autowired
|
||||
RabbitAdmin admin;
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("unchecked")
|
||||
void exceedingMaxLengthDeadLettersTheOldestWithReasonMaxlen() throws Exception {
|
||||
this.admin.deleteQueue(Q_BOUNDED);
|
||||
Queue bounded = QueueBuilder.durable(Q_BOUNDED)
|
||||
.maxLength(2)
|
||||
.deadLetterExchange(Topology.DLX)
|
||||
.deadLetterRoutingKey("failed")
|
||||
.build();
|
||||
this.admin.declareQueue(bounded);
|
||||
Binding binding = BindingBuilder.bind(bounded)
|
||||
.to(new DirectExchange(Topology.DIRECT)).with("bounded");
|
||||
this.admin.declareBinding(binding);
|
||||
this.admin.purgeQueue(Topology.Q_DLQ, false);
|
||||
|
||||
// Three messages into a queue that holds two. RabbitMQ drops from the HEAD, so the
|
||||
// FIRST message is the one dead-lettered - the oldest, not the newest.
|
||||
for (String id : List.of("m-1", "m-2", "m-3")) {
|
||||
this.template.convertAndSend(Topology.DIRECT, "bounded", OrderMessage.of(id));
|
||||
}
|
||||
|
||||
var message = this.template.receive(Topology.Q_DLQ, 10_000);
|
||||
assertThat(message).isNotNull();
|
||||
List<Map<String, Object>> deaths =
|
||||
(List<Map<String, Object>>) message.getMessageProperties().getHeader("x-death");
|
||||
System.out.println("=== x-death after x-max-length overflow ===");
|
||||
deaths.get(0).forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
|
||||
System.out.println(" body " + new String(message.getBody()));
|
||||
|
||||
assertThat(deaths.get(0)).containsEntry("reason", "maxlen");
|
||||
assertThat(deaths.get(0)).containsEntry("queue", Q_BOUNDED);
|
||||
assertThat(new String(message.getBody())).contains("m-1");
|
||||
}
|
||||
|
||||
}
|
||||
152
rabbitmq/src/test/java/com/ankurm/rabbit/RoutingTest.java
Normal file
152
rabbitmq/src/test/java/com/ankurm/rabbit/RoutingTest.java
Normal file
@@ -0,0 +1,152 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.amqp.core.MessageProperties;
|
||||
import org.springframework.amqp.rabbit.core.RabbitAdmin;
|
||||
import org.springframework.amqp.rabbit.core.RabbitTemplate;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* All four exchange types, driven with real publishes and counted by draining the queues.
|
||||
*
|
||||
* @see <a href="../../../../../docs/02-exchanges.md">docs/02-exchanges.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
class RoutingTest {
|
||||
|
||||
@Autowired
|
||||
RabbitTemplate template;
|
||||
|
||||
@Autowired
|
||||
RabbitAdmin admin;
|
||||
|
||||
private static final List<String> QUEUES = List.of(Topology.Q_NEW, Topology.Q_CANCEL,
|
||||
Topology.Q_AUDIT, Topology.Q_ANALYTICS, Topology.Q_EU, Topology.Q_HIGH,
|
||||
Topology.Q_PRIORITY, Topology.Q_ANY);
|
||||
|
||||
@BeforeEach
|
||||
void drain() {
|
||||
QUEUES.forEach((q) -> this.admin.purgeQueue(q, false));
|
||||
}
|
||||
|
||||
/**
|
||||
* Ready message count, straight from the broker.
|
||||
*
|
||||
* <p>Note the {@code Number}: {@code RabbitAdmin.QUEUE_MESSAGE_COUNT} holds a {@code Long}
|
||||
* in Spring AMQP 4.1. Casting it to {@code Integer}, as every older example does, is a
|
||||
* {@code ClassCastException} at runtime.
|
||||
*/
|
||||
private int depth(String queue) {
|
||||
Properties properties = this.admin.getQueueProperties(queue);
|
||||
return (properties == null) ? -1
|
||||
: ((Number) properties.get(RabbitAdmin.QUEUE_MESSAGE_COUNT)).intValue();
|
||||
}
|
||||
|
||||
/** Publishing is asynchronous; give the broker a moment to route before counting. */
|
||||
private int settledDepth(String queue) {
|
||||
int last = -1;
|
||||
for (int i = 0; i < 50; i++) {
|
||||
last = depth(queue);
|
||||
if (last > 0) {
|
||||
return last;
|
||||
}
|
||||
try {
|
||||
Thread.sleep(20);
|
||||
}
|
||||
catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
return last;
|
||||
}
|
||||
|
||||
@Test
|
||||
void directExchangeMatchesTheRoutingKeyExactly() {
|
||||
this.template.convertAndSend(Topology.DIRECT, "new", OrderMessage.of("o-1"));
|
||||
this.template.convertAndSend(Topology.DIRECT, "cancel", OrderMessage.of("o-2"));
|
||||
// No binding for "amend". The broker discards it - see UnroutableTest.
|
||||
this.template.convertAndSend(Topology.DIRECT, "amend", OrderMessage.of("o-3"));
|
||||
|
||||
assertThat(settledDepth(Topology.Q_NEW)).isEqualTo(1);
|
||||
assertThat(settledDepth(Topology.Q_CANCEL)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void fanoutIgnoresTheRoutingKeyCompletely() {
|
||||
this.template.convertAndSend(Topology.FANOUT, "this-is-ignored", OrderMessage.of("o-1"));
|
||||
|
||||
assertThat(settledDepth(Topology.Q_AUDIT)).isEqualTo(1);
|
||||
assertThat(settledDepth(Topology.Q_ANALYTICS)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void topicWildcardsAreWordsNotCharacters() {
|
||||
// binding "order.eu.*" - exactly one word after order.eu
|
||||
// binding "order.#.high" - zero or more words between order. and .high
|
||||
Map<String, String> routingKeys = new LinkedHashMap<>();
|
||||
routingKeys.put("order.eu.high", "matches both");
|
||||
routingKeys.put("order.eu.low", "matches order.eu.* only");
|
||||
routingKeys.put("order.us.high", "matches order.#.high only");
|
||||
routingKeys.put("order.eu.west.high", "matches order.#.high only - * is one word");
|
||||
routingKeys.put("order.high", "matches order.#.high - # can be zero words");
|
||||
|
||||
System.out.println("=== topic exchange ===");
|
||||
System.out.printf("%-24s %-12s %-12s %s%n", "routing key", "orders.eu", "orders.high", "note");
|
||||
for (Map.Entry<String, String> entry : routingKeys.entrySet()) {
|
||||
this.admin.purgeQueue(Topology.Q_EU, false);
|
||||
this.admin.purgeQueue(Topology.Q_HIGH, false);
|
||||
this.template.convertAndSend(Topology.TOPIC, entry.getKey(), OrderMessage.of("o"));
|
||||
System.out.printf("%-24s %-12s %-12s %s%n", entry.getKey(), settledDepth(Topology.Q_EU) == 1,
|
||||
settledDepth(Topology.Q_HIGH) == 1, entry.getValue());
|
||||
}
|
||||
|
||||
this.admin.purgeQueue(Topology.Q_EU, false);
|
||||
this.admin.purgeQueue(Topology.Q_HIGH, false);
|
||||
// The one everybody gets wrong: '*' is one WORD, so it does not match two.
|
||||
this.template.convertAndSend(Topology.TOPIC, "order.eu.west.high", OrderMessage.of("o"));
|
||||
assertThat(depth(Topology.Q_EU)).isEqualTo(0);
|
||||
assertThat(settledDepth(Topology.Q_HIGH)).isEqualTo(1);
|
||||
|
||||
this.admin.purgeQueue(Topology.Q_HIGH, false);
|
||||
// And '#' really does match zero words.
|
||||
this.template.convertAndSend(Topology.TOPIC, "order.high", OrderMessage.of("o"));
|
||||
assertThat(settledDepth(Topology.Q_HIGH)).isEqualTo(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void headersExchangeMatchesTheHeaderMapNotTheRoutingKey() {
|
||||
send(Map.of("priority", "high", "region", "eu"));
|
||||
assertThat(settledDepth(Topology.Q_PRIORITY)).isEqualTo(1); // x-match=all: both present
|
||||
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1); // x-match=any: either is enough
|
||||
|
||||
drain();
|
||||
send(Map.of("priority", "high"));
|
||||
assertThat(depth(Topology.Q_PRIORITY)).isEqualTo(0); // all: region missing
|
||||
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1);
|
||||
|
||||
drain();
|
||||
send(Map.of("priority", "high", "region", "us"));
|
||||
// x-match=all needs every header to match by VALUE, not merely to be present.
|
||||
assertThat(depth(Topology.Q_PRIORITY)).isEqualTo(0);
|
||||
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1);
|
||||
}
|
||||
|
||||
private void send(Map<String, Object> headers) {
|
||||
this.template.convertAndSend(Topology.HEADERS, "routing-key-is-ignored",
|
||||
OrderMessage.of("o-1"), (message) -> {
|
||||
MessageProperties properties = message.getMessageProperties();
|
||||
headers.forEach(properties::setHeader);
|
||||
return message;
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.springframework.boot.test.context.TestConfiguration;
|
||||
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.testcontainers.rabbitmq.RabbitMQContainer;
|
||||
import org.testcontainers.utility.DockerImageName;
|
||||
|
||||
/**
|
||||
* The Testcontainers route, for machines that have a Docker daemon.
|
||||
*
|
||||
* <p>{@code @ServiceConnection} supplies host, port, username and password to the
|
||||
* auto-configuration, so no {@code spring.rabbitmq.*} property and no
|
||||
* {@code @DynamicPropertySource} block is needed.
|
||||
*
|
||||
* <p>The Maven coordinate is <b>{@code org.testcontainers:testcontainers-rabbitmq}</b>.
|
||||
* Testcontainers 2.x prefixed every module artifact; the old {@code org.testcontainers:rabbitmq}
|
||||
* stopped at 1.21.4 and is not managed by the Boot 4.1 BOM.
|
||||
*
|
||||
* <p>The committed transcripts under {@code docs/output/} came from a broker started by
|
||||
* {@code scripts/broker.sh} instead, on a machine with no Docker — see the module README
|
||||
* for why that matters and what version it was.
|
||||
*/
|
||||
@TestConfiguration(proxyBeanMethods = false)
|
||||
public class TestcontainersConfiguration {
|
||||
|
||||
@Bean
|
||||
@ServiceConnection
|
||||
RabbitMQContainer rabbitContainer() {
|
||||
return new RabbitMQContainer(DockerImageName.parse("rabbitmq:4.1-management"));
|
||||
}
|
||||
|
||||
}
|
||||
100
rabbitmq/src/test/java/com/ankurm/rabbit/TopologyTrapsTest.java
Normal file
100
rabbitmq/src/test/java/com/ankurm/rabbit/TopologyTrapsTest.java
Normal file
@@ -0,0 +1,100 @@
|
||||
package com.ankurm.rabbit;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.amqp.AmqpIOException;
|
||||
import org.springframework.amqp.core.Queue;
|
||||
import org.springframework.amqp.core.QueueBuilder;
|
||||
import org.springframework.amqp.core.ReturnedMessage;
|
||||
import org.springframework.amqp.rabbit.core.RabbitAdmin;
|
||||
import org.springframework.amqp.rabbit.core.RabbitTemplate;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatExceptionOfType;
|
||||
|
||||
/**
|
||||
* Two ways a RabbitMQ topology loses your work quietly.
|
||||
*
|
||||
* @see <a href="../../../../../docs/03-the-silent-drop.md">docs/03-the-silent-drop.md</a>
|
||||
* @see <a href="../../../../../docs/06-changing-your-mind.md">docs/06-changing-your-mind.md</a>
|
||||
*/
|
||||
@SpringBootTest
|
||||
class TopologyTrapsTest {
|
||||
|
||||
@Autowired
|
||||
RabbitTemplate template;
|
||||
|
||||
@Autowired
|
||||
RabbitAdmin admin;
|
||||
|
||||
@Test
|
||||
void anUnroutableMessageIsDiscardedUnlessYouAskForItBack() throws Exception {
|
||||
List<ReturnedMessage> returned = new CopyOnWriteArrayList<>();
|
||||
this.template.setReturnsCallback(returned::add);
|
||||
|
||||
// "amend" matches no binding on orders.direct. The broker has nowhere to put it.
|
||||
this.template.convertAndSend(Topology.DIRECT, "amend", OrderMessage.of("o-1"));
|
||||
for (int i = 0; i < 100 && returned.isEmpty(); i++) {
|
||||
Thread.sleep(20);
|
||||
}
|
||||
|
||||
// spring.rabbitmq.template.mandatory=true is what turns a silent discard into a return.
|
||||
assertThat(this.template.isMandatoryFor(null)).isTrue();
|
||||
assertThat(returned).hasSize(1);
|
||||
ReturnedMessage message = returned.get(0);
|
||||
System.out.println("=== returned message ===");
|
||||
System.out.println(" replyCode " + message.getReplyCode());
|
||||
System.out.println(" replyText " + message.getReplyText());
|
||||
System.out.println(" exchange " + message.getExchange());
|
||||
System.out.println(" routingKey " + message.getRoutingKey());
|
||||
|
||||
assertThat(message.getReplyCode()).isEqualTo(312);
|
||||
assertThat(message.getReplyText()).isEqualTo("NO_ROUTE");
|
||||
assertThat(message.getRoutingKey()).isEqualTo("amend");
|
||||
// Note what did NOT happen: convertAndSend returned normally. Publishing is fire and
|
||||
// forget at the protocol level, so even with mandatory=true the failure arrives
|
||||
// asynchronously on another thread. Nothing throws.
|
||||
}
|
||||
|
||||
@Test
|
||||
void redeclaringAQueueWithDifferentArgumentsIsAPreconditionFailure() {
|
||||
// orders.ttl already exists with x-message-ttl=1500. Same name, different argument.
|
||||
Queue conflicting = QueueBuilder.durable(Topology.Q_TTL)
|
||||
.ttl(9999)
|
||||
.deadLetterExchange(Topology.DLX)
|
||||
.deadLetterRoutingKey("failed")
|
||||
.build();
|
||||
|
||||
assertThatExceptionOfType(AmqpIOException.class)
|
||||
.isThrownBy(() -> this.admin.declareQueue(conflicting))
|
||||
.satisfies((ex) -> {
|
||||
String detail = rootMessage(ex);
|
||||
System.out.println("=== redeclaring orders.ttl with x-message-ttl=9999 ===");
|
||||
System.out.println(" " + detail);
|
||||
assertThat(detail).contains("PRECONDITION_FAILED")
|
||||
.contains("inequivalent arg 'x-message-ttl'");
|
||||
});
|
||||
|
||||
// Queue arguments are immutable. There is no ALTER QUEUE. Changing a TTL, a max-length
|
||||
// or a dead-letter exchange on an existing queue means: declare a new queue, move the
|
||||
// consumers, drain the old one, delete it. Plan the rename into the change.
|
||||
}
|
||||
|
||||
private static String rootMessage(Throwable throwable) {
|
||||
// The useful text is on the cause. AmqpIOException's own message is just
|
||||
// "java.io.IOException", which is why this failure is so often reported as "IOException"
|
||||
// with no further detail.
|
||||
Throwable current = throwable;
|
||||
StringBuilder all = new StringBuilder();
|
||||
while (current != null) {
|
||||
all.append(current.getMessage()).append(" | ");
|
||||
current = current.getCause();
|
||||
}
|
||||
return all.toString();
|
||||
}
|
||||
|
||||
}
|
||||
Reference in New Issue
Block a user