1
0

Compare commits

...

3 Commits

Author SHA1 Message Date
e1f8aa7402 Add the broker-comparison module
Kafka, RabbitMQ and Pulsar measured side by side on ordering, replay and
consumer scaling by driving the three client libraries directly, plus an
operational-footprint measurement of each broker's own distribution.

Nine tests, three brokers started without Docker, and every number in the
documentation regenerated by scripts/run-all.sh.
2026-09-01 23:31:51 +05:30
f4b0112e2c Add the kafka-error-handling module 2026-08-29 10:30:21 +05:30
de9fc5ce4c Add the rabbitmq module 2026-08-29 10:30:21 +05:30
81 changed files with 4697 additions and 2 deletions

View File

@@ -8,6 +8,16 @@ module's `scripts/run-all.sh`, never typed by hand.
| Module | Article | What it demonstrates |
|---|---|---|
| [`kafka-basics/`](kafka-basics/README.md) | [Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/) | The on-ramp: what the starter gives you, the two Jackson serializer families, where a key lands and why, and which defaults are Kafka's rather than Spring's |
| [`kafka-error-handling/`](kafka-error-handling/README.md) | [Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/) | What the default error handler really does, why a poison pill stops a partition, and what blocking retries cost that retry topics do not |
| [`rabbitmq/`](rabbitmq/README.md) | [Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/) | All four exchange types against a real broker, manual acknowledgement, and a dead-letter path exercised through both rejection and TTL expiry |
| [`broker-comparison/`](broker-comparison/README.md) | [Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with Benchmarks](https://ankurm.com/kafka-vs-rabbitmq-vs-pulsar-java-decision-framework/) | The three brokers measured side by side on ordering, replay, consumer scaling and operational footprint, ending in a decision table where every row has a transcript behind it |
The brokers make an instructive set. Kafka's consumer holds an offset and the broker remembers
nothing about individual records; RabbitMQ's broker owns the message until it is acknowledged and
can route, expire and dead-letter it on its own; Pulsar keeps the log but lets each subscriber
choose how it is read. Almost every difference in how you handle failure follows from those three
sentences — which is why Kafka needs a retry topic to do what RabbitMQ does with a queue
argument, and why `broker-comparison` is mostly an argument about where a message lives.
They are meant to be read in order. `kafka-basics` establishes that the default acknowledgement
mode is `BATCH` and therefore that delivery is at-least-once; everything the error-handling
@@ -21,8 +31,8 @@ Central and from Boot's own `spring-boot-dependencies` POM, rather than from rel
announcements.
Every module runs its broker without Docker, so the transcripts can be regenerated on any
machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ via a
real broker started by the module's own scripts. Each module also ships a Testcontainers
machine with a JDK: Kafka via the in-process KRaft broker in `spring-kafka-test`, RabbitMQ and
Pulsar via real brokers started by the modules' own scripts. Each module also ships a Testcontainers
configuration for the cases where the deployed image is what you need to test against.
## Licence

View File

@@ -0,0 +1,80 @@
# broker-comparison
Companion project for **Kafka vs RabbitMQ vs Pulsar for Java Teams: A Decision Framework with
Benchmarks** on [ankurm.com](https://ankurm.com).
Three brokers, three questions, one set of measurements. Ordering, replay and consumer scaling are
asked identically of all three by driving the client libraries directly, so that what is compared
is the brokers' delivery models rather than three sets of Spring defaults. What each Spring
integration adds on top is [chapter 6](docs/06-what-spring-adds.md).
## Verified stack
| Component | Version | Source of the number |
|---|---|---|
| JDK (tests) | 25.0.4.1+1 (Temurin) | `java -version` |
| JDK (brokers) | 21.0.12.1+1 (Temurin) | `java -version` |
| Spring Boot | 4.1.1 | `maven-metadata.xml` on Maven Central |
| kafka-clients | 4.2.1 | `spring-boot-dependencies-4.1.1.pom` |
| amqp-client | 5.30.0 | `spring-boot-dependencies-4.1.1.pom` |
| pulsar-client | 4.2.4 | `spring-boot-dependencies-4.1.1.pom` |
| Kafka broker | 4.2.1 (KRaft) | `kafka_2.13-4.2.1.tgz` |
| RabbitMQ broker | 3.10.25 | generic-unix tarball |
| Pulsar broker | 4.2.4 standalone | `apache-pulsar-4.2.4-bin.tar.gz` |
RabbitMQ is pinned at 3.10.25 because the box that produced these transcripts has Erlang 24;
3.11 and later need Erlang 25. The AMQP 0-9-1 semantics measured here are unchanged in 4.x.
## Quickstart
The Kafka measurements need no external broker — `spring-kafka-test` starts a real KRaft broker
in-process. The other two need a broker each, and the scripts start both without Docker and
without root.
```bash
mvn -Dgroups=kafka test
ERL_ROOT=... RABBITMQ_HOME=... scripts/rabbit-broker.sh
mvn -Dgroups=rabbit test
PULSAR_HOME=... scripts/pulsar-broker.sh
mvn -Dgroups=pulsar test
scripts/footprint.sh kafka|rabbit|pulsar # the operational numbers
scripts/run-all.sh # regenerates every docs/output/ file
```
## Documentation
| Chapter | What it settles |
|---|---|
| [01 Three models](docs/01-three-models.md) | Where a message lives, and why everything else follows |
| [02 Ordering](docs/02-ordering.md) | What survives the second consumer, in all three |
| [03 Replay](docs/03-replay.md) | Reading it twice, and the one broker that cannot |
| [04 Consumer scaling](docs/04-consumer-scaling.md) | The partition ceiling, and the client buffer that defeats fan-out in the other two |
| [05 Operational footprint](docs/05-operational-footprint.md) | Startup, memory, ports, processes, configuration surface |
| [06 What Spring adds](docs/06-what-spring-adds.md) | The three integrations, and the Boot 4 starter trap |
| [07 The decision table](docs/07-the-decision-table.md) | Every row backed by a measurement, and how to choose |
## Captured output
| File | What it shows |
|---|---|
| [`kafka-ordering.txt`](docs/output/kafka-ordering.txt) | Per-key order kept, global order not — and why A, B and C were the wrong keys to demonstrate it |
| [`kafka-replay.txt`](docs/output/kafka-replay.txt) | The same 12 records read three times |
| [`kafka-consumer-scaling.txt`](docs/output/kafka-consumer-scaling.txt) | 5 consumers, 3 partitions, 2 idle |
| [`rabbit-ordering.txt`](docs/output/rabbit-ordering.txt) | Message 1 finishing tenth |
| [`rabbit-replay.txt`](docs/output/rabbit-replay.txt) | 12 messages, then 0, then an empty queue |
| [`rabbit-consumer-scaling.txt`](docs/output/rabbit-consumer-scaling.txt) | 40/0/0/0/0 without `basicQos`, 8 each with it |
| [`pulsar-ordering.txt`](docs/output/pulsar-ordering.txt) | `Shared` spreading a key, `Key_Shared` pinning it |
| [`pulsar-replay.txt`](docs/output/pulsar-replay.txt) | `seek(earliest)` and a new subscription |
| [`pulsar-consumer-scaling.txt`](docs/output/pulsar-consumer-scaling.txt) | The 1000-message receiver queue defeating fan-out |
| [`footprint.txt`](docs/output/footprint.txt) | Startup, memory, ports and config surface for all three |
| [`tests.txt`](docs/output/tests.txt) | The nine tests behind all of the above |
## What is not measured here
Throughput. A messages-per-second number from one 2-core container says nothing about any of
these brokers, and publishing one would be worse than publishing nothing. The measurements above
are structural: they hold on any hardware, because they are properties of the delivery models
rather than of the machine.

View File

@@ -0,0 +1,49 @@
[README](../README.md) · next: [Ordering](02-ordering.md)
# 1. Three storage models, and everything that follows from them
Almost every difference between these brokers follows from one sentence about where a message
lives.
- **Kafka** is an append-only log, partitioned. The broker keeps every record for the retention
period and remembers nothing about individual consumers except an offset. Reading does not
remove anything.
- **RabbitMQ** is a router with queues. The broker owns each message until a consumer
acknowledges it, and then deletes it. It can route, expire, and dead-letter on its own.
- **Pulsar** is a log too, but the cursor and the storage are separate services: brokers are
stateless and BookKeeper holds the data. Acknowledgement moves a cursor, and a message is
deleted once every subscription has passed it — unless a retention policy says otherwise.
Read those three sentences again before reading any comparison table, including the one in
[chapter 7](07-the-decision-table.md). They predict most of it.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| unit of parallelism | partition | queue | subscription |
| set when | topic is created | queue is declared | consumer subscribes |
| consuming | moves an offset | deletes the message | moves a cursor |
| a second reader | new consumer group | another queue, bound in advance | new subscription |
The last row is the one people underestimate. In Kafka and Pulsar you can add a reader that sees
history you have already processed. In RabbitMQ you cannot add one after the fact at all — the
messages are gone — so the decision to have a second consumer has to be made *before* the
messages arrive.
## What was measured, and how
Three questions, asked identically of all three brokers by driving the client libraries directly
rather than through three different Spring abstractions:
| Question | Chapter |
|---|---|
| What ordering survives when you add a second consumer? | [02](02-ordering.md) |
| Can you read the same message twice? | [03](03-replay.md) |
| How far do consumers scale, and what stops them? | [04](04-consumer-scaling.md) |
| What does each one cost to run? | [05](05-operational-footprint.md) |
Versions are the ones a Spring Boot 4.1.1 application gets: kafka-clients 4.2.1, amqp-client
5.30.0, pulsar-client 4.2.4, all read from `spring-boot-dependencies-4.1.1.pom`. The brokers are
Kafka 4.2.1, RabbitMQ 3.10.25 and Pulsar 4.2.4, each started from its own distribution by a script
in `scripts/`.
next: [Ordering](02-ordering.md)

View File

@@ -0,0 +1,66 @@
prev: [Three models](01-three-models.md) · [README](../README.md) · next: [Replay](03-replay.md)
# 2. Ordering
Every broker here is FIFO with one producer and one consumer. The question that matters is what
survives the second consumer, because that is the first thing you add.
## Kafka: ordered within a partition, and a key picks the partition
Twelve records, three keys, three partitions
([`docs/output/kafka-ordering.txt`](output/kafka-ordering.txt)):
```
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
```
Each key's values come back in the order they were produced. The global sequence does not: the
consumer drains one partition's buffer before moving to the next, so the delivered order is
2, 5, 8, 11, 1, 4, 7, 10, 3, 6, 9, 12.
That is the whole Kafka ordering guarantee, and it is usually enough, because "in order" nearly
always means "in order per customer / per account / per device" rather than globally.
**A detail worth stealing.** The keys in that transcript are D, A and F, not A, B and C. With
three partitions, murmur2 sends A, B *and* C all to partition 1. Picking three obvious keys to
demonstrate partitioning would have produced a transcript in which global order was accidentally
preserved. Before you conclude that your keys spread evenly, compute
`Utils.toPositive(Utils.murmur2(key)) % partitions` for the ones you actually use.
## RabbitMQ: FIFO per queue, and no key at all
One consumer sees the queue in order. Two consumers do not
([`docs/output/rabbit-ordering.txt`](output/rabbit-ordering.txt)):
```
one queue, one consumer : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
```
Message 1 went to the slower consumer and finished tenth. There is no key and no partition, so
there is nothing to pin related messages to one consumer. Ordering across related messages in
RabbitMQ means one queue and one consumer — which means no horizontal scaling of that queue —
or the consistent-hash exchange, which is a plugin and effectively reintroduces partitions by
hand.
## Pulsar: the subscription type decides, per subscriber
Two consumers on a `Shared` subscription and two on `Key_Shared`, same topic, same messages
([`docs/output/pulsar-ordering.txt`](output/pulsar-ordering.txt)):
```
Shared A -> [consumer-1, consumer-2] B -> [consumer-1, consumer-2] C -> [consumer-1, consumer-2]
Key_Shared A -> [consumer-2] B -> [consumer-1] C -> [consumer-2]
```
Under `Shared`, every key was handled by both consumers, so two messages with the same key can be
in flight at once and no ordering claim survives. Under `Key_Shared`, each key went to exactly one
consumer — Kafka's guarantee, but chosen by the subscriber rather than fixed by a partition count
someone picked when the topic was created, and recomputed as consumers join and leave.
That last property is the strongest single argument for Pulsar: two teams reading the same topic
can make different ordering-versus-throughput trades without negotiating.
next: [Replay](03-replay.md)

View File

@@ -0,0 +1,59 @@
prev: [Ordering](02-ordering.md) · [README](../README.md) · next: [Consumer scaling](04-consumer-scaling.md)
# 3. Replay
"Can I read that again?" is asked after a bad deploy, and by then it is too late to change
brokers.
## Kafka
```
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
```
([`docs/output/kafka-replay.txt`](output/kafka-replay.txt)) Consuming moves an offset and removes
nothing. A new group, a reset offset and a `seekToBeginning` all read the same records, for as
long as retention keeps them.
## RabbitMQ
```
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
```
([`docs/output/rabbit-replay.txt`](output/rabbit-replay.txt)) Acknowledging deletes. There is no
offset to rewind and no second reader that can see what the first consumed. Reading a message
twice has to be arranged in advance — a second queue bound to the same exchange, or a copy written
somewhere else — and it cannot be arranged afterwards.
This is not a defect. A router that deletes what it has delivered is much cheaper to operate than
a log, and most work queues genuinely do not need history. It only becomes a defect at the moment
you need history and do not have it.
RabbitMQ streams (3.9+) are a separate, log-shaped feature that does support replay. They are a
different thing living in the same broker, with their own client protocol and their own semantics
— worth knowing about, and not what you get from `queueDeclare`.
## Pulsar
```
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
```
([`docs/output/pulsar-replay.txt`](output/pulsar-replay.txt)) `seek(MessageId)` and
`seek(timestamp)` rewind a live subscription; a new subscription starting from `Earliest` reads
everything still stored.
The difference from Kafka is the **default**, and it is the one that surprises people: Kafka keeps
a record for the retention period regardless of who read it, while Pulsar deletes a message once
every subscription has acknowledged it, unless a retention policy on the namespace says otherwise.
A Pulsar namespace with the default retention and one well-behaved subscription keeps nothing —
so the replay you are counting on requires a policy you have to set.
next: [Consumer scaling](04-consumer-scaling.md)

View File

@@ -0,0 +1,80 @@
prev: [Replay](03-replay.md) · [README](../README.md) · next: [Operational footprint](05-operational-footprint.md)
# 4. Consumer scaling
The question is what happens when you add the fifth consumer.
## Kafka: partitions are a hard ceiling
Five consumers in one group on a three-partition topic
([`docs/output/kafka-consumer-scaling.txt`](output/kafka-consumer-scaling.txt)):
```
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
```
A partition belongs to at most one consumer in a group, so partition count is the ceiling. Two of
the five processes are running, connected, healthy, and doing nothing at all — and they will keep
reporting healthy forever.
Raising the partition count later is possible and is not free: it changes which partition a key
maps to, so the per-key ordering guarantee is broken across the change for every key that moves.
In practice the partition count is a capacity decision made at design time, on incomplete
information, that you then live with.
## RabbitMQ: no ceiling, but prefetch decides whether it is real
Five consumers on one queue, forty messages
([`docs/output/rabbit-consumer-scaling.txt`](output/rabbit-consumer-scaling.txt)):
```
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 consumer-2..5 -> 0
basicQos(1):
consumer-1..5 -> 8 each
```
There is no structural ceiling — but with the AMQP default the broker pushes as many messages as
a consumer will accept, so the first consumer to connect can be handed the entire backlog while
four idle processes wait. `basicQos` is not a tuning knob to postpone; it is what makes the
fan-out exist.
Spring AMQP sets it for you: `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` is 250.
Better than unlimited, and still enough to concentrate any backlog smaller than 250 messages on
one consumer.
## Pulsar: no ceiling either, and the same trap under a different name
Five consumers, one *non-partitioned* topic, forty messages
([`docs/output/pulsar-consumer-scaling.txt`](output/pulsar-consumer-scaling.txt)):
```
receiverQueueSize default (1000):
one or two consumers take everything; the rest receive nothing
receiverQueueSize(1):
consumer-1..5 -> 8 each
```
The topic has no partitions and five consumers still share the work, which in Kafka would require
at least five partitions decided in advance. But the default receiver queue is a thousand
messages, so the first consumers to connect pull the whole backlog into their own buffers first.
Same failure as RabbitMQ's unlimited prefetch, four times the default.
## The shape of the answer
- Kafka gives you ordering per key and pays for it with a fixed parallelism ceiling.
- RabbitMQ gives you unlimited competing consumers and pays for it with no ordering across them.
- Pulsar lets each subscription choose, which is genuinely the best of both — at the operational
cost measured in [the next chapter](05-operational-footprint.md).
And all three brokers have a client-side buffer that will quietly defeat your fan-out if you leave
it at the default and your backlog is small. That is the one finding here that applies to
whichever you pick.
next: [Operational footprint](05-operational-footprint.md)

View File

@@ -0,0 +1,55 @@
prev: [Consumer scaling](04-consumer-scaling.md) · [README](../README.md) · next: [What Spring adds](06-what-spring-adds.md)
# 5. Operational footprint
"Ops burden" cannot be benchmarked. The number of moving parts, the memory floor and the size of
the configuration surface can be, and they are what people are actually asking about.
Each broker started from its shipped distribution with default configuration and a 700 MB heap
cap, on one 2-core / 3.8 GB Linux box, Temurin JDK 21
([`docs/output/footprint.txt`](output/footprint.txt)):
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 standalone |
|---|---|---|---|
| launch to first accepted connection | 10.0 s | 15.1 s | 20.8 s |
| resident memory at idle | 333 MB | 115 MB | 610 MB |
| server processes | 1 | 1 (+ `epmd`) | 1 |
| listening ports | 9092, 9093 | 5672, 4369, 25672 | 6650, 8080, 2181 |
| unpacked distribution | 135 MB | 26 MB | 344 MB |
| settings in the shipped default config | 24 | **0** | 357 |
Read those numbers as orders of magnitude, not as a benchmark: one sample, one small machine.
Three things they say clearly.
**RabbitMQ ships no configuration file at all.** The boot log records `Config file(s): (none)`;
`rabbitmq.conf.example` is entirely commented out. Everything works out of the box, and the
smallest resident footprint here is the one that is not a JVM. Against that, `epmd` on 4369 and
the inter-node port on 25672 are the Erlang distribution, which is also how clustering works and
how clustering goes wrong.
**Pulsar standalone is three systems in one process.** Port 2181 is ZooKeeper and the on-disk
data is BookKeeper's; the standalone distribution hides that behind one command. A real
deployment does not: you operate ZooKeeper (or, from Pulsar 3.x, an alternative metadata store),
BookKeeper bookies and Pulsar brokers as three tiers with three scaling stories. The 357-setting
`standalone.conf` is the honest signal here — that is the configuration surface, and running it
seriously means learning most of it.
**Kafka since KRaft is genuinely simpler than it was.** One process, two ports, 24 settings, no
ZooKeeper. The old "Kafka means also running ZooKeeper" objection is a version behind; anyone
comparing on that basis is comparing to 2022.
## What the numbers do not include
The footprint above is one node at idle. The thing that actually determines ops burden is what
happens at three in the morning:
- **Kafka**: rebalances, consumer lag as the primary signal, partition-count decisions you cannot
cleanly reverse, and a broad ecosystem of tools that assume Kafka.
- **RabbitMQ**: the management plugin is genuinely good, queue depth is a direct and obvious
signal, and the hard problem is network partitions in a cluster — which the Erlang distribution
makes fast to detect and awkward to resolve.
- **Pulsar**: the fewest people on your team will have run it. That is not a technical property
and it is usually the deciding one.
next: [What Spring adds](06-what-spring-adds.md)

View File

@@ -0,0 +1,52 @@
prev: [Operational footprint](05-operational-footprint.md) · [README](../README.md) · next: [The decision table](07-the-decision-table.md)
# 6. What Spring adds, and what Boot 4 no longer gives you for free
The measurements in this module drive the client libraries directly, so that what is being
compared is the brokers rather than three sets of Spring defaults. In an application you would use
the Spring integrations, and they are not equivalent to one another.
| | Kafka | RabbitMQ | Pulsar |
|---|---|---|---|
| project | Spring for Apache Kafka 4.1.1 | Spring AMQP 4.1.1 | Spring for Apache Pulsar 2.0.7 |
| starter | `spring-boot-starter-kafka` | `spring-boot-starter-amqp` | `spring-boot-starter-pulsar` |
| listener | `@KafkaListener` | `@RabbitListener` | `@PulsarListener` |
| template | `KafkaTemplate` | `RabbitTemplate` | `PulsarTemplate` |
| retry / DLQ | `DefaultErrorHandler`, `@RetryableTopic` | dead-letter exchange, container error handler | `DeadLetterPolicy` on the listener |
All three are Boot-managed at 4.1.1, so you do not pin their versions.
## The Boot 4 trap that applies to all three
**In Spring Boot 4, depending on a messaging library directly rather than through its Boot starter
means you have no auto-configuration.** The auto-configuration classes moved out of
`spring-boot-autoconfigure` into per-technology modules that only the starters bring:
| depending on | what you lose | how it presents |
|---|---|---|
| `org.springframework.kafka:spring-kafka` | `spring-boot-kafka` | `No qualifying bean of type KafkaTemplate<...>` — the context starts fine |
| `org.springframework.amqp:spring-rabbit` | `spring-boot-amqp` | no `RabbitTemplate`, no `RabbitAdmin` |
| `org.springframework.pulsar:spring-pulsar` | `spring-boot-pulsar` | no `PulsarTemplate` |
Every Boot 3 tutorial gets this wrong now, and the symptom is a missing bean rather than anything
that names the cause. Use the starters.
## The Jackson fork, in both Kafka and RabbitMQ
Boot 4 moved to Jackson 3 (`tools.jackson`), and both messaging projects ship converters for
both generations. **A `2` in the class name means the previous Jackson** — which is the opposite
of the convention you would guess:
- Spring Kafka: `JsonSerializer`/`JsonDeserializer` are Jackson 2; `JacksonJsonSerializer`,
`JacksonJsonDeserializer` and `JacksonJsonSerde` are Jackson 3.
- Spring AMQP: `Jackson2JsonMessageConverter` is Jackson 2; `JacksonJsonMessageConverter` is
Jackson 3.
Picking the wrong one gives you `SerializationException: Can't serialize data`, whose cause is
`Java 8 date/time type java.time.Instant not supported by default` — a message about date types
for what is really a wrong-library problem.
The [kafka-basics](../kafka-basics/README.md) and [rabbitmq](../rabbitmq/README.md) modules in
this repository cover both stacks in detail.
next: [The decision table](07-the-decision-table.md)

View File

@@ -0,0 +1,59 @@
prev: [What Spring adds](06-what-spring-adds.md) · [README](../README.md)
# 7. The decision table
Every row below is either a measurement in `docs/output/` or a statement about operating the
thing. Nothing here is a vendor claim.
| | Kafka 4.2.1 | RabbitMQ 3.10.25 | Pulsar 4.2.4 |
|---|---|---|---|
| ordering with one consumer | FIFO per partition | FIFO per queue | FIFO |
| ordering with many consumers | per key, always | **none** | per key with `Key_Shared`, none with `Shared` |
| who chooses that | topic design | plugin, or one consumer | each subscription, independently |
| replay after the fact | yes, within retention | **no** | yes, if retention is configured |
| second independent reader | new consumer group, any time | must be arranged in advance | new subscription, any time |
| consumer parallelism ceiling | partition count | none | none |
| changing that ceiling | repartition; breaks key→partition | nothing to change | nothing to change |
| the client-side trap | none by default | unlimited prefetch (Spring: 250) | `receiverQueueSize` 1000 |
| memory at idle | 333 MB | **115 MB** | 610 MB |
| processes to operate | 1 (KRaft) | 1 + `epmd` | broker + BookKeeper + metadata store |
| shipped default settings | 24 | **0** | 357 |
| Spring project maturity | very high | very high | good, much smaller community |
## Choosing
**Choose RabbitMQ** when the work is a queue of tasks: each message is an instruction, order
between instructions does not matter, and once it is done it is done. It is the smallest thing
that works, it has the best out-of-the-box operability of the three, and queue depth is a metric
anyone can interpret. Choose it in the knowledge that you are giving up replay permanently —
adding it later is a broker migration, not a configuration change.
**Choose Kafka** when the messages are events other people will want to read: when more than one
consumer will exist, when someone will need to reprocess history after a bug, or when ordering per
entity is part of the contract. Pay for it with a partition count decided too early, consumers
that scale only as far as that number, and an operational model where lag is the thing you watch.
KRaft has removed the old ZooKeeper objection.
**Choose Pulsar** when you genuinely need what neither of the others gives you: per-subscription
choice of ordering versus fan-out, consumer counts not bounded by a number chosen at topic
creation, or multi-tenancy with real isolation. It is the most capable design here. It is also
three systems, 357 settings, five times RabbitMQ's memory floor, and the one your team has least
experience with — and that last point decides more incidents than the first three prevent.
**If the honest answer is "we do not know yet"**, that argues for Kafka, on grounds that have
nothing to do with the technology: you can hire for it, your monitoring vendor supports it, and
the failure modes are documented by thousands of people who hit them first. That is a real
engineering argument and it is fine to make it out loud.
## The question that dissolves the choice
A surprising number of these decisions are made for a system that has one producer, one consumer
and fewer than a hundred messages a second. At that volume all three brokers work, none of the
measurements above will ever be reached, and the decision is entirely about what your team can
operate at three in the morning.
The measurements matter when you can name which row you are relying on. If you cannot, you are
picking an operational burden, not a broker — and the cheapest one to operate is the one in the
`115 MB` cell.
[README](../README.md)

View File

@@ -0,0 +1,40 @@
== Operational footprint, measured on one 2-core / 3.8 GB Linux box ==
Each broker started from its shipped distribution with default configuration and a
700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as
orders of magnitude, not as a benchmark.
broker : kafka
time from launch to first
accepted connection : 10035 ms
resident memory at idle : 333 MB
server processes : 1
listening ports : 9092 (broker), 9093 (controller)
unpacked distribution size : 135 MB
settings in shipped default
configuration file : 24 (server.properties)
broker : rabbit
time from launch to first
accepted connection : 15053 ms
resident memory at idle : 115 MB
server processes : 1
listening ports : 5672 (AMQP), 4369 (epmd), 25672 (inter-node)
unpacked distribution size : 26 MB
settings in shipped default
configuration file : 0 (rabbitmq.conf.example)
epmd processes : 1
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
entirely commented out and the boot log records "Config file(s): (none)".
Everything in that file is documentation, not a default.
broker : pulsar
time from launch to first
accepted connection : 20801 ms
resident memory at idle : 610 MB
server processes : 1
listening ports : 6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)
unpacked distribution size : 344 MB
settings in shipped default
configuration file : 357 (standalone.conf)

View File

@@ -0,0 +1,16 @@
== Kafka: partitions are the ceiling on consumer parallelism ==
topic 'orders-scaling', 3 partitions, 5 consumers in one group
consumer-1 -> partitions [0]
consumer-2 -> partitions [1]
consumer-3 -> partitions [2]
consumer-4 -> no partitions (idle)
consumer-5 -> no partitions (idle)
consumers with no partitions: 2
A partition is assigned to at most one consumer in a group, so the number
of partitions is a hard ceiling on consumer parallelism. Adding consumers
beyond it adds idle processes, not throughput.

View File

@@ -0,0 +1,30 @@
== Kafka: ordering is per partition ==
produced 12 records, keys D/A/F round-robin, values 1..12 in order
(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C
all hash to partition 1 with three partitions, which is worth knowing
before you decide your keys are well distributed.)
consumed in this order:
F=2@p2
F=5@p2
F=8@p2
F=11@p2
A=1@p1
A=4@p1
A=7@p1
A=10@p1
D=3@p0
D=6@p0
D=9@p0
D=12@p0
per key:
F -> partition [2] values [2, 5, 8, 11]
A -> partition [1] values [1, 4, 7, 10]
D -> partition [0] values [3, 6, 9, 12]
Order is preserved within each key because a key hashes to one partition.
Across keys it is not: the values above are not 1..12 in order, because a
consumer drains one partition's buffer before the next.

View File

@@ -0,0 +1,10 @@
== Kafka: the log is the storage ==
group replay-group-1, first read : 12 records
group replay-group-2, brand new group : 12 records
group replay-group-1, after seekToBeginning: 12 records
Consuming does not remove anything. A consumer group is a cursor over a log
that the broker keeps until retention expires, so a new group, a reset
offset or a seek all read the same records again.

View File

@@ -0,0 +1,30 @@
== Pulsar: no partition ceiling, but the receiver queue decides who gets the work ==
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
receiverQueueSize left at the default (1000):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
receiverQueueSize(1):
consumer-1 -> 8 messages
consumer-2 -> 8 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 8 messages
consumers that received nothing : 0
The topic has no partitions and five consumers can still share the work --
in Kafka the same shape needs at least five partitions, chosen when the
topic was created. But the default receiver queue is 1000 messages, so the
first consumers to connect pull the whole 40-message backlog into their own
buffers before the rest ask for anything, and the subscription looks
broken. Which consumers win is a race and varies between runs -- one
consumer taking all forty, or two taking twenty each -- but the consumers
that lose it see nothing at all. This is the same trap as RabbitMQ's
unbounded prefetch, with a different name and a much larger default.

View File

@@ -0,0 +1,24 @@
== Pulsar: the subscription type decides ==
12 messages, keys A/B/C, values 1..12 in order, two consumers per
subscription, receiverQueueSize 1 so the first consumer cannot take the
whole backlog.
Shared subscription, which consumers saw each key:
A -> [consumer-1, consumer-2]
B -> [consumer-1, consumer-2]
C -> [consumer-1, consumer-2]
Key_Shared subscription, which consumers saw each key:
A -> [consumer-1]
B -> [consumer-1]
C -> [consumer-1]
A Shared subscription round-robins individual messages, so messages with
the same key end up on different consumers and can be processed at the
same time: there is no per-key order left to speak of. Key_Shared hashes
the key to one consumer, which is Kafka's guarantee -- except that the
assignment belongs to the subscription and is recomputed as consumers come
and go, rather than being fixed by a partition count chosen when the topic
was created.

View File

@@ -0,0 +1,14 @@
== Pulsar: acknowledged, but still there ==
subscription replay-sub, first read : 12 messages
subscription replay-sub, after seek(earliest): 12 messages
subscription replay-sub-2, brand new : 12 messages
Acknowledgement moves a cursor; the message itself lives in the managed
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
which Kafka can also do by resetting offsets. What differs is the default:
Pulsar deletes a message once every subscription has acknowledged it,
unless a retention policy on the namespace says otherwise, whereas Kafka
keeps it for the retention period regardless of who read it. A Pulsar
topic with no retention policy and no subscriptions keeps nothing.

View File

@@ -0,0 +1,34 @@
== RabbitMQ: consumers scale, and prefetch decides whether they actually do ==
one queue, 40 messages, 5 consumers, each taking 10 ms per message
no basicQos at all (unlimited prefetch, the AMQP default):
consumer-1 -> 40 messages
consumer-2 -> 0 messages
consumer-3 -> 0 messages
consumer-4 -> 0 messages
consumer-5 -> 0 messages
consumers that received nothing : 4
basicQos(1):
consumer-1 -> 8 messages
consumer-2 -> 9 messages
consumer-3 -> 8 messages
consumer-4 -> 8 messages
consumer-5 -> 7 messages
consumers that received nothing : 0
Every consumer on a queue competes for the same messages, so adding
consumers adds throughput and there is no structural ceiling of the kind
Kafka's partition count imposes. But with the AMQP default the broker
pushes as many messages as a consumer will take, so whichever consumer
connects first can be handed the entire backlog while the others sit idle.
basicQos is not a tuning knob you get to postpone; it is what makes the
fan-out real. Spring AMQP sets it for you --
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
better than unlimited and still large enough to concentrate a small
backlog on one consumer.
The price of all this is the ordering measurement: there is no key, so
nothing constrains related messages to one consumer.

View File

@@ -0,0 +1,18 @@
== RabbitMQ: FIFO per queue, per consumer ==
one queue, one consumer, 12 messages
completion order : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
completion order : [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 12]
out-of-order steps: 1
A queue is FIFO and a single consumer sees it that way. The moment a second
consumer is added the broker hands the next message to whichever consumer
is free, so the order in which work finishes is no longer the order in
which it was published. There is no key: RabbitMQ has no notion of a
partition to which related messages could be pinned. Ordering across
related messages means one queue and one consumer, and therefore no
horizontal scaling for that queue -- or a consistent-hash exchange, which
is a plugin.

View File

@@ -0,0 +1,12 @@
== RabbitMQ: there is nothing to replay ==
first drain of the queue : 12 messages
second drain of the queue : 0 messages
queue depth afterwards : 0
Acknowledging a message deletes it. The broker is a router with buffers,
not a log: there is no offset to rewind and no second reader that can see
what the first one consumed. Reading the same message twice means
arranging it in advance -- a second queue bound to the same exchange, or a
copy written somewhere else -- and it cannot be arranged after the fact.

View File

@@ -0,0 +1,12 @@
[INFO] Running com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.88 s -- in com.ankurm.brokers.KafkaComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.732 s -- in com.ankurm.brokers.RabbitComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
[INFO] Running com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.00 s -- in com.ankurm.brokers.PulsarComparisonTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

66
broker-comparison/pom.xml Normal file
View File

@@ -0,0 +1,66 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<!-- Every client version below is managed by spring-boot-dependencies 4.1.1, so the three
brokers are compared at the versions a Boot 4.1 application would actually get:
kafka-clients 4.2.1, amqp-client 5.30.0, pulsar-client 4.2.4. Read from
spring-boot-dependencies-4.1.1.pom, not from release announcements. -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>broker-comparison</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<!-- The three client libraries, driven directly. The comparison is between the brokers'
delivery models, and going through three different Spring abstractions would put three
different sets of defaults between the measurement and the thing being measured.
What each Boot starter adds on top is covered in docs/06-what-spring-adds.md. -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- Provides the in-process KRaft broker used by the Kafka measurements: a real Kafka
broker, no Docker. -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<!-- Each broker is started by its own script and the matching tagged tests are run on
their own; see scripts/run-all.sh, which passes -Dgroups=kafka|rabbit|pulsar. Running
"mvn test" with no group runs all three and needs all three brokers up. -->
</project>

View File

@@ -0,0 +1,73 @@
#!/usr/bin/env bash
# Measures the operational footprint of one broker: how long it takes to accept a connection,
# how much memory it holds at idle, how many processes and listening ports it opens, and how
# large its shipped default configuration is.
#
# These are proxies, not a benchmark. "Ops burden" is not directly measurable, but the number of
# moving parts, the memory floor and the size of the configuration surface are, and they are what
# people are actually asking about when they ask which broker is heavier to run.
#
# scripts/footprint.sh kafka|rabbit|pulsar
set -euo pipefail
BROKER="${1:?kafka|rabbit|pulsar}"
extra=""
port_open() { (echo > "/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
rss_mb() { ps -eo rss,cmd | grep -F "$1" | grep -v grep | awk '{s+=$1} END {printf "%.0f", s/1024}'; }
procs() { ps -eo cmd | grep -F "$1" | grep -cv grep; }
settings() { grep -cE '^[a-zA-Z]' "$1" 2>/dev/null || echo 0; }
start=$(date +%s%3N)
case "$BROKER" in
kafka)
: "${KAFKA_HOME:?set KAFKA_HOME}"
# Wipe whatever log.dirs points at, so the format below is not refused with
# "Invalid cluster.id ... Expected X, but read Y" from a previous run.
KAFKA_LOG_DIRS=$(grep -E '^log.dirs=' "$KAFKA_HOME/config/server.properties" | cut -d= -f2)
rm -rf ${KAFKA_LOG_DIRS//,/ }
"$KAFKA_HOME/bin/kafka-storage.sh" format --standalone -t "$("$KAFKA_HOME/bin/kafka-storage.sh" random-uuid)" \
-c "$KAFKA_HOME/config/server.properties" --ignore-formatted >/dev/null
start=$(date +%s%3N) # after formatting: kafka-storage.sh is two more JVM starts and is a
# one-off, not part of what a restart costs
setsid nohup "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/server.properties" \
> /tmp/kafka-start.log 2>&1 < /dev/null &
for _ in $(seq 1 90); do port_open 9092 && break; sleep 1; done
ready=$(( $(date +%s%3N) - start ))
marker="kafka.Kafka"; ports="9092 (broker), 9093 (controller)"; conf="$KAFKA_HOME/config/server.properties"
dist="$KAFKA_HOME"
;;
rabbit)
: "${RABBITMQ_HOME:?set RABBITMQ_HOME}"; : "${ERL_ROOT:?set ERL_ROOT}"
scripts/rabbit-broker.sh >/dev/null
ready=$(( $(date +%s%3N) - start ))
# The broker is an Erlang VM, so the process to measure is beam.smp, not anything named
# "rabbitmq". epmd is a second, tiny process and is counted separately below.
marker="beam.smp"; ports="5672 (AMQP), 4369 (epmd), 25672 (inter-node)"
conf="$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf.example"; dist="$RABBITMQ_HOME"
extra="epmd processes : $(procs epmd)
NOTE: RabbitMQ ships no configuration file at all. rabbitmq.conf.example is
entirely commented out and the boot log records \"Config file(s): (none)\".
Everything in that file is documentation, not a default."
;;
pulsar)
: "${PULSAR_HOME:?set PULSAR_HOME}"
scripts/pulsar-broker.sh >/dev/null
ready=$(( $(date +%s%3N) - start ))
marker="PulsarStandaloneStarter"; ports="6650 (binary), 8080 (admin/REST), 2181 (ZooKeeper)"
conf="$PULSAR_HOME/conf/standalone.conf"; dist="$PULSAR_HOME"
;;
*) echo "unknown broker $BROKER" >&2; exit 2 ;;
esac
cat <<EOF
broker : $BROKER
time from launch to first
accepted connection : ${ready} ms
resident memory at idle : $(rss_mb "$marker") MB
server processes : $(procs "$marker")
listening ports : $ports
unpacked distribution size : $(du -sm "$dist" | cut -f1) MB
settings in shipped default
configuration file : $(settings "$conf") ($(basename "$conf"))
${extra:-}
EOF

View File

@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Start a real Apache Pulsar broker with no Docker and no root.
#
# The standalone distribution bundles ZooKeeper, BookKeeper and the broker in one process, which
# is why a single tarball is enough. -nss skips the stream storage (BookKeeper's table service,
# needed only by stateful Pulsar Functions) and -nfw skips the functions worker; both save
# several hundred megabytes of heap and a few seconds of startup.
#
# PULSAR_HOME must point at an unpacked apache-pulsar-<version>-bin directory. Pulsar 4.2
# supports Java 17 and 21; the transcripts here were produced on Temurin 21.
set -eu
: "${PULSAR_HOME:?set PULSAR_HOME to an unpacked apache-pulsar-*-bin directory}"
export PULSAR_MEM="${PULSAR_MEM:--Xms384m -Xmx700m -XX:MaxDirectMemorySize=384m}"
cd "$PULSAR_HOME"
setsid nohup bin/pulsar standalone -nss -nfw > /tmp/pulsar-start.log 2>&1 < /dev/null &
for _ in $(seq 1 90); do
if [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/admin/v2/brokers/version)" = "200" ]; then
echo "broker ready on 6650 (admin 8080), version $(curl -s http://127.0.0.1:8080/admin/v2/brokers/version)"
exit 0
fi
sleep 2
done
echo "broker did not start; see /tmp/pulsar-start.log" >&2
exit 1

View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Start a real RabbitMQ broker with no Docker and no root.
#
# RabbitMQ is an Erlang application, so an Erlang runtime and epmd on the path are the whole
# dependency. Point ERL_ROOT at an Erlang installation and RABBITMQ_HOME at an unpacked
# rabbitmq-server-generic-unix tarball. Ubuntu 22.04 ships Erlang 24, whose newest compatible
# broker is 3.10.25; 3.11 and later need Erlang 25.
#
# The one non-obvious step is starting epmd yourself: rabbitmq-server's own attempt to start it
# fails in a container without a resolvable hostname, and the failure surfaces as a forty-line
# Erlang crash dump ending in {'EXIT',nodistribution}.
set -eu
: "${ERL_ROOT:?set ERL_ROOT to an Erlang installation directory}"
: "${RABBITMQ_HOME:?set RABBITMQ_HOME to an unpacked rabbitmq-server-generic-unix directory}"
export PATH="$ERL_ROOT/bin:$ERL_ROOT/erts-"*/bin":$PATH"
export RABBITMQ_MNESIA_BASE="${RABBITMQ_MNESIA_BASE:-/tmp/rmq/data}"
export RABBITMQ_LOG_BASE="${RABBITMQ_LOG_BASE:-/tmp/rmq/log}"
export RABBITMQ_NODENAME="${RABBITMQ_NODENAME:-rabbit@localhost}"
export HOME="${HOME:-/tmp/rmq}"
mkdir -p "$RABBITMQ_MNESIA_BASE" "$RABBITMQ_LOG_BASE"
epmd -daemon 2>/dev/null || true
sleep 1
setsid nohup "$RABBITMQ_HOME/sbin/rabbitmq-server" > /tmp/rmq/boot.log 2>&1 < /dev/null &
for _ in $(seq 1 60); do
if (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; then echo "broker ready on 5672"; exit 0; fi
sleep 1
done
echo "broker did not start; see /tmp/rmq/boot.log" >&2
exit 1

View File

@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Regenerates every file under docs/output/. Each broker is started, measured and left running
# only for its own group of tests, because three brokers do not fit comfortably in memory on a
# small machine at the same time.
#
# Required environment:
# KAFKA_HOME unpacked kafka_2.13-4.2.1
# ERL_ROOT an Erlang 24 installation (RabbitMQ 3.10.x)
# RABBITMQ_HOME unpacked rabbitmq-server-generic-unix-3.10.25
# PULSAR_HOME unpacked apache-pulsar-4.2.4-bin
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p docs/output
# Kafka needs no external broker for the measurements: spring-kafka-test starts a real KRaft
# broker in-process. KAFKA_HOME is only used by the footprint measurement.
mvn -B -Dgroups=kafka test 2>&1 | grep -E 'Running |Tests run:|BUILD ' > docs/output/tests.txt
scripts/rabbit-broker.sh
mvn -B -Dgroups=rabbit test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
scripts/pulsar-broker.sh
mvn -B -Dgroups=pulsar test 2>&1 | grep -E 'Running |Tests run:|BUILD ' >> docs/output/tests.txt
{
echo "== Operational footprint, measured on one 2-core / 3.8 GB Linux box =="
echo
echo "Each broker started from its shipped distribution with default configuration and a"
echo "700 MB heap cap, on Temurin JDK 21. Times are one sample on a small box: treat them as"
echo "orders of magnitude, not as a benchmark."
echo
scripts/footprint.sh kafka; echo
scripts/footprint.sh rabbit; echo
scripts/footprint.sh pulsar
} > docs/output/footprint.txt
echo "regenerated:"; ls -1 docs/output

View File

@@ -0,0 +1,15 @@
package com.ankurm.brokers;
/**
* There is deliberately no Spring application here. The measurements in src/test drive the three
* client libraries directly so that the thing being compared is the broker's delivery model and
* not three different sets of Spring defaults.
*
* <p>What each Spring Boot starter adds on top &mdash; and what it silently does not add, which
* in Boot 4 is more than people expect &mdash; is in docs/06-what-spring-adds.md.
*/
public final class BrokerComparisonApplication {
private BrokerComparisonApplication() {
}
}

View File

@@ -0,0 +1,23 @@
package com.ankurm.brokers;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
/** Writes a transcript under docs/output/. Nothing in the article is typed by hand. */
public final class Capture {
private Capture() {
}
public static void write(String fileName, String heading, String body) {
Path dir = Path.of(System.getProperty("user.dir"), "docs", "output");
try {
Files.createDirectories(dir);
Files.writeString(dir.resolve(fileName), "== " + heading + " ==\n\n" + body + "\n");
}
catch (IOException ex) {
throw new IllegalStateException("could not write " + fileName, ex);
}
}
}

View File

@@ -0,0 +1,232 @@
package com.ankurm.brokers;
import java.time.Duration;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.kafka.test.EmbeddedKafkaKraftBroker;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Kafka measured against a real broker: an in-process KRaft broker from spring-kafka-test, one
* node, one topic, three partitions.
*
* <p>The three questions are the same for all three brokers: what ordering survives, what can be
* replayed, and how far consumers scale.
*/
@Tag("kafka")
class KafkaComparisonTest {
private static final String ORDERING = "orders-ordering";
private static final String REPLAY = "orders-replay";
private static final String SCALING = "orders-scaling";
private static EmbeddedKafkaKraftBroker broker;
@BeforeAll
static void startBroker() {
broker = new EmbeddedKafkaKraftBroker(1, 3, ORDERING, REPLAY, SCALING);
broker.afterPropertiesSet();
}
@AfterAll
static void stopBroker() {
broker.destroy();
}
// ---------------------------------------------------------------- ordering
@Test
void orderingIsPerPartitionOnly() {
produce(ORDERING, 12);
List<ConsumerRecord<String, String>> received = consumeAll(ORDERING, "ordering-group", 12);
Map<String, List<String>> perKey = new LinkedHashMap<>();
Map<String, Set<Integer>> partitionsPerKey = new LinkedHashMap<>();
List<String> globalOrder = new ArrayList<>();
for (ConsumerRecord<String, String> record : received) {
perKey.computeIfAbsent(record.key(), (k) -> new ArrayList<>()).add(record.value());
partitionsPerKey.computeIfAbsent(record.key(), (k) -> new java.util.TreeSet<>())
.add(record.partition());
globalOrder.add(record.key() + "=" + record.value() + "@p" + record.partition());
}
// Per key, the sequence numbers must come back in the order they were produced. Compared
// as numbers: as strings, "10" sorts before "2" and the assertion is meaningless.
perKey.forEach((key, values) -> assertThat(values.stream().map(Integer::parseInt).toList())
.isSorted());
// Each key went to exactly one partition, which is why that holds...
partitionsPerKey.forEach((key, partitions) -> assertThat(partitions).hasSize(1));
// ...and the three keys are spread over all three partitions, so the transcript below is
// actually testing something.
assertThat(partitionsPerKey.values().stream().flatMap(Set::stream).distinct().count())
.isEqualTo(3);
List<Integer> asDelivered = globalOrder.stream()
.map((entry) -> Integer.parseInt(entry.split("=")[1].split("@")[0])).toList();
assertThat(asDelivered).isNotEqualTo(
java.util.stream.IntStream.rangeClosed(1, 12).boxed().toList());
StringBuilder body = new StringBuilder();
body.append("produced 12 records, keys D/A/F round-robin, values 1..12 in order\n")
.append("(keys chosen so that murmur2 spreads them: D->0, A->1, F->2. A, B and C\n")
.append(" all hash to partition 1 with three partitions, which is worth knowing\n")
.append(" before you decide your keys are well distributed.)\n\n");
body.append("consumed in this order:\n ").append(String.join("\n ", globalOrder))
.append("\n\nper key:\n");
perKey.forEach((key, values) -> body.append(" ").append(key).append(" -> partition ")
.append(partitionsPerKey.get(key)).append(" values ").append(values).append('\n'));
body.append("\nOrder is preserved within each key because a key hashes to one partition.\n")
.append("Across keys it is not: the values above are not 1..12 in order, because a\n")
.append("consumer drains one partition's buffer before the next.\n");
Capture.write("kafka-ordering.txt", "Kafka: ordering is per partition", body.toString());
}
// ---------------------------------------------------------------- replay
@Test
void anythingStillWithinRetentionCanBeReadAgain() {
produce(REPLAY, 12);
int first = consumeAll(REPLAY, "replay-group-1", 12).size();
int again = consumeAll(REPLAY, "replay-group-2", 12).size();
List<ConsumerRecord<String, String>> third;
try (KafkaConsumer<String, String> consumer = consumer("replay-group-1")) {
consumer.subscribe(List.of(REPLAY));
consumer.poll(Duration.ofSeconds(2));
consumer.seekToBeginning(consumer.assignment());
third = drain(consumer, 12);
}
assertThat(first).isEqualTo(12);
assertThat(again).isEqualTo(12);
assertThat(third).hasSize(12);
Capture.write("kafka-replay.txt", "Kafka: the log is the storage",
"""
group replay-group-1, first read : %d records
group replay-group-2, brand new group : %d records
group replay-group-1, after seekToBeginning: %d records
Consuming does not remove anything. A consumer group is a cursor over a log
that the broker keeps until retention expires, so a new group, a reset
offset or a seek all read the same records again.
""".formatted(first, again, third.size()));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void consumerParallelismIsCappedByPartitionCount() throws Exception {
produce(SCALING, 12);
List<KafkaConsumer<String, String>> consumers = new ArrayList<>();
Map<String, Set<Integer>> assignment = new LinkedHashMap<>();
try {
for (int i = 1; i <= 5; i++) {
KafkaConsumer<String, String> consumer = consumer("scaling-group");
consumer.subscribe(List.of(SCALING));
consumers.add(consumer);
}
// Poll each consumer until the group has settled and every member knows its share.
for (int round = 0; round < 8; round++) {
for (KafkaConsumer<String, String> consumer : consumers) {
consumer.poll(Duration.ofMillis(500));
}
}
for (int i = 0; i < consumers.size(); i++) {
Set<Integer> partitions = new java.util.TreeSet<>();
for (TopicPartition tp : consumers.get(i).assignment()) {
partitions.add(tp.partition());
}
assignment.put("consumer-" + (i + 1), partitions);
}
}
finally {
consumers.forEach(KafkaConsumer::close);
}
long idle = assignment.values().stream().filter(Set::isEmpty).count();
assertThat(assignment).hasSize(5);
assertThat(idle).isEqualTo(2);
StringBuilder body = new StringBuilder("topic 'orders-scaling', 3 partitions, 5 consumers in one group\n\n");
assignment.forEach((name, partitions) -> body.append(" ").append(name).append(" -> ")
.append(partitions.isEmpty() ? "no partitions (idle)" : "partitions " + partitions)
.append('\n'));
body.append("\nconsumers with no partitions: ").append(idle).append('\n')
.append("\nA partition is assigned to at most one consumer in a group, so the number\n")
.append("of partitions is a hard ceiling on consumer parallelism. Adding consumers\n")
.append("beyond it adds idle processes, not throughput.\n");
Capture.write("kafka-consumer-scaling.txt",
"Kafka: partitions are the ceiling on consumer parallelism", body.toString());
}
// ---------------------------------------------------------------- helpers
private void produce(String topic, int count) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// D, A and F, not A, B and C: with three partitions murmur2 maps A, B and C all to
// partition 1, so the obvious choice of keys would have produced a transcript in which
// global order happened to be preserved and proved nothing. D->0, A->1, F->2.
String[] keys = { "D", "A", "F" };
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
for (int i = 1; i <= count; i++) {
producer.send(new ProducerRecord<>(topic, keys[i % keys.length], String.valueOf(i)));
}
producer.flush();
}
}
private KafkaConsumer<String, String> consumer(String group) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ConsumerConfig.GROUP_ID_CONFIG, group);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
return new KafkaConsumer<>(props);
}
private List<ConsumerRecord<String, String>> consumeAll(String topic, String group, int expected) {
try (KafkaConsumer<String, String> consumer = consumer(group)) {
consumer.subscribe(List.of(topic));
return drain(consumer, expected);
}
}
private List<ConsumerRecord<String, String>> drain(KafkaConsumer<String, String> consumer, int expected) {
List<ConsumerRecord<String, String>> received = new ArrayList<>();
for (int i = 0; i < 20 && received.size() < expected; i++) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
records.forEach(received::add);
}
return received;
}
}

View File

@@ -0,0 +1,298 @@
package com.ankurm.brokers;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
import org.apache.pulsar.client.api.SubscriptionType;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Pulsar measured against a real standalone broker on localhost:6650, started by
* scripts/pulsar-broker.sh.
*
* <p>Pulsar is the interesting third option because it does not force the trade the other two
* force. The subscription type is chosen per subscriber rather than being a property of the
* topic, so ordering and consumer fan-out are separate decisions.
*/
@Tag("pulsar")
class PulsarComparisonTest {
private static PulsarClient client;
@BeforeAll
static void connect() throws Exception {
client = PulsarClient.builder().serviceUrl("pulsar://127.0.0.1:6650").build();
}
@AfterAll
static void close() throws Exception {
client.close();
}
// ---------------------------------------------------------------- ordering
@Test
void sharedSpreadsAKeyAndKeySharedPinsIt() throws Exception {
String topic = "persistent://public/default/ordering-" + System.nanoTime();
produce(topic, 12);
Map<String, Set<String>> shared = consumersPerKey(topic, "sub-shared", SubscriptionType.Shared, 12);
Map<String, Set<String>> keyShared = consumersPerKey(topic, "sub-keyshared",
SubscriptionType.Key_Shared, 12);
// Key_Shared: every key was handled by exactly one consumer, so per-key order is safe.
keyShared.forEach((key, handlers) -> assertThat(handlers).hasSize(1));
// Shared: at least one key was handled by both, so two messages with that key were in
// flight on two consumers and nothing orders them.
assertThat(shared.values().stream().anyMatch((handlers) -> handlers.size() > 1)).isTrue();
Capture.write("pulsar-ordering.txt", "Pulsar: the subscription type decides",
"""
12 messages, keys A/B/C, values 1..12 in order, two consumers per
subscription, receiverQueueSize 1 so the first consumer cannot take the
whole backlog.
Shared subscription, which consumers saw each key:
%s
Key_Shared subscription, which consumers saw each key:
%s
A Shared subscription round-robins individual messages, so messages with
the same key end up on different consumers and can be processed at the
same time: there is no per-key order left to speak of. Key_Shared hashes
the key to one consumer, which is Kafka's guarantee -- except that the
assignment belongs to the subscription and is recomputed as consumers come
and go, rather than being fixed by a partition count chosen when the topic
was created.
""".formatted(render3(shared), render3(keyShared)));
}
// ---------------------------------------------------------------- replay
@Test
void aSubscriptionCanBeRewound() throws Exception {
String topic = "persistent://public/default/replay-" + System.nanoTime();
produce(topic, 12);
int first;
int afterSeek;
try (Consumer<String> consumer = consumer(topic, "replay-sub", SubscriptionType.Exclusive)) {
first = drain(consumer, 12).size();
consumer.seek(MessageId.earliest);
afterSeek = drain(consumer, 12).size();
}
int newSubscription;
try (Consumer<String> consumer = consumer(topic, "replay-sub-2", SubscriptionType.Exclusive)) {
newSubscription = drain(consumer, 12).size();
}
assertThat(first).isEqualTo(12);
assertThat(afterSeek).isEqualTo(12);
assertThat(newSubscription).isEqualTo(12);
Capture.write("pulsar-replay.txt", "Pulsar: acknowledged, but still there",
"""
subscription replay-sub, first read : %d messages
subscription replay-sub, after seek(earliest): %d messages
subscription replay-sub-2, brand new : %d messages
Acknowledgement moves a cursor; the message itself lives in the managed
ledger. seek(MessageId) and seek(timestamp) rewind a live subscription,
which Kafka can also do by resetting offsets. What differs is the default:
Pulsar deletes a message once every subscription has acknowledged it,
unless a retention policy on the namespace says otherwise, whereas Kafka
keeps it for the retention period regardless of who read it. A Pulsar
topic with no retention policy and no subscriptions keeps nothing.
""".formatted(first, afterSeek, newSubscription));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void sharedSubscriptionsHaveNoPartitionCeilingButHaveAReceiverQueue() throws Exception {
String topicA = "persistent://public/default/scaling-default-" + System.nanoTime();
produce(topicA, 40);
Map<String, Integer> withDefaultQueue = drainAcross(topicA, "scaling-sub", 5, 40, 0);
String topicB = "persistent://public/default/scaling-q1-" + System.nanoTime();
produce(topicB, 40);
Map<String, Integer> withQueueOfOne = drainAcross(topicB, "scaling-sub", 5, 40, 1);
long idleDefault = withDefaultQueue.values().stream().filter((n) -> n == 0).count();
long idleTuned = withQueueOfOne.values().stream().filter((n) -> n == 0).count();
assertThat(idleTuned).isZero();
assertThat(idleDefault).isGreaterThan(0);
Capture.write("pulsar-consumer-scaling.txt",
"Pulsar: no partition ceiling, but the receiver queue decides who gets the work",
"""
one non-partitioned topic, 40 messages, 5 consumers, Shared subscription
receiverQueueSize left at the default (1000):
%s consumers that received nothing : %d
receiverQueueSize(1):
%s consumers that received nothing : %d
The topic has no partitions and five consumers can still share the work --
in Kafka the same shape needs at least five partitions, chosen when the
topic was created. But the default receiver queue is 1000 messages, so the
first consumers to connect pull the whole 40-message backlog into their own
buffers before the rest ask for anything, and the subscription looks
broken. Which consumers win is a race and varies between runs -- one
consumer taking all forty, or two taking twenty each -- but the consumers
that lose it see nothing at all. This is the same trap as RabbitMQ's
unbounded prefetch, with a different name and a much larger default.
""".formatted(render2(withDefaultQueue), idleDefault,
render2(withQueueOfOne), idleTuned));
}
// ---------------------------------------------------------------- helpers
private void produce(String topic, int count) throws Exception {
String[] keys = { "A", "B", "C" };
try (Producer<String> producer = client.newProducer(Schema.STRING).topic(topic).create()) {
for (int i = 1; i <= count; i++) {
producer.newMessage().key(keys[i % keys.length]).value(String.valueOf(i)).send();
}
}
}
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type)
throws Exception {
return consumer(topic, subscription, type, 0);
}
/**
* @param receiverQueueSize 0 leaves Pulsar's default of 1000 in place; anything else sets it.
* The default matters more than it looks: it is how many messages a single consumer will pull
* into its own buffer before another consumer on the same subscription gets a chance.
*/
private Consumer<String> consumer(String topic, String subscription, SubscriptionType type,
int receiverQueueSize) throws Exception {
var builder = client.newConsumer(Schema.STRING)
.topic(topic)
.subscriptionName(subscription)
.subscriptionType(type)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
if (receiverQueueSize > 0) {
builder = builder.receiverQueueSize(receiverQueueSize);
}
return builder.subscribe();
}
private List<String> drain(Consumer<String> consumer, int expected) throws Exception {
List<String> received = new ArrayList<>();
while (received.size() < expected) {
Message<String> message = consumer.receive(2, TimeUnit.SECONDS);
if (message == null) {
break;
}
received.add(message.getValue());
consumer.acknowledge(message);
}
return received;
}
/** Which consumers saw each key, with a receiver queue of one so nothing hoards. */
private Map<String, Set<String>> consumersPerKey(String topic, String subscription,
SubscriptionType type, int expected) throws Exception {
Map<String, Set<String>> perKey = new LinkedHashMap<>();
List<Consumer<String>> consumers = new ArrayList<>();
try {
consumers.add(consumer(topic, subscription, type, 1));
consumers.add(consumer(topic, subscription, type, 1));
int received = 0;
while (received < expected) {
boolean any = false;
for (int i = 0; i < consumers.size(); i++) {
Message<String> message = consumers.get(i).receive(500, TimeUnit.MILLISECONDS);
if (message != null) {
perKey.computeIfAbsent(message.getKey(), (k) -> new java.util.TreeSet<>())
.add("consumer-" + (i + 1));
consumers.get(i).acknowledge(message);
received++;
any = true;
}
}
if (!any) {
break;
}
}
}
finally {
for (Consumer<String> consumer : consumers) {
consumer.close();
}
}
return perKey;
}
/** Drains a Shared subscription across {@code n} consumers and reports the distribution. */
private Map<String, Integer> drainAcross(String topic, String subscription, int n, int expected,
int receiverQueueSize) throws Exception {
Map<String, Integer> counts = new LinkedHashMap<>();
List<Consumer<String>> consumers = new ArrayList<>();
try {
for (int i = 1; i <= n; i++) {
consumers.add(consumer(topic, subscription, SubscriptionType.Shared, receiverQueueSize));
counts.put("consumer-" + i, 0);
}
int received = 0;
while (received < expected) {
boolean any = false;
for (int i = 0; i < consumers.size(); i++) {
Message<String> message = consumers.get(i).receive(300, TimeUnit.MILLISECONDS);
if (message != null) {
consumers.get(i).acknowledge(message);
counts.merge("consumer-" + (i + 1), 1, Integer::sum);
received++;
any = true;
}
}
if (!any) {
break;
}
}
}
finally {
for (Consumer<String> consumer : consumers) {
consumer.close();
}
}
return counts;
}
private static String render3(Map<String, Set<String>> perKey) {
StringBuilder out = new StringBuilder();
perKey.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append('\n'));
return out.toString();
}
private static String render2(Map<String, Integer> counts) {
StringBuilder out = new StringBuilder();
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append(" messages\n"));
return out.toString();
}
}

View File

@@ -0,0 +1,286 @@
package com.ankurm.brokers;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DefaultConsumer;
import com.rabbitmq.client.Envelope;
import com.rabbitmq.client.GetResponse;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/**
* RabbitMQ measured against a real broker on localhost:5672, started by
* scripts/rabbit-broker.sh.
*
* <p>The same three questions as the Kafka measurements, against a broker that owns the message
* until it is acknowledged rather than a log that keeps it.
*/
@Tag("rabbit")
class RabbitComparisonTest {
private static Connection connection;
@BeforeAll
static void connect() throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setPort(5672);
connection = factory.newConnection();
}
@AfterAll
static void close() throws Exception {
connection.close();
}
// ---------------------------------------------------------------- ordering
@Test
void oneConsumerIsFifoAndTwoConsumersAreNot() throws Exception {
String single = declare("order-single");
publish(single, 12);
List<String> fifo = getAll(single, 12);
String shared = declare("order-shared");
publish(shared, 12);
List<String> completionOrder = consumeWithTwoConsumers(shared, 12);
assertThat(fifo).containsExactly("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12");
int inversions = 0;
for (int i = 1; i < completionOrder.size(); i++) {
if (Integer.parseInt(completionOrder.get(i)) < Integer.parseInt(completionOrder.get(i - 1))) {
inversions++;
}
}
Capture.write("rabbit-ordering.txt", "RabbitMQ: FIFO per queue, per consumer",
"""
one queue, one consumer, 12 messages
completion order : %s
one queue, two consumers, prefetch 1, consumer-1 slower than consumer-2
completion order : %s
out-of-order steps: %d
A queue is FIFO and a single consumer sees it that way. The moment a second
consumer is added the broker hands the next message to whichever consumer
is free, so the order in which work finishes is no longer the order in
which it was published. There is no key: RabbitMQ has no notion of a
partition to which related messages could be pinned. Ordering across
related messages means one queue and one consumer, and therefore no
horizontal scaling for that queue -- or a consistent-hash exchange, which
is a plugin.
""".formatted(fifo, completionOrder, inversions));
}
// ---------------------------------------------------------------- replay
@Test
void anAcknowledgedMessageIsGone() throws Exception {
String queue = declare("replay-probe");
publish(queue, 12);
List<String> first = getAll(queue, 12);
List<String> second = getAll(queue, 12);
int depth = depth(queue);
assertThat(first).hasSize(12);
assertThat(second).isEmpty();
assertThat(depth).isZero();
Capture.write("rabbit-replay.txt", "RabbitMQ: there is nothing to replay",
"""
first drain of the queue : %d messages
second drain of the queue : %d messages
queue depth afterwards : %d
Acknowledging a message deletes it. The broker is a router with buffers,
not a log: there is no offset to rewind and no second reader that can see
what the first one consumed. Reading the same message twice means
arranging it in advance -- a second queue bound to the same exchange, or a
copy written somewhere else -- and it cannot be arranged after the fact.
""".formatted(first.size(), second.size(), depth));
}
// ---------------------------------------------------------------- consumer scaling
@Test
void consumersScaleWithoutAPartitionCeilingButPrefetchDecidesWho() throws Exception {
String unlimited = declare("scaling-unlimited");
publish(unlimited, 40);
Map<String, Integer> withDefaultPrefetch = drainAcross(unlimited, 5, 40, 0);
String throttled = declare("scaling-prefetch-1");
publish(throttled, 40);
Map<String, Integer> withPrefetchOne = drainAcross(throttled, 5, 40, 1);
long idleDefault = withDefaultPrefetch.values().stream().filter((n) -> n == 0).count();
long idleTuned = withPrefetchOne.values().stream().filter((n) -> n == 0).count();
assertThat(idleTuned).isZero();
Capture.write("rabbit-consumer-scaling.txt",
"RabbitMQ: consumers scale, and prefetch decides whether they actually do",
"""
one queue, 40 messages, 5 consumers, each taking 10 ms per message
no basicQos at all (unlimited prefetch, the AMQP default):
%s consumers that received nothing : %d
basicQos(1):
%s consumers that received nothing : %d
Every consumer on a queue competes for the same messages, so adding
consumers adds throughput and there is no structural ceiling of the kind
Kafka's partition count imposes. But with the AMQP default the broker
pushes as many messages as a consumer will take, so whichever consumer
connects first can be handed the entire backlog while the others sit idle.
basicQos is not a tuning knob you get to postpone; it is what makes the
fan-out real. Spring AMQP sets it for you --
AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT is 250 -- which is
better than unlimited and still large enough to concentrate a small
backlog on one consumer.
The price of all this is the ordering measurement: there is no key, so
nothing constrains related messages to one consumer.
""".formatted(render(withDefaultPrefetch), idleDefault,
render(withPrefetchOne), idleTuned));
}
// ---------------------------------------------------------------- helpers
private String declare(String name) throws Exception {
try (Channel channel = connection.createChannel()) {
channel.queueDelete(name);
channel.queueDeclare(name, false, false, true, null);
}
return name;
}
private void publish(String queue, int count) throws Exception {
try (Channel channel = connection.createChannel()) {
for (int i = 1; i <= count; i++) {
channel.basicPublish("", queue, null, String.valueOf(i).getBytes(StandardCharsets.UTF_8));
}
}
}
private List<String> getAll(String queue, int max) throws Exception {
List<String> received = new ArrayList<>();
try (Channel channel = connection.createChannel()) {
for (int i = 0; i < max; i++) {
GetResponse response = channel.basicGet(queue, true);
if (response == null) {
break;
}
received.add(new String(response.getBody(), StandardCharsets.UTF_8));
}
}
return received;
}
private int depth(String queue) throws Exception {
try (Channel channel = connection.createChannel()) {
return channel.queueDeclarePassive(queue).getMessageCount();
}
}
private List<String> consumeWithTwoConsumers(String queue, int count) throws Exception {
List<String> completion = new CopyOnWriteArrayList<>();
CountDownLatch done = new CountDownLatch(count);
List<Channel> channels = new ArrayList<>();
for (int i = 1; i <= 2; i++) {
long delay = (i == 1) ? 60 : 5;
Channel channel = connection.createChannel();
channel.basicQos(1);
channels.add(channel);
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
@Override
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
byte[] body) throws IOException {
try {
Thread.sleep(delay);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
completion.add(new String(body, StandardCharsets.UTF_8));
getChannel().basicAck(envelope.getDeliveryTag(), false);
done.countDown();
}
});
}
done.await(30, TimeUnit.SECONDS);
for (Channel channel : channels) {
channel.close();
}
return List.copyOf(completion);
}
/** Drains a queue across {@code n} competing consumers. prefetch 0 means "do not set it". */
private Map<String, Integer> drainAcross(String queue, int n, int expected, int prefetch)
throws Exception {
Map<String, AtomicInteger> perConsumer = new ConcurrentHashMap<>();
CountDownLatch done = new CountDownLatch(expected);
List<Channel> channels = new ArrayList<>();
for (int i = 1; i <= n; i++) {
String name = "consumer-" + i;
Channel channel = connection.createChannel();
if (prefetch > 0) {
channel.basicQos(prefetch);
}
channels.add(channel);
perConsumer.put(name, new AtomicInteger());
channel.basicConsume(queue, false, new DefaultConsumer(channel) {
@Override
public void handleDelivery(String tag, Envelope envelope, AMQP.BasicProperties props,
byte[] body) throws IOException {
perConsumer.get(name).incrementAndGet();
try {
Thread.sleep(10);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
getChannel().basicAck(envelope.getDeliveryTag(), false);
done.countDown();
}
});
}
done.await(30, TimeUnit.SECONDS);
for (Channel channel : channels) {
channel.close();
}
Map<String, Integer> counts = new LinkedHashMap<>();
perConsumer.forEach((name, count) -> counts.put(name, count.get()));
return counts;
}
private static String render(Map<String, Integer> counts) {
StringBuilder out = new StringBuilder();
counts.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach((entry) ->
out.append(" ").append(entry.getKey()).append(" -> ").append(entry.getValue())
.append(" messages\n"));
return out.toString();
}
}

View File

@@ -0,0 +1,59 @@
# `kafka-error-handling` — retries, DLT and poison pills
Companion project for
[**Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills**](https://ankurm.com/spring-kafka-4-1-error-handling-dlt-retry-topics/)
on ankurm.com.
Six tests against a **real Kafka broker** started in-process in KRaft mode. No Docker, no local
install. `./scripts/run-all.sh` regenerates everything under [`docs/output/`](docs/output/).
## Versions
| | Version |
|---|---|
| JDK | 25 (Temurin 25.0.4.1+1) |
| Spring Boot | 4.1.1 |
| Spring Kafka | 4.1.1 |
| kafka-clients | 4.2.1 (Boot-managed) |
## Profiles
| Profile | What it wires |
|---|---|
| `dlt` | `DefaultErrorHandler` + `DeadLetterPublishingRecoverer`, `FixedBackOff(1000, 2)`, `PermanentFailure` classified non-retryable |
| `dltbytes` | the same, with a `byte[]`-aware template map so poison pills keep their original bytes |
| `retrytopic` | `@RetryableTopic` non-blocking retries with a `@DltHandler` |
| `defaults` | the stock `DefaultErrorHandler`, for reading its behaviour |
## Documentation
1. [Two kinds of failure, and why they need different machinery](docs/01-two-kinds-of-failure.md)
2. [What the default actually does](docs/02-default-error-handler.md)
3. [Poison pills](docs/03-poison-pills.md)
4. [The dead-letter topic](docs/04-the-dlt.md)
5. [Non-blocking retries with `@RetryableTopic`](docs/05-retryable-topic.md)
## Captured output
| File | Shows |
|---|---|
| [`default-backoff.txt`](docs/output/default-backoff.txt) | ten deliveries, zero delay |
| [`retry-and-dlt.txt`](docs/output/retry-and-dlt.txt) | measured back-off and the DLT headers |
| [`poison-pill.txt`](docs/output/poison-pill.txt) | base64 payload, and the byte-aware fix |
| [`retry-topics.txt`](docs/output/retry-topics.txt) | the non-blocking delivery trace |
| [`tests.txt`](docs/output/tests.txt) | 6 tests |
## Six things this module exists to prove
1. **The default is ten deliveries, zero milliseconds apart, then the record is dropped.** Not
"retry with backoff", and not "dead-letter".
2. **The DLT suffix is `-dlt`, not `.DLT`.** Get it wrong and the recoverer logs a WARN and the
record is lost — your safety net silently drops it.
3. **`kafka_dlt-exception-fqcn` is always `ListenerExecutionFailedException`** for listener
failures. The useful header is `-exception-cause-fqcn`.
4. **A poison pill reaches the DLT base64-encoded**, because the recoverer reuses the JSON
producer. A per-type template map fixes it; the module shows both transcripts.
5. **`@RetryableTopic` names retry topics by delay** — `-retry-500`, `-retry-1000` — so changing
the multiplier renames them.
6. **`@Backoff` from spring-retry no longer exists here.** Spring Kafka 4 ships its own
`@BackOff`, and the attribute is `backOff`.

View File

@@ -0,0 +1,53 @@
[Module README](../README.md) &middot; [DefaultErrorHandler &rarr;](02-default-error-handler.md)
# 1. Two kinds of failure, and why they need different machinery
Kafka delivery is at-least-once. The container commits offsets after your listener returns
([the basics article](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/)
covers why), so if the listener throws, the offset does not move and the record comes back.
Everything in this module is about what happens next.
There are two failures, and conflating them is why a bad record can take a partition down for
hours.
## Failure inside the listener
Your code threw. The record deserialized fine; the work failed. The container catches it, hands
it to a `CommonErrorHandler`, and that decides whether to retry, how long to wait, and what to do
when the attempts run out.
This splits again, and the split matters more than any back-off setting:
| | example | retrying it |
|---|---|---|
| **transient** | timeout, 503, deadlock, connection reset | may succeed |
| **permanent** | validation failure, missing entity, malformed field | will fail identically |
Retrying a permanent failure ten times buys nothing and costs ten times the latency plus nine
misleading log lines. Spring Kafka lets you say so:
```java
handler.addNotRetryableExceptions(PermanentFailure.class);
```
That single line is worth more than tuning the back-off, and it is the one most people skip.
## Failure before the listener
The bytes on the topic are not what the deserializer expects. Someone changed a schema, or
published with a different serializer, or your `__TypeId__` header names a class you do not
trust.
This one is nastier, because it happens **inside `poll()`**, before any listener exists to throw
from. There is no error handler in the path. The consumer cannot advance past the offset,
retries the same record on the next poll, fails again, and does that forever — at whatever rate
the poll loop runs. The partition is stopped and the only symptom is a growing lag with a
consumer that looks healthy.
That is a **poison pill**, and the cure is a different mechanism from the one above:
`ErrorHandlingDeserializer`, covered in [chapter 3](03-poison-pills.md).
Keeping these two apart is the whole point of this module. One needs a retry policy; the other
needs a wrapper around the deserializer. Neither fixes the other.
[DefaultErrorHandler &rarr;](02-default-error-handler.md)

View File

@@ -0,0 +1,71 @@
[&larr; Two kinds of failure](01-two-kinds-of-failure.md) &middot; [Module README](../README.md) &middot; [Poison pills &rarr;](03-poison-pills.md)
# 2. What the default actually does
If you configure nothing, the container factory installs a `DefaultErrorHandler` with
`SeekUtils.DEFAULT_BACK_OFF` and a recoverer that logs. Run the back-off and read it off
([`docs/output/default-backoff.txt`](output/default-backoff.txt)):
```
=== DefaultErrorHandler default back-off ===
interval 0 ms
max attempts 9 retries
SeekUtils.DEFAULT_MAX_FAILURES = 10
retry intervals [0, 0, 0, 0, 0, 0, 0, 0, 0]
total deliveries 10
```
**Ten deliveries, zero milliseconds apart, and then the record is dropped.**
Both halves of that surprise people. It is not "retry with backoff" — it is ten immediate
attempts as fast as the consumer thread can run them, which against a downstream that is
overloaded is ten times the load at the worst moment. And "then dropped" means exactly that: the
default recoverer logs the failure and the offset moves on. There is no dead-letter topic unless
you make one.
## Giving it a back-off and a destination
```java
@Bean
DefaultErrorHandler errorHandler(KafkaOperations<String, Object> template) {
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
handler.addNotRetryableExceptions(PermanentFailure.class);
return handler;
}
```
`FixedBackOff(1000L, 2)` is one delivery plus two retries. Measured
([`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt)):
```
=== transient failure ===
deliveries 3
gap between 1&2 1007 ms (FixedBackOff interval 1000)
```
and the classified permanent failure gets exactly one delivery before going to the DLT.
**`ExponentialBackOffWithMaxRetries` is usually the better choice** than `FixedBackOff` for a
transient downstream, because a fixed interval synchronises every consumer in the group into
retrying at the same instant.
## The cost of blocking retries
`DefaultErrorHandler` retries **on the consumer thread**. For the whole back-off, that partition
processes nothing else. `FixedBackOff(1000L, 2)` is three seconds of a stalled partition per
failing record — fine. A one-minute exponential back-off over five attempts is five minutes, and
if failures are correlated you have a stalled consumer group, not a retry policy.
Two consequences worth planning for:
- **`max.poll.interval.ms` is your ceiling.** Default five minutes. Block longer than that
between polls and the broker evicts the consumer from the group, triggering a rebalance —
which usually makes things worse. A back-off schedule that can exceed it is a bug.
- **Ordering is preserved**, which is the one thing blocking retries give you that
[retry topics](05-retryable-topic.md) do not.
That trade — ordering versus throughput under failure — is the real decision, and it is covered
in [chapter 5](05-retryable-topic.md).
[Poison pills &rarr;](03-poison-pills.md)

View File

@@ -0,0 +1,95 @@
[&larr; DefaultErrorHandler](02-default-error-handler.md) &middot; [Module README](../README.md) &middot; [The DLT &rarr;](04-the-dlt.md)
# 3. Poison pills
A record whose bytes cannot be deserialized fails inside `poll()`, before any listener exists.
No error handler is in the path. The offset cannot advance, so the next poll fetches the same
record and fails identically. Forever.
The consumer is up, the group is stable, no exception reaches your code, and lag grows. It is
one of the few Kafka failures with no good symptom.
## `ErrorHandlingDeserializer`
```yaml
spring:
kafka:
consumer:
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
properties:
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
```
It wraps the real deserializer, catches the failure, and returns a **null value with the
exception in a header**. The record then flows normally into the container, the listener is
skipped, and the error handler gets a record it can recover — which is to say, the poison pill
becomes an ordinary failure.
Use `spring.deserializer.key.delegate.class` for keys. A malformed key is rarer and just as
fatal.
## What arrives on the DLT
From [`docs/output/poison-pill.txt`](output/poison-pill.txt):
```
kafka_dlt-exception-fqcn org.springframework.kafka.support.serializer.DeserializationException
kafka_dlt-exception-cause-fqcn org.springframework.kafka.support.serializer.DeserializationException
kafka_dlt-exception-message failed to deserialize
kafka_dlt-original-topic payments
kafka_dlt-original-partition 0x00000000
kafka_dlt-original-offset 0x0000000000000000
kafka_dlt-original-consumer-group payments
__TypeId__ [B
DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
```
The listener was never invoked — asserted in the test — and the record is off the partition,
which is the whole win.
Two details in that block are worth stopping on.
**`original-partition` and `original-offset` are binary.** They are big-endian `int` and `long`,
not text. Printing them as a string gives you mojibake, which is why the transcript renders them
as hex. A DLT tool that treats every header as UTF-8 will show garbage for exactly the three
fields you need in order to find the original record.
**The payload is base64.** `"eyB0aGlzIGlzIG5vdCBqc29u"` decodes to `{ this is not json`. The
recoverer publishes with the **application's** producer, whose value serializer is
`JacksonJsonSerializer`; the failed value is a `byte[]`; Jackson writes a `byte[]` as a base64
JSON string. So the DLT does not hold what arrived — it holds base64 of it, wrapped in quotes.
Replaying that topic naively republishes a quoted base64 string, which fails to deserialize, and
now you have a poison pill in your poison-pill queue.
## The fix
Give the recoverer a template per value type:
```java
Map<Class<?>, KafkaOperations<?, ?>> templates = new LinkedHashMap<>();
templates.put(byte[].class, byteTemplate); // ByteArraySerializer
templates.put(Object.class, jsonTemplate);
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);
```
```
=== byte-aware DLT ===
DLT payload -> { this is not json
```
Byte for byte what was published. Replay is now a copy from one topic to another.
Three things had to be right to get there, and each failed first — they are commented in
[`ErrorHandlerConfiguration`](../src/main/java/com/ankurm/kafkaerrors/ErrorHandlerConfiguration.java):
1. **`KafkaAutoConfiguration`'s template is `@ConditionalOnMissingBean(KafkaTemplate.class)`.**
Declaring `byteTemplate` removed the auto-configured `KafkaTemplate` from the context
entirely. Once you declare one template, you own all of them.
2. With two templates present, `KafkaOperations<String, Object>` stops resolving, and
`@Qualifier` alone does **not** rescue it — the generic check runs first. Use
`KafkaOperations<?, ?>`, which is what the recoverer's constructor wants anyway.
3. The map is `Map<Class<?>, KafkaOperations<?, ?>>`, matched by value type, with
`Object.class` as the fallback.
[The DLT &rarr;](04-the-dlt.md)

View File

@@ -0,0 +1,74 @@
[&larr; Poison pills](03-poison-pills.md) &middot; [Module README](../README.md) &middot; [Retry topics &rarr;](05-retryable-topic.md)
# 4. The dead-letter topic
## The suffix is `-dlt`, not `.DLT`
```java
public static final String RetryTopicConstants.DEFAULT_RETRY_SUFFIX = "-retry";
public static final String RetryTopicConstants.DEFAULT_DLT_SUFFIX = "-dlt";
```
Older Spring Kafka used `.DLT`, and most of the material online still says so. Getting it wrong
is not an exception — it is this, at WARN, once per record:
```
o.s.k.l.DeadLetterPublishingRecoverer : Destination resolver returned non-existent partition
payments-dlt-0, KafkaProducer will determine partition to use for this topic
[Producer] ... {payments-dlt=UNKNOWN_TOPIC_OR_PARTITION}
```
and then, on a cluster with auto-topic-creation disabled, the record is **gone**. Your safety net
dropped it and logged a warning. This module's tests were written against `payments.DLT` first
and failed exactly this way.
Two things follow: pre-create your DLT topics as part of provisioning, and alert on that WARN.
## Same partition by default
`DeadLetterPublishingRecoverer` publishes to the **same partition number** as the original. If
your DLT has fewer partitions than the source topic, records from the high-numbered partitions
have nowhere to go. Either give the DLT the same partition count, or set
```java
recoverer.setPartitionResolver((record, ex) -> null); // let the producer choose
```
## The headers, and the one that will mislead you
From [`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt):
```
kafka_dlt-exception-fqcn org.springframework.kafka.listener.ListenerExecutionFailedException
kafka_dlt-exception-cause-fqcn com.ankurm.kafkaerrors.Failures$TransientFailure
kafka_dlt-original-topic payments
kafka_dlt-original-consumer-group payments
```
**`kafka_dlt-exception-fqcn` is always the wrapper** for a listener failure. Build a DLT triage
dashboard grouped by that header and every failure in the estate lands in one bucket called
`ListenerExecutionFailedException`. The field you want is `kafka_dlt-exception-cause-fqcn`.
(For a deserialization failure there is no wrapper, so the two headers agree. That inconsistency
is worth knowing if you are writing a tool over them.)
`kafka_dlt-original-consumer-group` is the one that saves you when several groups consume the
same topic and share a DLT.
## Replay
A DLT is only useful if you can put records back. The mechanics are a copy:
1. read from `<topic>-dlt` with a **byte-array** deserializer — the payload may be the thing that
could not be deserialized
2. read `kafka_dlt-original-topic` and `kafka_dlt-original-consumer-group` to decide where it
belongs and whether it is yours
3. republish to the original topic, **stripping the `kafka_dlt-*` headers** so a second failure
is not confused with the first
4. do it deliberately, in bounded batches, after the cause is fixed
Automatic replay is almost always wrong: the records are on the DLT precisely because something
was not transient, and a loop that moves them back on a timer is a slow-motion outage. A replay
you run by hand, having read the failure, is the tool worth building.
[Retry topics &rarr;](05-retryable-topic.md)

View File

@@ -0,0 +1,89 @@
[&larr; The DLT](04-the-dlt.md) &middot; [Module README](../README.md)
# 5. Non-blocking retries with `@RetryableTopic`
Blocking retries stall the partition. `@RetryableTopic` republishes the failed record to a
separate topic and lets the main partition carry on.
```java
@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
exclude = Failures.PermanentFailure.class)
@KafkaListener(topics = "invoices", groupId = "invoices")
public void onInvoice(ConsumerRecord<String, Payment> record, ...) { ... }
```
**Two API changes in Spring Kafka 4.x will stop older examples compiling:**
- the attribute is **`backOff`**, not `backoff`
- the annotation is **`org.springframework.kafka.annotation.BackOff`**, not
`org.springframework.retry.annotation.Backoff`. Spring Kafka 4 dropped the spring-retry
dependency and brought its own.
The failure is `package org.springframework.retry.annotation does not exist`, which reads like a
missing dependency and is not.
Also new in 4.1: `sameIntervalTopicReuseStrategy` defaults to `SINGLE_TOPIC` in
`RetryTopicConfigurationBuilder`, aligning it with the annotation's default.
## What it actually does
From [`docs/output/retry-topics.txt`](output/retry-topics.txt) — a failing record and a good one
published back to back on the same partition:
```
=== @RetryableTopic delivery trace ===
+0 ms invoices transient-1
+531 ms invoices-retry-500 transient-1
+550 ms invoices ok-1
+1554 ms invoices-retry-1000 transient-1
+3560 ms invoices-retry-2000 transient-1
DLT: [transient-1 on invoices-dlt]
```
Read the third line. `ok-1` was processed at +550 ms, while `transient-1` was still two retries
from giving up. With a blocking handler it would have waited for the whole schedule.
**Retry topics are named by the delay, not the attempt number.** `invoices-retry-500`,
`invoices-retry-1000`, `invoices-retry-2000` — that is
`TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE`, the default. So provisioning topics ahead of
time means knowing your whole back-off schedule in advance, and **changing the multiplier changes
the topic names**, orphaning whatever is still sitting in the old ones. Deploy that change the
way you would a rename.
## The cost
**Per-key ordering is gone for any record that fails.** That is not a side effect; it is the
mechanism. If `invoice-7` fails and `invoice-7`'s next event succeeds, they are processed out of
order, and no configuration prevents it.
So the decision is not "blocking or non-blocking", it is:
| | blocking (`DefaultErrorHandler`) | non-blocking (`@RetryableTopic`) |
|---|---|---|
| ordering under failure | preserved | lost for the failing key |
| partition throughput under failure | stalled | unaffected |
| topics to provision | 1 + DLT | 1 + one per distinct delay + DLT |
| long back-offs | limited by `max.poll.interval.ms` | unlimited |
If your consumer is idempotent and order-insensitive — most notification, indexing and cache-warm
consumers are — retry topics are strictly better. If it applies state transitions per key,
blocking retries with a short schedule and a fast DLT are usually the safer answer.
Use `exclude` (or `include`) rather than retrying everything: a `PermanentFailure` here skips the
retry topics entirely and goes straight to `invoices-dlt`.
## `@DltHandler`
```java
@DltHandler
public void onDlt(ConsumerRecord<String, Payment> record,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { ... }
```
Without one, the framework still creates and populates the DLT — it just logs and moves on, and
nothing in your application has looked at the record. A `@DltHandler` that increments a counter
and writes a structured log line is the minimum worth having, because a DLT nobody watches is a
queue that grows until someone notices the disk.
[Module README](../README.md)

View File

@@ -0,0 +1,6 @@
=== DefaultErrorHandler default back-off ===
interval 0 ms
max attempts 9 retries
SeekUtils.DEFAULT_MAX_FAILURES = 10
retry intervals [0, 0, 0, 0, 0, 0, 0, 0, 0]
total deliveries 10

View File

@@ -0,0 +1,15 @@
=== poison pill on the DLT ===
kafka_dlt-exception-fqcn org.springframework.kafka.support.serializer.DeserializationException
kafka_dlt-exception-cause-fqcn org.springframework.kafka.support.serializer.DeserializationException
kafka_dlt-exception-message failed to deserialize
kafka_dlt-exception-stacktrace org.springframework.kafka.support.serializer.DeserializationException: failed to deseriali...
kafka_dlt-original-topic payments
kafka_dlt-original-partition 0x00000000
kafka_dlt-original-offset 0x0000000000000000
kafka_dlt-original-timestamp 0x000001a04bd35409
kafka_dlt-original-timestamp-type CreateTime
kafka_dlt-original-consumer-group payments
__TypeId__ [B
DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
=== byte-aware DLT ===
DLT payload -> { this is not json

View File

@@ -0,0 +1,15 @@
=== transient failure ===
deliveries 3
gap between 1&2 1007 ms (FixedBackOff interval 1000)
=== transient failure on the DLT ===
kafka_dlt-exception-fqcn org.springframework.kafka.listener.ListenerExecutionFailedException
kafka_dlt-exception-cause-fqcn com.ankurm.kafkaerrors.Failures$TransientFailure
kafka_dlt-exception-message Listener method 'public void com.ankurm.kafkaerrors.PaymentListener.onPayment(org.apache.k...
kafka_dlt-exception-stacktrace org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'publ...
kafka_dlt-original-topic payments
kafka_dlt-original-partition 0x00000000
kafka_dlt-original-offset 0x0000000000000002
kafka_dlt-original-timestamp 0x000001a04bd35f33
kafka_dlt-original-timestamp-type CreateTime
kafka_dlt-original-consumer-group payments
__TypeId__ com.ankurm.kafkaerrors.Payment

View File

@@ -0,0 +1,8 @@
=== @RetryableTopic delivery trace ===
+0 ms invoices transient-1
+522 ms invoices-retry-500 transient-1
+539 ms invoices ok-1
+1548 ms invoices-retry-1000 transient-1
+3554 ms invoices-retry-2000 transient-1
DLT: [transient-1 on invoices-dlt]
topics touched: [invoices, invoices-retry-1000, invoices-retry-2000, invoices-retry-500]

View File

@@ -0,0 +1,4 @@
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.009 s -- in com.ankurm.kafkaerrors.RetryableTopicTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.809 s -- in com.ankurm.kafkaerrors.DeadLetterTopicTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.ankurm.kafkaerrors.DefaultBackOffTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.935 s -- in com.ankurm.kafkaerrors.ByteAwareDltTest

View File

@@ -0,0 +1,84 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<!-- Inheriting spring-boot-starter-parent so spring-kafka, kafka-clients, Jackson and the
test stack are all Boot-managed. Boot 4.1.1 manages Spring Kafka 4.1.1 and
kafka-clients 4.3.1; do not pin those yourself. -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>kafka-error-handling</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<!-- spring-boot-starter-kafka, NOT a bare org.springframework.kafka:spring-kafka
dependency. In Boot 4 the auto-configuration lives in the spring-boot-kafka module
(package org.springframework.boot.kafka.autoconfigure), which the starter brings and
spring-kafka does not. Depending on spring-kafka alone compiles, starts, and gives you
no KafkaTemplate bean: "No qualifying bean of type KafkaTemplate<...>". Every Boot 3
tutorial gets this wrong now. See docs/01-the-on-ramp.md. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
<!-- JsonSerializer/JsonDeserializer need a Jackson ObjectMapper. Boot 4 moved to
Jackson 3 (tools.jackson), which changes the import in your own code. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jackson</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- The broker used by the committed transcripts: a real Kafka broker in KRaft mode,
started in-process. No Docker required, which is why docs/output/ can be regenerated
anywhere. See docs/07-testing.md for the Testcontainers route. -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-testcontainers</artifactId>
<scope>test</scope>
</dependency>
<!-- NOT org.testcontainers:kafka. Testcontainers 2.x renamed every module with a
"testcontainers-" prefix, and Boot 4.1.1 imports testcontainers-bom 2.0.5, which
manages the new name only. The old coordinate stopped at 1.21.4 and fails the build
with "'dependencies.dependency.version' ... is missing", which does not mention the
rename. See docs/07-testing.md. -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers-kafka</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

View File

@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# Regenerate every file under docs/output/. A real Kafka broker starts in-process in KRaft mode;
# no Docker and no local install.
set -eu
cd "$(dirname "$0")/.."
mvn -B test 2>&1 | tr -d '\000' | tee /tmp/eh-test.log > /dev/null
sed -n '/=== DefaultErrorHandler default back-off ===/,/total deliveries/p' /tmp/eh-test.log > docs/output/default-backoff.txt
sed -n '/=== transient failure ===/,/gap between/p' /tmp/eh-test.log > docs/output/retry-and-dlt.txt
sed -n '/=== transient failure on the DLT ===/,/__TypeId__/p' /tmp/eh-test.log >> docs/output/retry-and-dlt.txt
sed -n '/=== poison pill on the DLT ===/,/DLT payload/p' /tmp/eh-test.log > docs/output/poison-pill.txt
sed -n '/=== byte-aware DLT ===/,/DLT payload/p' /tmp/eh-test.log >> docs/output/poison-pill.txt
sed -n '/=== @RetryableTopic delivery trace ===/,/topics touched/p' /tmp/eh-test.log > docs/output/retry-topics.txt
grep -aE 'Tests run:.*in com\.ankurm' /tmp/eh-test.log | sed 's/^\[INFO\] //' > docs/output/tests.txt
echo "regenerated:"; ls -1 docs/output/

View File

@@ -0,0 +1,114 @@
package com.ankurm.kafkaerrors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.kafka.core.KafkaOperations;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.boot.kafka.autoconfigure.KafkaProperties;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.springframework.util.backoff.FixedBackOff;
import java.util.HashMap;
import java.util.Map;
/**
* The error handler, in three configurations selected by profile.
*
* <p>A {@code CommonErrorHandler} bean replaces the container factory's default one wholesale.
* That default is a {@code DefaultErrorHandler} with {@code SeekUtils.DEFAULT_BACK_OFF} and a
* recoverer that only logs &mdash; so out of the box a record is retried and then <b>dropped</b>,
* which is the behaviour most people are surprised by.
*
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
*/
@Configuration(proxyBeanMethods = false)
public class ErrorHandlerConfiguration {
private static final Logger log = LoggerFactory.getLogger(ErrorHandlerConfiguration.class);
/**
* Publishes the exhausted record to {@code <topic>.DLT}. Without a recoverer bean the record
* is logged and discarded; this is the one line that turns "we lost it" into "it is on a
* topic we can replay".
*/
@Bean
@Profile("dlt")
DefaultErrorHandler dltErrorHandler(KafkaOperations<String, Object> template) {
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
// Three attempts, one second apart, so the timing is visible in a transcript.
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
// Classification matters more than the back-off. A PermanentFailure retried ten times is
// ten times the latency for the same answer, and nine extra log lines that look like an
// outage.
handler.addNotRetryableExceptions(Failures.PermanentFailure.class);
handler.setRetryListeners((record, ex, deliveryAttempt) ->
log.info("retry listener: attempt {} for offset {} ({})", deliveryAttempt,
record.offset(), ex.getClass().getSimpleName()));
return handler;
}
/**
* The same thing, publishing raw bytes correctly.
*
* <p>{@code DeadLetterPublishingRecoverer} takes a map from value type to template. Give it
* a {@code byte[]} template backed by {@code ByteArraySerializer} and a deserialization
* failure lands on the DLT as the exact bytes that arrived, instead of as base64 of them.
* Without this, replaying a poison-pill DLT republishes a quoted base64 string.
*/
// Three things had to be right here, and each one failed first:
//
// 1. KafkaAutoConfiguration's template is @ConditionalOnMissingBean(KafkaTemplate.class),
// so declaring byteTemplate below REMOVES the auto-configured KafkaTemplate from the
// context entirely. Once you declare one, you own all of them - hence jsonTemplate.
// 2. With two templates present, KafkaOperations<String, Object> no longer resolves; the
// generic check runs before the qualifier, so @Qualifier alone does not rescue it.
// 3. The recoverer's constructor takes Map<Class<?>, KafkaOperations<?, ?>>, so wildcards
// are what it wants anyway.
@Bean
@Profile("dltbytes")
DefaultErrorHandler byteAwareDltErrorHandler(
@Qualifier("jsonTemplate") KafkaOperations<?, ?> jsonTemplate,
@Qualifier("byteTemplate") KafkaOperations<?, ?> byteTemplate) {
Map<Class<?>, KafkaOperations<?, ?>> templates = new java.util.LinkedHashMap<>();
templates.put(byte[].class, byteTemplate);
templates.put(Object.class, jsonTemplate);
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 0));
handler.addNotRetryableExceptions(Failures.PermanentFailure.class);
return handler;
}
/**
* Replaces the auto-configured template, which backed off the moment {@code byteTemplate}
* was declared.
*/
@Bean
@Profile("dltbytes")
KafkaTemplate<String, Object> jsonTemplate(KafkaProperties properties) {
return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(properties.buildProducerProperties()));
}
@Bean
@Profile("dltbytes")
KafkaTemplate<String, byte[]> byteTemplate(KafkaProperties properties) {
Map<String, Object> config = new HashMap<>(properties.buildProducerProperties());
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(config));
}
/** The stock handler, declared explicitly so a test can read its behaviour. */
@Bean
@Profile("defaults")
DefaultErrorHandler defaultErrorHandler() {
return new DefaultErrorHandler();
}
}

View File

@@ -0,0 +1,20 @@
package com.ankurm.kafkaerrors;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* What Spring Kafka does when your listener throws, and what it does when the record cannot even
* be turned into an object.
*
* <p>Those are two different failures with two different cures, and conflating them is why a
* poison pill takes a partition down for hours.
*/
@SpringBootApplication
public class ErrorHandlingApplication {
public static void main(String[] args) {
SpringApplication.run(ErrorHandlingApplication.class, args);
}
}

View File

@@ -0,0 +1,33 @@
package com.ankurm.kafkaerrors;
/**
* The two failure classes that Spring Kafka treats completely differently, made explicit so the
* tests can be about the distinction rather than about a generic RuntimeException.
*/
public final class Failures {
/** Worth retrying: a timeout, a 503, a deadlock. The same input may succeed later. */
public static class TransientFailure extends RuntimeException {
public TransientFailure(String message) {
super(message);
}
}
/**
* Not worth retrying: a validation failure, a missing referenced entity, a malformed field.
* The same input will fail identically ten times, and retrying it is pure latency.
*/
public static class PermanentFailure extends RuntimeException {
public PermanentFailure(String message) {
super(message);
}
}
private Failures() {
}
}

View File

@@ -0,0 +1,10 @@
package com.ankurm.kafkaerrors;
/** The payload. {@code paymentId} doubles as the record key. */
public record Payment(String paymentId, String status) {
public static Payment of(String paymentId) {
return new Payment(paymentId, "PENDING");
}
}

View File

@@ -0,0 +1,59 @@
package com.ankurm.kafkaerrors;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
/**
* A listener that fails on demand, so the retry machinery can be observed rather than described.
*
* <p>Payment ids beginning {@code transient-} throw a retryable exception; ids beginning
* {@code permanent-} throw a non-retryable one; everything else succeeds.
*
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
*/
@Component
public class PaymentListener {
private static final Logger log = LoggerFactory.getLogger(PaymentListener.class);
public static final String TOPIC = "payments";
/** Every delivery attempt, with the wall-clock time it happened. */
public record Attempt(String paymentId, long atMillis, int partition, long offset) {
}
private final List<Attempt> attempts = new CopyOnWriteArrayList<>();
@KafkaListener(topics = TOPIC, groupId = "payments")
public void onPayment(ConsumerRecord<String, Payment> record) {
Payment payment = record.value();
this.attempts.add(new Attempt(payment.paymentId(), System.currentTimeMillis(),
record.partition(), record.offset()));
log.info("attempt {} for {}", this.attempts.size(), payment.paymentId());
if (payment.paymentId().startsWith("transient-")) {
throw new Failures.TransientFailure("downstream unavailable for " + payment.paymentId());
}
if (payment.paymentId().startsWith("permanent-")) {
throw new Failures.PermanentFailure("invalid payment " + payment.paymentId());
}
}
public List<Attempt> attempts() {
return this.attempts;
}
public List<Attempt> attemptsFor(String paymentId) {
return this.attempts.stream().filter((a) -> a.paymentId().equals(paymentId)).toList();
}
public void clear() {
this.attempts.clear();
}
}

View File

@@ -0,0 +1,85 @@
package com.ankurm.kafkaerrors;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Profile;
import org.springframework.kafka.annotation.BackOff;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.SameIntervalTopicReuseStrategy;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
/**
* Non-blocking retries. Instead of holding the consumer thread while it re-delivers,
* {@code @RetryableTopic} republishes the failed record to a separate retry topic with a
* timestamp header, and a container for that topic waits before processing it.
*
* <p>The consequence people miss: <b>the main partition advances immediately</b>. That is the
* entire point &mdash; and it is also why per-key ordering is gone for any record that fails.
* Record 2 for the same key is processed while record 1 is sitting in a retry topic.
*
* @see <a href="../../../../../docs/05-retryable-topic.md">docs/05-retryable-topic.md</a>
*/
@Component
@Profile("retrytopic")
public class RetryableTopicListener {
private static final Logger log = LoggerFactory.getLogger(RetryableTopicListener.class);
public static final String TOPIC = "invoices";
public record Delivery(String topic, String paymentId, long atMillis) {
}
private final List<Delivery> deliveries = new CopyOnWriteArrayList<>();
private final List<Delivery> dead = new CopyOnWriteArrayList<>();
// backOff, not backoff - and org.springframework.kafka.annotation.BackOff, not
// org.springframework.retry.annotation.Backoff. Spring Kafka 4.x dropped the spring-retry
// dependency and brought its own annotation, so every pre-4.x @RetryableTopic example is a
// compile error: "package org.springframework.retry.annotation does not exist".
@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
exclude = Failures.PermanentFailure.class)
@KafkaListener(topics = TOPIC, groupId = "invoices")
public void onInvoice(ConsumerRecord<String, Payment> record,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
this.deliveries.add(new Delivery(topic, record.value().paymentId(), System.currentTimeMillis()));
log.info("delivery {} on topic {}", this.deliveries.size(), topic);
if (record.value().paymentId().startsWith("transient-")) {
throw new Failures.TransientFailure("still failing");
}
if (record.value().paymentId().startsWith("permanent-")) {
throw new Failures.PermanentFailure("never going to work");
}
}
/**
* Where a record lands once the attempts are exhausted. Without a {@code @DltHandler} the
* framework logs it and moves on &mdash; the topic still exists and still has the record, but
* nothing in your application has looked at it.
*/
@DltHandler
public void onDlt(ConsumerRecord<String, Payment> record,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
this.dead.add(new Delivery(topic, record.value().paymentId(), System.currentTimeMillis()));
log.info("DLT handler: {} on {}", record.value().paymentId(), topic);
}
public List<Delivery> deliveries() {
return this.deliveries;
}
public List<Delivery> dead() {
return this.dead;
}
}

View File

@@ -0,0 +1,27 @@
spring:
application:
name: kafka-error-handling
main:
banner-mode: off
kafka:
bootstrap-servers: localhost:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
consumer:
group-id: payments
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
# ErrorHandlingDeserializer wraps the real one. Without it a record that cannot be
# deserialized is thrown BEFORE the listener exists, the container cannot advance past it,
# and the same offset is retried forever. See docs/03-poison-pills.md.
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
properties:
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
spring.json.trusted.packages: com.ankurm.kafkaerrors
spring.json.value.default.type: com.ankurm.kafkaerrors.Payment
logging:
level:
root: WARN
com.ankurm: INFO
org.springframework.kafka.listener: WARN

View File

@@ -0,0 +1,83 @@
package com.ankurm.kafkaerrors;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.kafka.test.utils.KafkaTestUtils;
import org.springframework.test.context.ActiveProfiles;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The fix for the base64 problem in {@link DeadLetterTopicTest}: a per-value-type template map
* so that a {@code byte[]} is published with a {@code ByteArraySerializer}.
*
* @see <a href="../../../../../docs/03-poison-pills.md">docs/03-poison-pills.md</a>
*/
@SpringBootTest
@ActiveProfiles({ "test", "dltbytes" })
@EmbeddedKafka(topics = { "payments", "payments-dlt" }, partitions = 1)
class ByteAwareDltTest {
@Autowired
EmbeddedKafkaBroker broker;
@Test
void aByteArrayTemplateKeepsTheOriginalBytesIntact() {
Map<String, Object> producerProps = new HashMap<>(KafkaTestUtils.producerProps(this.broker.getBrokersAsString()));
producerProps.put("key.serializer", StringSerializer.class);
producerProps.put("value.serializer", ByteArraySerializer.class);
DefaultKafkaProducerFactory<String, byte[]> factory = new DefaultKafkaProducerFactory<>(producerProps);
try {
new KafkaTemplate<>(factory).send(PaymentListener.TOPIC, "poison-2",
"{ this is not json".getBytes(StandardCharsets.UTF_8));
}
finally {
factory.destroy();
}
Map<String, Object> consumerProps = new HashMap<>(
KafkaTestUtils.consumerProps(this.broker.getBrokersAsString(), "byte-dlt-reader", true));
consumerProps.put("key.deserializer", StringDeserializer.class);
consumerProps.put("value.deserializer", ByteArrayDeserializer.class);
consumerProps.put("auto.offset.reset", "earliest");
try (Consumer<String, byte[]> consumer = new KafkaConsumer<>(consumerProps)) {
consumer.subscribe(List.of("payments-dlt"));
long deadline = System.currentTimeMillis() + 30_000;
while (System.currentTimeMillis() < deadline) {
ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, byte[]> record : records) {
if ("poison-2".equals(record.key())) {
String payload = new String(record.value(), StandardCharsets.UTF_8);
System.out.println("=== byte-aware DLT ===");
System.out.println(" DLT payload -> " + payload);
// Byte for byte what was published. No base64, no quotes. Replay is now
// a copy from one topic to another.
assertThat(payload).isEqualTo("{ this is not json");
return;
}
}
}
throw new AssertionError("nothing arrived on payments-dlt");
}
}
}

View File

@@ -0,0 +1,192 @@
package com.ankurm.kafkaerrors;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.serializer.ErrorHandlingDeserializer;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.kafka.test.utils.KafkaTestUtils;
import org.springframework.test.context.ActiveProfiles;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
/**
* Retries, classification, and the two very different reasons a record ends up on the DLT.
*
* @see <a href="../../../../../docs/04-the-dlt.md">docs/04-the-dlt.md</a>
*/
@SpringBootTest
@ActiveProfiles({ "test", "dlt" })
@EmbeddedKafka(topics = { "payments", "payments-dlt" }, partitions = 1)
class DeadLetterTopicTest {
@Autowired
KafkaTemplate<String, Payment> template;
@Autowired
PaymentListener listener;
@Autowired
EmbeddedKafkaBroker broker;
@BeforeEach
void clear() {
this.listener.clear();
}
/**
* Reads the DLT as raw bytes, because a dead-lettered record may be exactly the thing that
* could not be deserialized.
*
* <p>Note the topic name: {@code payments-dlt}. {@code RetryTopicConstants.DEFAULT_DLT_SUFFIX}
* is {@code "-dlt"}, not {@code ".DLT"} &mdash; and if the topic does not exist, the
* recoverer logs a WARN and the record is gone.
*/
private Consumer<String, byte[]> dltConsumer() {
Map<String, Object> props = new HashMap<>(
KafkaTestUtils.consumerProps(this.broker.getBrokersAsString(), "dlt-reader-" + System.nanoTime(), true));
props.put("key.deserializer", org.apache.kafka.common.serialization.StringDeserializer.class);
props.put("value.deserializer", org.apache.kafka.common.serialization.ByteArrayDeserializer.class);
props.put("auto.offset.reset", "earliest");
Consumer<String, byte[]> consumer = new org.apache.kafka.clients.consumer.KafkaConsumer<>(props);
consumer.subscribe(java.util.List.of("payments-dlt"));
return consumer;
}
@Test
void aRetryableFailureIsRetriedAndThenPublishedToTheDlt() {
this.template.send(PaymentListener.TOPIC, "transient-1", Payment.of("transient-1"));
// FixedBackOff(1000, 2) = one delivery plus two retries.
await().atMost(Duration.ofSeconds(30))
.until(() -> this.listener.attemptsFor("transient-1").size() >= 3);
assertThat(this.listener.attemptsFor("transient-1")).hasSize(3);
var attempts = this.listener.attemptsFor("transient-1");
long gap = attempts.get(1).atMillis() - attempts.get(0).atMillis();
System.out.println("=== transient failure ===");
System.out.println(" deliveries " + attempts.size());
System.out.println(" gap between 1&2 " + gap + " ms (FixedBackOff interval 1000)");
assertThat(gap).isGreaterThanOrEqualTo(900);
try (Consumer<String, byte[]> consumer = dltConsumer()) {
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "transient-1");
assertThat(dead).isNotNull();
printHeaders("transient failure on the DLT", dead);
// The TOP-LEVEL exception header is always the wrapper. Group a DLT triage dashboard
// by kafka_dlt-exception-fqcn and every listener failure in the estate lands in one
// bucket called ListenerExecutionFailedException. The useful field is the cause.
assertThat(header(dead, "kafka_dlt-exception-fqcn"))
.isEqualTo("org.springframework.kafka.listener.ListenerExecutionFailedException");
assertThat(header(dead, "kafka_dlt-exception-cause-fqcn"))
.isEqualTo(Failures.TransientFailure.class.getName());
assertThat(header(dead, "kafka_dlt-original-topic")).isEqualTo("payments");
}
}
@Test
void aNonRetryableFailureGoesStraightToTheDltWithNoRetries() {
this.template.send(PaymentListener.TOPIC, "permanent-1", Payment.of("permanent-1"));
try (Consumer<String, byte[]> consumer = dltConsumer()) {
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "permanent-1");
assertThat(dead).isNotNull();
// addNotRetryableExceptions(PermanentFailure.class) means exactly one delivery.
System.out.println("=== permanent failure ===");
System.out.println(" deliveries " + this.listener.attemptsFor("permanent-1").size());
assertThat(this.listener.attemptsFor("permanent-1")).hasSize(1);
assertThat(header(dead, "kafka_dlt-exception-cause-fqcn"))
.isEqualTo(Failures.PermanentFailure.class.getName());
}
}
@Test
void aPoisonPillNeverReachesTheListenerAndIsStillRecovered() {
// Raw bytes that are not valid JSON, published with a byte[] serializer so nothing on
// the producing side objects. This is what a schema change from another team looks like.
Map<String, Object> props = new HashMap<>(KafkaTestUtils.producerProps(this.broker.getBrokersAsString()));
props.put("key.serializer", StringSerializer.class);
props.put("value.serializer", ByteArraySerializer.class);
DefaultKafkaProducerFactory<String, byte[]> factory = new DefaultKafkaProducerFactory<>(props);
try {
new KafkaTemplate<>(factory).send(PaymentListener.TOPIC, "poison-1",
"{ this is not json".getBytes(StandardCharsets.UTF_8));
}
finally {
factory.destroy();
}
try (Consumer<String, byte[]> consumer = dltConsumer()) {
ConsumerRecord<String, byte[]> dead = pollFor(consumer, "poison-1");
assertThat(dead).isNotNull();
printHeaders("poison pill on the DLT", dead);
// The listener was never invoked - the failure happened in the deserializer, before
// the listener existed. ErrorHandlingDeserializer is what turns that into a record
// the error handler can recover instead of a poll that fails forever.
assertThat(this.listener.attemptsFor("poison-1")).isEmpty();
assertThat(header(dead, "kafka_dlt-exception-fqcn"))
.contains("DeserializationException");
// And here is the sting. The recoverer publishes with the APPLICATION's producer,
// whose value serializer is JacksonJsonSerializer. The failed value is a byte[], and
// Jackson writes a byte[] as a base64 JSON string. So the DLT does not hold the
// original bytes - it holds base64 of them, wrapped in quotes.
String payload = new String(dead.value(), StandardCharsets.UTF_8);
System.out.println(" DLT payload -> " + payload);
assertThat(payload).isEqualTo("\"eyB0aGlzIGlzIG5vdCBqc29u\"");
assertThat(new String(java.util.Base64.getDecoder().decode(
payload.replace("\"", "")), StandardCharsets.UTF_8)).isEqualTo("{ this is not json");
}
}
private ConsumerRecord<String, byte[]> pollFor(Consumer<String, byte[]> consumer, String key) {
long deadline = System.currentTimeMillis() + 30_000;
while (System.currentTimeMillis() < deadline) {
ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, byte[]> record : records) {
if (key.equals(record.key())) {
return record;
}
}
}
return null;
}
private static String header(ConsumerRecord<String, byte[]> record, String name) {
var header = record.headers().lastHeader(name);
return (header == null) ? null : new String(header.value(), StandardCharsets.UTF_8);
}
private static void printHeaders(String title, ConsumerRecord<String, byte[]> record) {
System.out.println("=== " + title + " ===");
record.headers().forEach((h) -> {
// Some of these headers are binary (the original partition, offset and timestamp are
// big-endian ints and longs, not text), so render anything non-printable as hex.
String value = new String(h.value(), StandardCharsets.UTF_8).replaceAll("\\s+", " ");
if (!value.chars().allMatch((c) -> c >= 0x20 && c < 0x7f)) {
StringBuilder hex = new StringBuilder("0x");
for (byte b : h.value()) {
hex.append("%02x".formatted(b));
}
value = hex.toString();
}
System.out.printf(" %-34s %s%n", h.key(), value.length() > 90 ? value.substring(0, 90) + "..." : value);
});
}
}

View File

@@ -0,0 +1,48 @@
package com.ankurm.kafkaerrors;
import org.junit.jupiter.api.Test;
import org.springframework.kafka.listener.SeekUtils;
import org.springframework.util.backoff.BackOffExecution;
import org.springframework.util.backoff.FixedBackOff;
import java.util.ArrayList;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
/**
* What the stock {@code DefaultErrorHandler} actually does, read out of its own back-off rather
* than out of the documentation. No broker and no context: a {@code BackOff} is a state machine
* you can just run.
*
* @see <a href="../../../../../docs/02-default-error-handler.md">docs/02-default-error-handler.md</a>
*/
class DefaultBackOffTest {
@Test
void theDefaultIsTenAttemptsWithNoDelayAtAll() {
FixedBackOff backOff = SeekUtils.DEFAULT_BACK_OFF;
BackOffExecution execution = backOff.start();
List<Long> intervals = new ArrayList<>();
long next;
while ((next = execution.nextBackOff()) != BackOffExecution.STOP) {
intervals.add(next);
}
System.out.println("=== DefaultErrorHandler default back-off ===");
System.out.println(" interval " + backOff.getInterval() + " ms");
System.out.println(" max attempts " + backOff.getMaxAttempts() + " retries");
System.out.println(" SeekUtils.DEFAULT_MAX_FAILURES = " + SeekUtils.DEFAULT_MAX_FAILURES);
System.out.println(" retry intervals " + intervals);
System.out.println(" total deliveries " + (intervals.size() + 1));
// Nine retries after the first delivery = ten deliveries, matching DEFAULT_MAX_FAILURES.
assertThat(intervals).hasSize(SeekUtils.DEFAULT_MAX_FAILURES - 1);
// And every one of them is zero. The default is not "retry with backoff"; it is
// "hammer the same record ten times as fast as the consumer thread can go, then give up".
assertThat(intervals).containsOnly(0L);
assertThat(backOff.getInterval()).isZero();
}
}

View File

@@ -0,0 +1,82 @@
package com.ankurm.kafkaerrors;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.ActiveProfiles;
import java.time.Duration;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
/**
* Non-blocking retries, and the topics they create.
*
* @see <a href="../../../../../docs/05-retryable-topic.md">docs/05-retryable-topic.md</a>
*/
@SpringBootTest
@ActiveProfiles({ "test", "retrytopic" })
@EmbeddedKafka(topics = { "invoices" }, partitions = 1)
class RetryableTopicTest {
@Autowired
KafkaTemplate<String, Payment> template;
@Autowired
RetryableTopicListener listener;
@Autowired
EmbeddedKafkaBroker broker;
@Test
void retriesHappenOnSeparateTopicsAndTheMainPartitionKeepsMoving() {
this.template.send(RetryableTopicListener.TOPIC, "transient-1", Payment.of("transient-1"));
// Sent immediately after the failing one, on the same partition. If retries were
// blocking, this could not be processed until the first record gave up.
this.template.send(RetryableTopicListener.TOPIC, "ok-1", Payment.of("ok-1"));
await().atMost(Duration.ofSeconds(60)).until(() -> !this.listener.dead().isEmpty());
List<RetryableTopicListener.Delivery> deliveries = this.listener.deliveries();
System.out.println("=== @RetryableTopic delivery trace ===");
long start = deliveries.get(0).atMillis();
deliveries.forEach((d) -> System.out.printf(" +%-6d ms %-28s %s%n",
d.atMillis() - start, d.topic(), d.paymentId()));
System.out.println(" DLT: " + this.listener.dead().stream()
.map((d) -> d.paymentId() + " on " + d.topic()).toList());
System.out.println(" topics touched: " + deliveries.stream()
.map(RetryableTopicListener.Delivery::topic).distinct().sorted().toList());
// attempts = "4" means the first delivery plus three retries.
assertThat(deliveries).filteredOn((d) -> d.paymentId().equals("transient-1")).hasSize(4);
// Retries land on generated topics, not on the original one.
// Retry topics are named by the DELAY, not by the attempt number. With a multiplier,
// each distinct interval gets its own topic: invoices-retry-500, -1000, -2000. That is
// TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE, the default. Provisioning a cluster
// for this means knowing the whole back-off schedule in advance.
assertThat(deliveries).filteredOn((d) -> d.paymentId().equals("transient-1"))
.extracting(RetryableTopicListener.Delivery::topic)
.containsExactly("invoices", "invoices-retry-500", "invoices-retry-1000",
"invoices-retry-2000");
assertThat(this.listener.dead()).extracting(RetryableTopicListener.Delivery::topic)
.containsExactly("invoices-dlt");
// The good record was processed while the bad one was still being retried. That is the
// whole benefit, and the whole cost: ordering on this partition is gone.
long okAt = deliveries.stream().filter((d) -> d.paymentId().equals("ok-1"))
.findFirst().orElseThrow().atMillis();
long lastRetryAt = deliveries.stream().filter((d) -> d.paymentId().equals("transient-1"))
.mapToLong(RetryableTopicListener.Delivery::atMillis).max().orElseThrow();
assertThat(okAt).isLessThan(lastRetryAt);
assertThat(this.listener.dead()).extracting(RetryableTopicListener.Delivery::paymentId)
.containsExactly("transient-1");
}
}

View File

@@ -0,0 +1,3 @@
spring:
kafka:
bootstrap-servers: ${spring.embedded.kafka.brokers}

80
rabbitmq/README.md Normal file
View File

@@ -0,0 +1,80 @@
# `rabbitmq` — exchanges, bindings and a dead-letter queue that works
Companion project for
[**Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue**](https://ankurm.com/spring-boot-rabbitmq-exchanges-dead-letter-queue/)
on ankurm.com.
Ten tests against a **real RabbitMQ broker**. All four exchange types, manual acknowledgement,
a dead-letter path exercised through `basicNack` and through TTL expiry, and the three ways a
topology loses work quietly.
## Versions
| | Version | Notes |
|---|---|---|
| JDK | 25 (Temurin 25.0.4.1+1) | current LTS |
| Spring Boot | 4.1.1 | |
| Spring AMQP / Spring Rabbit | 4.1.1 | Boot-managed |
| `com.rabbitmq:amqp-client` | **5.30.0** | Boot-managed. Central has 5.35.0 |
| Testcontainers | 2.0.5 | artifact is `testcontainers-rabbitmq`, not `rabbitmq` |
| **Broker for the committed transcripts** | **RabbitMQ 3.10.25** | see the note below |
Client-side versions were read from `repo1.maven.org/.../maven-metadata.xml` and Boot's
`spring-boot-dependencies` POM.
> **About the broker version.** The machine that regenerates `docs/output/` has no Docker daemon
> and no root, so `scripts/broker.sh` starts a generic-unix RabbitMQ against an extracted Erlang
> 24 runtime, and the newest release that pairs with Erlang 24 is 3.10.25. Everything exercised
> here — the four exchange types, `x-dead-letter-exchange`, `x-dead-letter-routing-key`,
> `x-message-ttl`, `x-death`, manual ack, `mandatory` returns, `PRECONDITION_FAILED` on
> inequivalent arguments — is AMQP 0-9-1 behaviour that is unchanged in RabbitMQ 4.x. The one
> broker-version difference worth knowing for 4.x is that **quorum queues are the default queue
> type** and classic mirrored queues are gone; nothing in this module declares a queue type, so
> the topology is valid on both. If you have Docker, run the same tests against
> `rabbitmq:4.1-management` with
> [`TestcontainersConfiguration`](src/test/java/com/ankurm/rabbit/TestcontainersConfiguration.java).
## Quickstart
```bash
# with Docker: point the tests at Testcontainers, or run your own broker on 5672
mvn test
# without Docker:
export ERL_ROOT=/path/to/erlang RABBITMQ_HOME=/path/to/rabbitmq_server-3.10.25
./scripts/run-all.sh
```
## Documentation
1. [The on-ramp](docs/01-the-on-ramp.md)
2. [Four exchange types](docs/02-exchanges.md)
3. [The message that goes nowhere and says nothing](docs/03-the-silent-drop.md)
4. [A dead-letter queue that actually works](docs/04-dead-lettering.md)
5. [Acknowledgement](docs/05-acknowledgement.md)
6. [Changing your mind about a queue](docs/06-changing-your-mind.md)
## Captured output
| File | Shows |
|---|---|
| [`topic-wildcards.txt`](docs/output/topic-wildcards.txt) | `*` vs `#` over five routing keys |
| [`dead-letter.txt`](docs/output/dead-letter.txt) | `x-death` for `rejected` and for `expired` |
| [`requeue-loop.txt`](docs/output/requeue-loop.txt) | 199 redeliveries, 0 dead-lettered |
| [`unroutable.txt`](docs/output/unroutable.txt) | `312 NO_ROUTE` |
| [`precondition-failed.txt`](docs/output/precondition-failed.txt) | `406` on an inequivalent argument |
| [`tests.txt`](docs/output/tests.txt) | 10 tests |
## Five things this module exists to prove
1. **An unroutable message is discarded silently**, and finding out needs `publisher-returns`
*and* `mandatory` — two settings in two different places. Setting only `mandatory` does
nothing.
2. **`requeue=true` never dead-letters.** 200 attempts, 199 redeliveries, an empty DLQ and a
queue depth that stays at 1 the whole time.
3. **`x-death.reason` distinguishes `rejected` from `expired`**, which is the difference between
a consumer that refused the work and a consumer that never got to it.
4. **Queue arguments are immutable**`406 PRECONDITION_FAILED`, wrapped in an exception whose
own message is the string `java.io.IOException`.
5. **Boot auto-configures no JSON converter for RabbitMQ**, and the converter you want is
`JacksonJsonMessageConverter`, not `Jackson2JsonMessageConverter`.

View File

@@ -0,0 +1,59 @@
[Module README](../README.md) &middot; [Exchanges &rarr;](02-exchanges.md)
# 1. The on-ramp
Two dependency decisions decide whether anything works, and neither produces a helpful error.
## Use the starter, not `spring-rabbit`
Boot 4 moved every auto-configuration out of `spring-boot-autoconfigure` into a per-technology
module. RabbitMQ's lives in `spring-boot-amqp`, package
`org.springframework.boot.amqp.autoconfigure`, and a bare `org.springframework.amqp:spring-rabbit`
dependency does not bring it. You get no `RabbitTemplate`, no `RabbitAdmin`, no listener
container factory — and a context that starts cleanly.
```xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
```
## Boot does not give you a JSON converter
Spring Kafka defaults to serializers you configure. Spring AMQP defaults to
`SimpleMessageConverter`, which handles `String`, `byte[]` and `Serializable` and refuses
everything else:
```
IllegalArgumentException: SimpleMessageConverter only supports String, byte[] and
Serializable payloads, received: com.ankurm.rabbit.OrderMessage
```
One bean fixes both directions, because the auto-configured `RabbitTemplate` and the listener
container factory both look for a `MessageConverter`:
```java
@Bean
MessageConverter messageConverter() {
return new JacksonJsonMessageConverter();
}
```
**Note the class name.** Spring AMQP 4.1 ships `Jackson2JsonMessageConverter` (Jackson 2) and
`JacksonJsonMessageConverter` (Jackson 3) side by side — exactly as Spring Kafka ships
`JsonSerializer` and `JacksonJsonSerializer`. Boot 4 is a Jackson 3 application. The rule across
both stacks is the same: **if the class name contains a `2`, it belongs to the previous major
version of Jackson.**
## Versions
Boot 4.1.1 manages Spring AMQP **4.1.1** and `com.rabbitmq:amqp-client` **5.30.0**. Maven Central
has amqp-client 5.35.0; overriding the managed version to reach it is a change you should have a
reason for.
One API change to know about: `RabbitAdmin.QUEUE_MESSAGE_COUNT` now holds a `Long`. Every older
example casts it to `Integer`, which is a `ClassCastException` at runtime and not at compile
time.
[Exchanges &rarr;](02-exchanges.md)

View File

@@ -0,0 +1,71 @@
[&larr; The on-ramp](01-the-on-ramp.md) &middot; [Module README](../README.md) &middot; [The silent drop &rarr;](03-the-silent-drop.md)
# 2. Four exchange types
A producer never publishes to a queue. It publishes to an **exchange** with a **routing key**,
and bindings decide where that lands. The whole topology is in
[`Topology.java`](../src/main/java/com/ankurm/rabbit/Topology.java) as beans; `RabbitAdmin`
declares them when the connection opens.
## Direct — exact match
Binding key `new` receives routing key `new`. Nothing else. This is the workhorse: one queue per
command type.
## Fanout — routing key ignored entirely
Every bound queue gets a copy. `audit.all` and `analytics.all` both receive it, and the routing
key you passed is not consulted at all. Use it for broadcast; use it knowing that adding a queue
adds a full copy of the traffic.
## Topic — wildcards over dot-separated words
`*` is **exactly one word**. `#` is **zero or more**. The distinction is the one people get
wrong, so here it is against a real broker
([`docs/output/topic-wildcards.txt`](output/topic-wildcards.txt)):
```
routing key orders.eu orders.high note
order.eu.high true true matches both
order.eu.low true false matches order.eu.* only
order.us.high false true matches order.#.high only
order.eu.west.high false true matches order.#.high only - * is one word
order.high false true matches order.#.high - # can be zero words
```
Bindings are `order.eu.*` and `order.#.high`. Two rows are worth pausing on:
- `order.eu.west.high` does **not** match `order.eu.*`, because `*` matches one word and `west.high`
is two. Regex intuition says otherwise.
- `order.high` **does** match `order.#.high`, because `#` matches zero words. So a binding you
wrote to mean "something in the middle" also matches "nothing in the middle".
Design routing keys most-general-to-most-specific (`order.eu.west.high`, not
`high.order.eu.west`), because `#` and `*` work left to right and a hierarchy you can bind
usefully is one that starts broad.
## Headers — match a map, ignore the routing key
`x-match=all` requires every named header to be present **and equal**. `x-match=any` requires
one. From `RoutingTest`:
| Message headers | `x-match=all` queue | `x-match=any` queue |
|---|---|---|
| `priority=high, region=eu` | yes | yes |
| `priority=high` | no | yes |
| `priority=high, region=us` | no | yes |
The third row is the one to remember: `all` matches on **value**, not on presence. A header that
is there with the wrong value fails the same way a missing one does.
Headers exchanges are slower than topic exchanges and much less common. Reach for them when the
routing criteria are genuinely multi-dimensional and do not compose into a hierarchy.
## The default exchange
Publishing to the empty exchange name `""` routes by **queue name**, using the routing key as the
queue name. Every queue is implicitly bound to it. That is how
`convertAndSend("", "orders.work", message)` works, and it is the one piece of AMQP that behaves
like a magic constant.
[The silent drop &rarr;](03-the-silent-drop.md)

View File

@@ -0,0 +1,78 @@
[&larr; Exchanges](02-exchanges.md) &middot; [Module README](../README.md) &middot; [Dead-lettering &rarr;](04-dead-lettering.md)
# 3. The message that goes nowhere and says nothing
Publish to `orders.direct` with routing key `amend`. No binding matches. The broker discards the
message. `convertAndSend` returns normally. Nothing is logged, by the broker or by Spring, and
the message is gone.
This is correct AMQP behaviour and it is the single most expensive default in RabbitMQ, because
the symptom is "the consumer isn't running" and the cause is three services away.
**Two settings are required, and they live in different places:**
```yaml
spring:
rabbitmq:
publisher-returns: true # on the connection factory
template:
mandatory: true # on the template
```
`publisher-returns` puts the connection into a mode where returns are listened for at all.
`mandatory` asks the broker to return *this* publish if it is unroutable. Setting only
`mandatory` — which is what most examples show — changes nothing, and the test in this module
was written that way first and failed.
Then register a callback:
```java
template.setReturnsCallback((returned) -> log.error("unroutable: exchange={} key={} {} {}",
returned.getExchange(), returned.getRoutingKey(),
returned.getReplyCode(), returned.getReplyText()));
```
From [`docs/output/unroutable.txt`](output/unroutable.txt):
```
replyCode 312
replyText NO_ROUTE
exchange orders.direct
routingKey amend
```
`312 NO_ROUTE` is the code worth putting in an alert.
## What returns still do not give you
The callback fires **asynchronously, on the connection's thread**. `convertAndSend` had already
returned successfully by then. So a return tells you a message was lost; it does not let you
stop the caller from believing it succeeded.
If the caller must know, you need **publisher confirms** as well:
```yaml
spring:
rabbitmq:
publisher-confirm-type: correlated
```
Confirms and returns answer different questions, and you usually want both:
| | answers |
|---|---|
| **return** | the broker had nowhere to route this |
| **confirm** | the broker took responsibility for this (`ack`), or refused it (`nack`) |
A message that is unroutable gets a **return followed by an `ack`** — the broker successfully did
nothing with it. So a confirm alone will not tell you the message vanished, which is the trap
inside the trap.
## The alternate exchange
The topology-level version of the same fix: give an exchange an `alternate-exchange` argument and
unroutable messages go there instead of being dropped, with no publisher-side configuration at
all. It costs one exchange and one queue, and it catches the publishes made by services that
forgot to set `mandatory`.
[Dead-lettering &rarr;](04-dead-lettering.md)

View File

@@ -0,0 +1,112 @@
[&larr; The silent drop](03-the-silent-drop.md) &middot; [Module README](../README.md) &middot; [Acknowledgement &rarr;](05-acknowledgement.md)
# 4. A dead-letter queue that actually works
There is no "send to DLQ" operation in AMQP. A dead-letter exchange is a **queue argument**, and
messages arrive there as a side effect of three specific events.
```java
QueueBuilder.durable("orders.work")
.deadLetterExchange("orders.dlx")
.deadLetterRoutingKey("failed")
.build();
```
`deadLetterRoutingKey` matters more than it looks. Without it, the message is republished with
its **original** routing key, so your DLQ binding has to anticipate every routing key the
upstream might have used. Setting it to a constant means one binding catches everything.
## The three triggers
| Event | `x-death` reason |
|---|---|
| `basicNack` / `basicReject` with `requeue=false` | `rejected` |
| message TTL expires (`x-message-ttl` or per-message) | `expired` |
| queue length or byte limit exceeded (`x-max-length`) | `maxlen` |
And the one that is **not** a trigger: `requeue=true`. More on that below.
## What arrives
Both captured from real runs, in
[`docs/output/dead-letter.txt`](output/dead-letter.txt):
```
=== x-death after basicNack(requeue=false) ===
reason rejected
count 1
exchange
time Sat Aug 29 09:53:17 IST 2026
routing-keys [orders.work]
queue orders.work
=== x-death after x-message-ttl expiry ===
reason expired
count 1
queue orders.ttl
```
Same destination queue, different `reason`. That field is the whole diagnostic: `rejected` means
a consumer looked at the work and refused it; `expired` means nobody got to it in time. Those are
different incidents with different fixes, and they are indistinguishable without reading the
header.
The third trigger has a detail worth knowing: when `x-max-length` is exceeded RabbitMQ drops from
the **head**, so the message that gets dead-lettered is the **oldest** one already queued, not the
one that just arrived:
```
=== x-death after x-max-length overflow ===
reason maxlen
queue orders.bounded
body {"orderId":"m-1","detail":"detail for m-1"}
```
Three messages into a queue that holds two, and `m-1` is the one on the DLQ. A bounded queue
under sustained overload therefore dead-letters your *backlog* while continuing to accept new
work &mdash; which is usually what you want for telemetry and exactly wrong for orders.
`x-death` is a **list**, not a map — one entry per queue the message has been dead-lettered from,
and `count` accumulates. That is how you build a retry limit: read
`x-death[0].count`, and stop republishing past a threshold.
## The infinite loop
```java
channel.basicNack(deliveryTag, false, true); // requeue = true
```
From [`docs/output/requeue-loop.txt`](output/requeue-loop.txt), 200 delivery attempts:
```
redelivered 199 times
dead-lettered 0
still on queue 1
```
The dead-letter exchange is never consulted. `x-death` is never written. There is no counter to
exhaust and no backoff. One poison message with an unconditional `requeue=true` handler will
saturate a consumer indefinitely, and the queue depth stays at 1 the whole time — so a depth
alarm will not fire either. The only visible signal is CPU.
**`requeue=true` is only correct when the failure is transient and you have a delay.** With no
delay, a message that fails because a database is down comes straight back while the database is
still down. The `redelivered` flag on the envelope is your one piece of state:
```java
boolean retried = response.getEnvelope().isRedeliver();
channel.basicNack(tag, false, !retried); // one requeue, then dead-letter
```
That is a one-attempt retry with no extra infrastructure. For anything more, use a delay queue:
a queue with a TTL and a DLX pointing back at the work exchange turns "expire after 30 seconds"
into "retry in 30 seconds".
## Do not point the DLQ's DLX at its own source
If `orders.dlq` dead-letters to an exchange bound back to `orders.work`, a message that fails
consistently cycles between the two forever, gaining an `x-death` entry each time until the
header itself is the biggest thing in the message. The DLQ in this module deliberately has no
dead-letter exchange at all.
[Acknowledgement &rarr;](05-acknowledgement.md)

View File

@@ -0,0 +1,82 @@
[&larr; Dead-lettering](04-dead-lettering.md) &middot; [Module README](../README.md) &middot; [Changing your mind &rarr;](06-changing-your-mind.md)
# 5. Acknowledgement
```yaml
spring:
rabbitmq:
listener:
simple:
acknowledge-mode: manual
prefetch: 1
```
Three modes, and the default is not the one you want for work that matters.
| Mode | Behaviour |
|---|---|
| `auto` (default) | the container acks after the listener returns, nacks on exception |
| `manual` | your code calls `basicAck` / `basicNack`; nothing else does |
| `none` | the broker forgets the message the moment it is delivered |
(Boot publishes no default for `acknowledge-mode` either; the field initialiser in
`AbstractMessageListenerContainer` is `AcknowledgeMode.AUTO`.)
`none` is genuinely fire-and-forget: no redelivery, ever, and no flow control either, because
`prefetch` is meaningless without acknowledgement. It is the right choice for metrics and the
wrong choice for everything else.
`auto` is a reasonable default and its name is misleading: it does not mean "auto-ack" in the
AMQP sense (that is `none`). It means the container decides, and it decides by whether your
listener threw.
`manual` is what you want when the acknowledgement must happen after some other durable side
effect — a database commit, a downstream call. In `manual` mode the listener takes a `Channel`
and a delivery tag:
```java
@RabbitListener(queues = "orders.work")
void onOrder(OrderMessage message, Channel channel,
@Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
try {
process(message);
channel.basicAck(tag, false);
}
catch (TransientFailure ex) {
channel.basicNack(tag, false, !alreadyRedelivered);
}
catch (PermanentFailure ex) {
channel.basicNack(tag, false, false); // dead-letter it
}
}
```
The two catch blocks are the point. A single `catch (Exception)` that nacks with `requeue=true`
is the loop from [chapter 4](04-dead-lettering.md); a single one that nacks with `requeue=false`
dead-letters transient failures that would have succeeded on a retry. Deciding which kind of
failure you had is work you cannot delegate to the container.
## `prefetch` is the flow-control knob
`prefetch` is how many un-acknowledged messages the broker will send a consumer. Spring Boot sets
no default, so `AbstractMessageListenerContainer.DEFAULT_PREFETCH_COUNT` applies, and that
constant is **250** &mdash; throughput-oriented: a slow consumer holds 250 messages that no
other consumer can take, so one stuck worker parks 250 messages behind it.
Set it to 1 when messages are expensive and unevenly sized — it gives you round-robin
distribution across consumers. Leave it high when messages are small and uniform and you want
throughput. This is the single most effective RabbitMQ tuning parameter and it is almost never
touched.
## The forgotten ack
Not calling `basicAck` does not lose the message and does not throw. The message stays
un-acknowledged, counting against `prefetch`, until the channel closes. With `prefetch: 1` the
consumer stops after exactly one message; with the default it stops after 250. Both look like the
consumer "just stopped", hours apart in wall-clock time depending on traffic.
`rabbitmqctl list_queues name messages_ready messages_unacknowledged` is the command that
distinguishes them. A large `messages_unacknowledged` with a live consumer means somebody forgot
an ack.
[Changing your mind &rarr;](06-changing-your-mind.md)

View File

@@ -0,0 +1,58 @@
[&larr; Acknowledgement](05-acknowledgement.md) &middot; [Module README](../README.md)
# 6. Changing your mind about a queue
Queue arguments are immutable. There is no `ALTER QUEUE`. Redeclaring an existing queue with
different arguments is a channel-level error
([`docs/output/precondition-failed.txt`](output/precondition-failed.txt)):
```
PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/':
received '9999' but current is '1500', class-id=50, method-id=10
```
Reply code **406**. The message is precise and helpful, and you will probably never see it,
because of what Spring wraps it in:
```
AmqpIOException: java.io.IOException
caused by: null
caused by: channel error; protocol method: #method<channel.close>(reply-code=406, ...)
```
`AmqpIOException`'s own message is the string `java.io.IOException`. The next cause's message is
`null`. The useful text is two levels down. This is why the failure gets reported as "some
IOException from RabbitMQ" and why it is worth writing a log statement that walks the cause
chain.
## What this means operationally
Changing a TTL, a max-length, a dead-letter exchange or a queue type on a live queue is not a
config edit. It is a migration:
1. declare the new queue under a new name
2. bind it alongside the old one
3. move consumers over
4. drain the old queue
5. delete it
Plan the rename into the change from the start — `orders.work.v2` is not ugly, it is honest.
## Related failure modes
**A failed declaration stops the ones after it.** `RabbitAdmin` declares beans on connection, and
a `PRECONDITION_FAILED` closes the channel. Declarations queued behind it on that channel do not
happen. So one stale queue argument can leave half your topology undeclared, and the symptom is a
*different* queue being missing.
**Declarations happen on connection, not on context refresh.** A broken topology does not fail
startup; it fails the first publish. If you want it to fail at boot, force a connection early —
`connectionFactory.createConnection()` in an `ApplicationRunner` is enough.
**`missing-queues-fatal` defaults differently for the two container types.** It is `true` for
`spring.rabbitmq.listener.simple` and `false` for `spring.rabbitmq.listener.direct`. So the same
missing queue kills a simple container and is quietly tolerated by a direct one, which makes
"it works in that service" an unhelpful data point. Combined with the point above, a topology
error can present as a listener startup problem in one service and as silence in another.
[Module README](../README.md)

View File

@@ -0,0 +1,22 @@
=== x-death after basicNack(requeue=false) ===
reason rejected
count 1
exchange
time Sat Aug 29 10:30:04 IST 2026
routing-keys [orders.work]
queue orders.work
=== x-death after x-message-ttl expiry ===
reason expired
count 1
exchange
time Sat Aug 29 10:30:06 IST 2026
routing-keys [orders.ttl]
queue orders.ttl
=== x-death after x-max-length overflow ===
reason maxlen
count 1
exchange orders.direct
time Sat Aug 29 10:30:06 IST 2026
routing-keys [bounded]
queue orders.bounded
body {"orderId":"m-1","detail":"detail for m-1"}

View File

@@ -0,0 +1,2 @@
=== redeclaring orders.ttl with x-message-ttl=9999 ===
java.io.IOException | null | channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/': received '9999' but current is '1500', class-id=50, method-id=10) | channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'x-message-ttl' for queue 'orders.ttl' in vhost '/': received '9999' but current is '1500', class-id=50, method-id=10) |

View File

@@ -0,0 +1,4 @@
=== basicNack(requeue=true), 200 attempts ===
redelivered 199 times
dead-lettered 0
still on queue 1

View File

@@ -0,0 +1,4 @@
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.132 s -- in com.ankurm.rabbit.TopologyTrapsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.714 s -- in com.ankurm.rabbit.DeadLetterTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 s -- in com.ankurm.rabbit.MaxLengthTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.476 s -- in com.ankurm.rabbit.RoutingTest

View File

@@ -0,0 +1,7 @@
=== topic exchange ===
routing key orders.eu orders.high note
order.eu.high true true matches both
order.eu.low true false matches order.eu.* only
order.us.high false true matches order.#.high only
order.eu.west.high false true matches order.#.high only - * is one word
order.high false true matches order.#.high - # can be zero words

View File

@@ -0,0 +1,5 @@
=== returned message ===
replyCode 312
replyText NO_ROUTE
exchange orders.direct
routingKey amend

67
rabbitmq/pom.xml Normal file
View File

@@ -0,0 +1,67 @@
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>rabbitmq-exchanges-and-dlq</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<!-- spring-boot-starter-amqp, not a bare org.springframework.amqp:spring-rabbit. In Boot 4
the auto-configuration lives in the spring-boot-amqp module (package
org.springframework.boot.amqp.autoconfigure); depending on spring-rabbit alone gives
you no RabbitTemplate and no listener container factory. Same trap as Kafka. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jackson</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit-test</artifactId>
<scope>test</scope>
</dependency>
<!-- NOT org.testcontainers:rabbitmq, which stopped at 1.21.4. Testcontainers 2.x prefixed
every module artifact and Boot 4.1.1 imports testcontainers-bom 2.0.5. -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers-rabbitmq</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-testcontainers</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

28
rabbitmq/scripts/broker.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
# Start a real RabbitMQ broker with no Docker and no root.
#
# RabbitMQ is an Erlang application, so it needs an Erlang runtime and epmd on the path - that
# is the whole dependency. Point ERL_ROOT at an Erlang installation and RABBITMQ_HOME at an
# unpacked rabbitmq-server-generic-unix tarball, and it starts in about fifteen seconds.
#
# If you have Docker, use Testcontainers instead: see src/test/java/.../TestcontainersConfiguration.java
set -eu
: "${ERL_ROOT:?set ERL_ROOT to an Erlang installation directory}"
: "${RABBITMQ_HOME:?set RABBITMQ_HOME to an unpacked rabbitmq-server-generic-unix directory}"
export PATH="$ERL_ROOT/bin:$ERL_ROOT/erts-"*/bin":$PATH"
export RABBITMQ_MNESIA_BASE="${RABBITMQ_MNESIA_BASE:-/tmp/rmq/data}"
export RABBITMQ_LOG_BASE="${RABBITMQ_LOG_BASE:-/tmp/rmq/log}"
export RABBITMQ_NODENAME="${RABBITMQ_NODENAME:-rabbit@localhost}"
export HOME="${HOME:-/tmp/rmq}"
mkdir -p "$RABBITMQ_MNESIA_BASE" "$RABBITMQ_LOG_BASE"
epmd -daemon 2>/dev/null || true
sleep 1
setsid nohup "$RABBITMQ_HOME/sbin/rabbitmq-server" > /tmp/rmq/boot.log 2>&1 < /dev/null &
for _ in $(seq 1 60); do
if (echo > /dev/tcp/127.0.0.1/5672) 2>/dev/null; then echo "broker ready on 5672"; exit 0; fi
sleep 1
done
echo "broker did not start; see /tmp/rmq/boot.log" >&2
exit 1

19
rabbitmq/scripts/run-all.sh Executable file
View File

@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Regenerate every file under docs/output/.
#
# Starts a real RabbitMQ broker first. Set ERL_ROOT and RABBITMQ_HOME (see scripts/broker.sh),
# or point spring.rabbitmq.host at a broker you already have and skip the first line.
set -eu
cd "$(dirname "$0")/.."
./scripts/broker.sh
mvn -B test 2>&1 | tee /tmp/rabbit-test.log > /dev/null
sed -n '/=== topic exchange ===/,/^order\.high/p' /tmp/rabbit-test.log > docs/output/topic-wildcards.txt
sed -n '/=== x-death after basicNack/,/^ queue/p' /tmp/rabbit-test.log > docs/output/dead-letter.txt
sed -n '/=== x-death after x-message-ttl/,/^ queue/p' /tmp/rabbit-test.log >> docs/output/dead-letter.txt
sed -n '/=== x-death after x-max-length/,/^ body/p' /tmp/rabbit-test.log >> docs/output/dead-letter.txt
sed -n '/=== basicNack(requeue=true)/,/still on queue/p' /tmp/rabbit-test.log > docs/output/requeue-loop.txt
sed -n '/=== returned message ===/,/routingKey/p' /tmp/rabbit-test.log > docs/output/unroutable.txt
sed -n '/=== redeclaring orders.ttl/,+1p' /tmp/rabbit-test.log > docs/output/precondition-failed.txt
grep -E 'Tests run:.*in com\.ankurm' /tmp/rabbit-test.log | sed 's/^\[INFO\] //' > docs/output/tests.txt
echo "regenerated:"; ls -1 docs/output/

View File

@@ -0,0 +1,36 @@
package com.ankurm.rabbit;
import org.springframework.amqp.support.converter.JacksonJsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Spring Boot does <b>not</b> auto-configure a JSON message converter for RabbitMQ. The default
* is {@code SimpleMessageConverter}, which handles {@code String}, {@code byte[]} and
* {@code Serializable} and nothing else:
*
* <pre>
* IllegalArgumentException: SimpleMessageConverter only supports String, byte[] and
* Serializable payloads, received: com.ankurm.rabbit.OrderMessage
* </pre>
*
* <p>Declaring one {@code MessageConverter} bean fixes both directions &mdash; the auto-configured
* {@code RabbitTemplate} and the listener container factory both pick it up.
*
* <p>Note the class name. Spring AMQP 4.1 ships {@code Jackson2JsonMessageConverter} (Jackson 2)
* and {@code JacksonJsonMessageConverter} (Jackson 3) side by side, exactly as Spring Kafka
* ships {@code JsonSerializer} and {@code JacksonJsonSerializer}. Boot 4 is a Jackson 3
* application; pick the one without the 2.
*
* @see <a href="../../../../../docs/01-the-on-ramp.md">docs/01-the-on-ramp.md</a>
*/
@Configuration(proxyBeanMethods = false)
public class ConverterConfiguration {
@Bean
MessageConverter messageConverter() {
return new JacksonJsonMessageConverter();
}
}

View File

@@ -0,0 +1,13 @@
package com.ankurm.rabbit;
/**
* The payload. Kept to a String field so the tests are about routing and acknowledgement rather
* than about converters.
*/
public record OrderMessage(String orderId, String detail) {
public static OrderMessage of(String orderId) {
return new OrderMessage(orderId, "detail for " + orderId);
}
}

View File

@@ -0,0 +1,20 @@
package com.ankurm.rabbit;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* All four AMQP 0-9-1 exchange types, a working dead-letter path, and the traps that make
* RabbitMQ feel unpredictable until you know them.
*
* <p>Everything here runs against a real broker. See the module README for how the scripts start
* one without Docker.
*/
@SpringBootApplication
public class RabbitDemoApplication {
public static void main(String[] args) {
SpringApplication.run(RabbitDemoApplication.class, args);
}
}

View File

@@ -0,0 +1,217 @@
package com.ankurm.rabbit;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.DirectExchange;
import org.springframework.amqp.core.FanoutExchange;
import org.springframework.amqp.core.HeadersExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.Map;
/**
* The whole topology as beans. {@code RabbitAdmin} declares every {@code Exchange},
* {@code Queue} and {@code Binding} bean when the connection is first opened.
*
* <p>Two things about that are worth knowing before you rely on it:
* <ul>
* <li>Declaration is <b>idempotent only if the arguments match</b>. Redeclaring an existing
* queue with different arguments is a channel-level {@code PRECONDITION_FAILED} &mdash; see
* docs/06-changing-your-mind.md.</li>
* <li>Declarations happen on connection, not on context refresh, so a topology error surfaces
* at the first publish rather than at startup.</li>
* </ul>
*
* @see <a href="../../../../../docs/02-exchanges.md">docs/02-exchanges.md</a>
*/
@Configuration(proxyBeanMethods = false)
public class Topology {
public static final String DIRECT = "orders.direct";
public static final String FANOUT = "orders.fanout";
public static final String TOPIC = "orders.topic";
public static final String HEADERS = "orders.headers";
public static final String DLX = "orders.dlx";
public static final String Q_NEW = "orders.new";
public static final String Q_CANCEL = "orders.cancel";
public static final String Q_AUDIT = "audit.all";
public static final String Q_ANALYTICS = "analytics.all";
public static final String Q_EU = "orders.eu";
public static final String Q_HIGH = "orders.high";
public static final String Q_PRIORITY = "orders.priority";
public static final String Q_ANY = "orders.any";
public static final String Q_WORK = "orders.work";
public static final String Q_TTL = "orders.ttl";
public static final String Q_DLQ = "orders.dlq";
// --- direct: routing key must match the binding key exactly -----------------------------
@Bean
DirectExchange directExchange() {
return new DirectExchange(DIRECT);
}
@Bean
Queue newQueue() {
return QueueBuilder.durable(Q_NEW).build();
}
@Bean
Queue cancelQueue() {
return QueueBuilder.durable(Q_CANCEL).build();
}
@Bean
Binding bindNew(DirectExchange directExchange, Queue newQueue) {
return BindingBuilder.bind(newQueue).to(directExchange).with("new");
}
@Bean
Binding bindCancel(DirectExchange directExchange, Queue cancelQueue) {
return BindingBuilder.bind(cancelQueue).to(directExchange).with("cancel");
}
// --- fanout: routing key ignored entirely -----------------------------------------------
@Bean
FanoutExchange fanoutExchange() {
return new FanoutExchange(FANOUT);
}
@Bean
Queue auditQueue() {
return QueueBuilder.durable(Q_AUDIT).build();
}
@Bean
Queue analyticsQueue() {
return QueueBuilder.durable(Q_ANALYTICS).build();
}
@Bean
Binding bindAudit(FanoutExchange fanoutExchange, Queue auditQueue) {
return BindingBuilder.bind(auditQueue).to(fanoutExchange);
}
@Bean
Binding bindAnalytics(FanoutExchange fanoutExchange, Queue analyticsQueue) {
return BindingBuilder.bind(analyticsQueue).to(fanoutExchange);
}
// --- topic: '*' is exactly one word, '#' is zero or more -------------------------------
@Bean
TopicExchange topicExchange() {
return new TopicExchange(TOPIC);
}
@Bean
Queue euQueue() {
return QueueBuilder.durable(Q_EU).build();
}
@Bean
Queue highQueue() {
return QueueBuilder.durable(Q_HIGH).build();
}
@Bean
Binding bindEu(TopicExchange topicExchange, Queue euQueue) {
return BindingBuilder.bind(euQueue).to(topicExchange).with("order.eu.*");
}
@Bean
Binding bindHigh(TopicExchange topicExchange, Queue highQueue) {
return BindingBuilder.bind(highQueue).to(topicExchange).with("order.#.high");
}
// --- headers: routing key ignored, header map matched ------------------------------------
@Bean
HeadersExchange headersExchange() {
return new HeadersExchange(HEADERS);
}
@Bean
Queue priorityQueue() {
return QueueBuilder.durable(Q_PRIORITY).build();
}
@Bean
Queue anyQueue() {
return QueueBuilder.durable(Q_ANY).build();
}
/** x-match=all: every header must match. */
@Bean
Binding bindPriority(HeadersExchange headersExchange, Queue priorityQueue) {
return BindingBuilder.bind(priorityQueue).to(headersExchange)
.whereAll(Map.of("priority", "high", "region", "eu")).match();
}
/** x-match=any: one is enough. */
@Bean
Binding bindAny(HeadersExchange headersExchange, Queue anyQueue) {
return BindingBuilder.bind(anyQueue).to(headersExchange)
.whereAny(Map.of("priority", "high", "region", "eu")).match();
}
// --- the dead-letter path ----------------------------------------------------------------
@Bean
DirectExchange deadLetterExchange() {
return new DirectExchange(DLX);
}
@Bean
Queue deadLetterQueue() {
// Deliberately NO dead-letter-exchange on the DLQ itself. Pointing a DLQ's DLX back at
// the exchange that feeds it is the classic infinite loop.
return QueueBuilder.durable(Q_DLQ).build();
}
@Bean
Binding bindDlq(DirectExchange deadLetterExchange, Queue deadLetterQueue) {
// The routing key used when dead-lettering is the message's ORIGINAL routing key unless
// deadLetterRoutingKey() overrides it. Both work queues below set it explicitly to
// "failed", so this one binding catches everything.
return BindingBuilder.bind(deadLetterQueue).to(deadLetterExchange).with("failed");
}
@Bean
Queue workQueue() {
return QueueBuilder.durable(Q_WORK)
.deadLetterExchange(DLX)
.deadLetterRoutingKey("failed")
.build();
}
@Bean
Queue ttlQueue() {
return QueueBuilder.durable(Q_TTL)
.ttl(1500)
.deadLetterExchange(DLX)
.deadLetterRoutingKey("failed")
.build();
}
}

View File

@@ -0,0 +1,31 @@
spring:
application:
name: rabbitmq-exchanges-and-dlq
main:
banner-mode: off
rabbitmq:
host: localhost
port: 5672
username: guest
password: guest
# BOTH of these are required to find out about an unroutable message, and they are set in
# two different places. publisher-returns puts the connection factory into a mode where it
# listens for returns at all; mandatory asks the broker to return this particular publish.
# Setting only mandatory - which is what most examples show - changes nothing at all.
# See docs/03-the-silent-drop.md.
publisher-returns: true
template:
mandatory: true
listener:
simple:
# MANUAL means nothing is removed from the queue until your code says so - and that
# basicNack(requeue=false) is what feeds the dead-letter exchange.
acknowledge-mode: manual
prefetch: 1
# Listener containers are not started automatically here; each test starts the one it
# needs, so that a redelivery loop in one test cannot consume another test's messages.
auto-startup: false
logging:
level:
root: WARN
com.ankurm: INFO

View File

@@ -0,0 +1,143 @@
package com.ankurm.rabbit;
import com.rabbitmq.client.GetResponse;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.amqp.rabbit.core.RabbitAdmin;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The dead-letter path, driven with the AMQP primitives directly rather than through a listener
* container. {@code basicGet} + {@code basicNack} is what a container does underneath, and doing
* it by hand removes all the timing from the test.
*
* @see <a href="../../../../../docs/04-dead-lettering.md">docs/04-dead-lettering.md</a>
*/
@SpringBootTest
class DeadLetterTest {
@Autowired
RabbitTemplate template;
@Autowired
RabbitAdmin admin;
@BeforeEach
void drain() {
List.of(Topology.Q_WORK, Topology.Q_TTL, Topology.Q_DLQ)
.forEach((q) -> this.admin.purgeQueue(q, false));
}
private int depth(String queue) {
Properties properties = this.admin.getQueueProperties(queue);
return (properties == null) ? -1
: ((Number) properties.get(RabbitAdmin.QUEUE_MESSAGE_COUNT)).intValue();
}
private int settled(String queue, int expected) throws InterruptedException {
for (int i = 0; i < 150; i++) {
if (depth(queue) == expected) {
return expected;
}
Thread.sleep(20);
}
return depth(queue);
}
@Test
void nackWithoutRequeueDeadLettersAndStampsTheReason() throws Exception {
this.template.convertAndSend(Topology.DIRECT, "ignored", OrderMessage.of("o-1"),
(m) -> m, null);
// Publish straight to the work queue via the default exchange: the empty exchange name
// routes by queue name, which is the one piece of AMQP that behaves like a magic
// constant and is worth knowing.
this.template.convertAndSend("", Topology.Q_WORK, OrderMessage.of("o-1"));
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
this.template.execute((channel) -> {
GetResponse response = channel.basicGet(Topology.Q_WORK, false);
assertThat(response).isNotNull();
// requeue=false is the entire dead-letter trigger. There is no separate "send to
// DLQ" call in AMQP.
channel.basicNack(response.getEnvelope().getDeliveryTag(), false, false);
return null;
});
assertThat(settled(Topology.Q_DLQ, 1)).isEqualTo(1);
assertThat(depth(Topology.Q_WORK)).isZero();
Map<String, Object> death = firstDeath(Topology.Q_DLQ);
System.out.println("=== x-death after basicNack(requeue=false) ===");
death.forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
assertThat(death).containsEntry("reason", "rejected");
assertThat(death).containsEntry("queue", Topology.Q_WORK);
assertThat(((Number) death.get("count")).intValue()).isEqualTo(1);
}
@Test
void messageTtlExpiryAlsoDeadLettersButWithADifferentReason() throws Exception {
this.template.convertAndSend("", Topology.Q_TTL, OrderMessage.of("o-ttl"));
// orders.ttl carries x-message-ttl=1500. Nobody consumes it; the broker expires it.
assertThat(settled(Topology.Q_DLQ, 1)).isEqualTo(1);
Map<String, Object> death = firstDeath(Topology.Q_DLQ);
System.out.println("=== x-death after x-message-ttl expiry ===");
death.forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
// Same destination, different reason. This is the field that tells an operator whether
// the consumer rejected the work or never got to it.
assertThat(death).containsEntry("reason", "expired");
assertThat(death).containsEntry("queue", Topology.Q_TTL);
}
@Test
void nackWithRequeueIsAnInfiniteLoopAndNeverDeadLetters() throws Exception {
this.template.convertAndSend("", Topology.Q_WORK, OrderMessage.of("o-loop"));
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
int redeliveries = this.template.execute((channel) -> {
int count = 0;
for (int i = 0; i < 200; i++) {
GetResponse response = channel.basicGet(Topology.Q_WORK, false);
if (response == null) {
break;
}
if (response.getEnvelope().isRedeliver()) {
count++;
}
// requeue=true puts it back. The dead-letter exchange is never consulted,
// x-death is never written, and the loop has no counter to exhaust.
channel.basicNack(response.getEnvelope().getDeliveryTag(), false, true);
}
return count;
});
System.out.println("=== basicNack(requeue=true), 200 attempts ===");
System.out.println(" redelivered " + redeliveries + " times");
System.out.println(" dead-lettered " + depth(Topology.Q_DLQ));
System.out.println(" still on queue " + depth(Topology.Q_WORK));
assertThat(redeliveries).isGreaterThan(150);
// The message is exactly where it started, having been processed 200 times.
assertThat(depth(Topology.Q_DLQ)).isZero();
assertThat(settled(Topology.Q_WORK, 1)).isEqualTo(1);
}
@SuppressWarnings("unchecked")
private Map<String, Object> firstDeath(String queue) {
var message = this.template.receive(queue, 5000);
assertThat(message).isNotNull();
List<Map<String, Object>> deaths =
(List<Map<String, Object>>) message.getMessageProperties().getHeader("x-death");
assertThat(deaths).isNotNull().isNotEmpty();
return deaths.get(0);
}
}

View File

@@ -0,0 +1,73 @@
package com.ankurm.rabbit;
import com.rabbitmq.client.GetResponse;
import org.junit.jupiter.api.Test;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.DirectExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.rabbit.core.RabbitAdmin;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The third dead-letter trigger: a queue that is full.
*
* <p>Included because the article states all three triggers in one table, and this was the only
* row not produced by a run. It is now.
*
* @see <a href="../../../../../docs/04-dead-lettering.md">docs/04-dead-lettering.md</a>
*/
@SpringBootTest
class MaxLengthTest {
static final String Q_BOUNDED = "orders.bounded";
@Autowired
RabbitTemplate template;
@Autowired
RabbitAdmin admin;
@Test
@SuppressWarnings("unchecked")
void exceedingMaxLengthDeadLettersTheOldestWithReasonMaxlen() throws Exception {
this.admin.deleteQueue(Q_BOUNDED);
Queue bounded = QueueBuilder.durable(Q_BOUNDED)
.maxLength(2)
.deadLetterExchange(Topology.DLX)
.deadLetterRoutingKey("failed")
.build();
this.admin.declareQueue(bounded);
Binding binding = BindingBuilder.bind(bounded)
.to(new DirectExchange(Topology.DIRECT)).with("bounded");
this.admin.declareBinding(binding);
this.admin.purgeQueue(Topology.Q_DLQ, false);
// Three messages into a queue that holds two. RabbitMQ drops from the HEAD, so the
// FIRST message is the one dead-lettered - the oldest, not the newest.
for (String id : List.of("m-1", "m-2", "m-3")) {
this.template.convertAndSend(Topology.DIRECT, "bounded", OrderMessage.of(id));
}
var message = this.template.receive(Topology.Q_DLQ, 10_000);
assertThat(message).isNotNull();
List<Map<String, Object>> deaths =
(List<Map<String, Object>>) message.getMessageProperties().getHeader("x-death");
System.out.println("=== x-death after x-max-length overflow ===");
deaths.get(0).forEach((k, v) -> System.out.printf(" %-16s %s%n", k, v));
System.out.println(" body " + new String(message.getBody()));
assertThat(deaths.get(0)).containsEntry("reason", "maxlen");
assertThat(deaths.get(0)).containsEntry("queue", Q_BOUNDED);
assertThat(new String(message.getBody())).contains("m-1");
}
}

View File

@@ -0,0 +1,152 @@
package com.ankurm.rabbit;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.amqp.core.MessageProperties;
import org.springframework.amqp.rabbit.core.RabbitAdmin;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import static org.assertj.core.api.Assertions.assertThat;
/**
* All four exchange types, driven with real publishes and counted by draining the queues.
*
* @see <a href="../../../../../docs/02-exchanges.md">docs/02-exchanges.md</a>
*/
@SpringBootTest
class RoutingTest {
@Autowired
RabbitTemplate template;
@Autowired
RabbitAdmin admin;
private static final List<String> QUEUES = List.of(Topology.Q_NEW, Topology.Q_CANCEL,
Topology.Q_AUDIT, Topology.Q_ANALYTICS, Topology.Q_EU, Topology.Q_HIGH,
Topology.Q_PRIORITY, Topology.Q_ANY);
@BeforeEach
void drain() {
QUEUES.forEach((q) -> this.admin.purgeQueue(q, false));
}
/**
* Ready message count, straight from the broker.
*
* <p>Note the {@code Number}: {@code RabbitAdmin.QUEUE_MESSAGE_COUNT} holds a {@code Long}
* in Spring AMQP 4.1. Casting it to {@code Integer}, as every older example does, is a
* {@code ClassCastException} at runtime.
*/
private int depth(String queue) {
Properties properties = this.admin.getQueueProperties(queue);
return (properties == null) ? -1
: ((Number) properties.get(RabbitAdmin.QUEUE_MESSAGE_COUNT)).intValue();
}
/** Publishing is asynchronous; give the broker a moment to route before counting. */
private int settledDepth(String queue) {
int last = -1;
for (int i = 0; i < 50; i++) {
last = depth(queue);
if (last > 0) {
return last;
}
try {
Thread.sleep(20);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
return last;
}
@Test
void directExchangeMatchesTheRoutingKeyExactly() {
this.template.convertAndSend(Topology.DIRECT, "new", OrderMessage.of("o-1"));
this.template.convertAndSend(Topology.DIRECT, "cancel", OrderMessage.of("o-2"));
// No binding for "amend". The broker discards it - see UnroutableTest.
this.template.convertAndSend(Topology.DIRECT, "amend", OrderMessage.of("o-3"));
assertThat(settledDepth(Topology.Q_NEW)).isEqualTo(1);
assertThat(settledDepth(Topology.Q_CANCEL)).isEqualTo(1);
}
@Test
void fanoutIgnoresTheRoutingKeyCompletely() {
this.template.convertAndSend(Topology.FANOUT, "this-is-ignored", OrderMessage.of("o-1"));
assertThat(settledDepth(Topology.Q_AUDIT)).isEqualTo(1);
assertThat(settledDepth(Topology.Q_ANALYTICS)).isEqualTo(1);
}
@Test
void topicWildcardsAreWordsNotCharacters() {
// binding "order.eu.*" - exactly one word after order.eu
// binding "order.#.high" - zero or more words between order. and .high
Map<String, String> routingKeys = new LinkedHashMap<>();
routingKeys.put("order.eu.high", "matches both");
routingKeys.put("order.eu.low", "matches order.eu.* only");
routingKeys.put("order.us.high", "matches order.#.high only");
routingKeys.put("order.eu.west.high", "matches order.#.high only - * is one word");
routingKeys.put("order.high", "matches order.#.high - # can be zero words");
System.out.println("=== topic exchange ===");
System.out.printf("%-24s %-12s %-12s %s%n", "routing key", "orders.eu", "orders.high", "note");
for (Map.Entry<String, String> entry : routingKeys.entrySet()) {
this.admin.purgeQueue(Topology.Q_EU, false);
this.admin.purgeQueue(Topology.Q_HIGH, false);
this.template.convertAndSend(Topology.TOPIC, entry.getKey(), OrderMessage.of("o"));
System.out.printf("%-24s %-12s %-12s %s%n", entry.getKey(), settledDepth(Topology.Q_EU) == 1,
settledDepth(Topology.Q_HIGH) == 1, entry.getValue());
}
this.admin.purgeQueue(Topology.Q_EU, false);
this.admin.purgeQueue(Topology.Q_HIGH, false);
// The one everybody gets wrong: '*' is one WORD, so it does not match two.
this.template.convertAndSend(Topology.TOPIC, "order.eu.west.high", OrderMessage.of("o"));
assertThat(depth(Topology.Q_EU)).isEqualTo(0);
assertThat(settledDepth(Topology.Q_HIGH)).isEqualTo(1);
this.admin.purgeQueue(Topology.Q_HIGH, false);
// And '#' really does match zero words.
this.template.convertAndSend(Topology.TOPIC, "order.high", OrderMessage.of("o"));
assertThat(settledDepth(Topology.Q_HIGH)).isEqualTo(1);
}
@Test
void headersExchangeMatchesTheHeaderMapNotTheRoutingKey() {
send(Map.of("priority", "high", "region", "eu"));
assertThat(settledDepth(Topology.Q_PRIORITY)).isEqualTo(1); // x-match=all: both present
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1); // x-match=any: either is enough
drain();
send(Map.of("priority", "high"));
assertThat(depth(Topology.Q_PRIORITY)).isEqualTo(0); // all: region missing
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1);
drain();
send(Map.of("priority", "high", "region", "us"));
// x-match=all needs every header to match by VALUE, not merely to be present.
assertThat(depth(Topology.Q_PRIORITY)).isEqualTo(0);
assertThat(settledDepth(Topology.Q_ANY)).isEqualTo(1);
}
private void send(Map<String, Object> headers) {
this.template.convertAndSend(Topology.HEADERS, "routing-key-is-ignored",
OrderMessage.of("o-1"), (message) -> {
MessageProperties properties = message.getMessageProperties();
headers.forEach(properties::setHeader);
return message;
});
}
}

View File

@@ -0,0 +1,33 @@
package com.ankurm.rabbit;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.context.annotation.Bean;
import org.testcontainers.rabbitmq.RabbitMQContainer;
import org.testcontainers.utility.DockerImageName;
/**
* The Testcontainers route, for machines that have a Docker daemon.
*
* <p>{@code @ServiceConnection} supplies host, port, username and password to the
* auto-configuration, so no {@code spring.rabbitmq.*} property and no
* {@code @DynamicPropertySource} block is needed.
*
* <p>The Maven coordinate is <b>{@code org.testcontainers:testcontainers-rabbitmq}</b>.
* Testcontainers 2.x prefixed every module artifact; the old {@code org.testcontainers:rabbitmq}
* stopped at 1.21.4 and is not managed by the Boot 4.1 BOM.
*
* <p>The committed transcripts under {@code docs/output/} came from a broker started by
* {@code scripts/broker.sh} instead, on a machine with no Docker &mdash; see the module README
* for why that matters and what version it was.
*/
@TestConfiguration(proxyBeanMethods = false)
public class TestcontainersConfiguration {
@Bean
@ServiceConnection
RabbitMQContainer rabbitContainer() {
return new RabbitMQContainer(DockerImageName.parse("rabbitmq:4.1-management"));
}
}

View File

@@ -0,0 +1,100 @@
package com.ankurm.rabbit;
import org.junit.jupiter.api.Test;
import org.springframework.amqp.AmqpIOException;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.core.ReturnedMessage;
import org.springframework.amqp.rabbit.core.RabbitAdmin;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatExceptionOfType;
/**
* Two ways a RabbitMQ topology loses your work quietly.
*
* @see <a href="../../../../../docs/03-the-silent-drop.md">docs/03-the-silent-drop.md</a>
* @see <a href="../../../../../docs/06-changing-your-mind.md">docs/06-changing-your-mind.md</a>
*/
@SpringBootTest
class TopologyTrapsTest {
@Autowired
RabbitTemplate template;
@Autowired
RabbitAdmin admin;
@Test
void anUnroutableMessageIsDiscardedUnlessYouAskForItBack() throws Exception {
List<ReturnedMessage> returned = new CopyOnWriteArrayList<>();
this.template.setReturnsCallback(returned::add);
// "amend" matches no binding on orders.direct. The broker has nowhere to put it.
this.template.convertAndSend(Topology.DIRECT, "amend", OrderMessage.of("o-1"));
for (int i = 0; i < 100 && returned.isEmpty(); i++) {
Thread.sleep(20);
}
// spring.rabbitmq.template.mandatory=true is what turns a silent discard into a return.
assertThat(this.template.isMandatoryFor(null)).isTrue();
assertThat(returned).hasSize(1);
ReturnedMessage message = returned.get(0);
System.out.println("=== returned message ===");
System.out.println(" replyCode " + message.getReplyCode());
System.out.println(" replyText " + message.getReplyText());
System.out.println(" exchange " + message.getExchange());
System.out.println(" routingKey " + message.getRoutingKey());
assertThat(message.getReplyCode()).isEqualTo(312);
assertThat(message.getReplyText()).isEqualTo("NO_ROUTE");
assertThat(message.getRoutingKey()).isEqualTo("amend");
// Note what did NOT happen: convertAndSend returned normally. Publishing is fire and
// forget at the protocol level, so even with mandatory=true the failure arrives
// asynchronously on another thread. Nothing throws.
}
@Test
void redeclaringAQueueWithDifferentArgumentsIsAPreconditionFailure() {
// orders.ttl already exists with x-message-ttl=1500. Same name, different argument.
Queue conflicting = QueueBuilder.durable(Topology.Q_TTL)
.ttl(9999)
.deadLetterExchange(Topology.DLX)
.deadLetterRoutingKey("failed")
.build();
assertThatExceptionOfType(AmqpIOException.class)
.isThrownBy(() -> this.admin.declareQueue(conflicting))
.satisfies((ex) -> {
String detail = rootMessage(ex);
System.out.println("=== redeclaring orders.ttl with x-message-ttl=9999 ===");
System.out.println(" " + detail);
assertThat(detail).contains("PRECONDITION_FAILED")
.contains("inequivalent arg 'x-message-ttl'");
});
// Queue arguments are immutable. There is no ALTER QUEUE. Changing a TTL, a max-length
// or a dead-letter exchange on an existing queue means: declare a new queue, move the
// consumers, drain the old one, delete it. Plan the rename into the change.
}
private static String rootMessage(Throwable throwable) {
// The useful text is on the cause. AmqpIOException's own message is just
// "java.io.IOException", which is why this failure is so often reported as "IOException"
// with no further detail.
Throwable current = throwable;
StringBuilder all = new StringBuilder();
while (current != null) {
all.append(current.getMessage()).append(" | ");
current = current.getCause();
}
return all.toString();
}
}