Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.
Demonstrations
_01_idempotence verifies idempotence from the BROKER via Admin.describeProducers rather
than by reading config back: a default producer registers one producer
state, an acks=1 producer registers zero. Also covers the ConfigException
you get only when idempotence is requested explicitly.
_02_transactions transaction markers consuming offsets, aborted records surviving in the
log, read_committed vs read_uncommitted, and a correction: markers do NOT
inflate consumer lag (a drained consumer reaches zero), they break
counting records by offset arithmetic (measured 100% overstatement).
_03_read_process_write
the Spring loop with container-managed transactions; a listener that
fails after sending runs twice and is committed once.
_04_fencing shared transactional.id fences the zombie; a random per-startup id lets
both producers commit and silently defeats the guarantee entirely.
_05_database Kafka EOS does not extend to a database. Naive listener writes 2 rows for
1 message; the same listener keyed on topic-partition-offset writes 1.
_06_performance cost per transaction size, median of 5 interleaved rounds.
Docs
Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
outbox pattern, performance, and corner cases.
Measured highlights
acks=1 producer 0 broker-tracked producer states (idempotence silently off)
5 records in one tx endOffset 6; offset span overstates record count by 100%
1 record per tx 711 records/s
100 records per tx 78,770 records/s (~111x)
This commit is contained in:
111
README.md
Normal file
111
README.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Exactly-once with Spring Kafka on Spring Boot 4
|
||||
|
||||
Companion repository for **[Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions](https://ankurm.com/spring-kafka-exactly-once-boot-4/)**
|
||||
on [ankurm.com](https://ankurm.com).
|
||||
|
||||
The blog post covers the concepts most people need. **This repository covers everything** — every
|
||||
demonstration, every failure mode, every corner case, as runnable tests with real captured output.
|
||||
|
||||
Verified against **Spring Boot 4.1.0**, **Spring Kafka 4.1.0**, **kafka-clients 4.2.1**, **JDK 25.0.3**.
|
||||
Raw output of the full suite is in [`results/full-run.txt`](results/full-run.txt).
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker.
|
||||
|
||||
```console
|
||||
git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git
|
||||
cd spring-kafka-exactly-once-boot4
|
||||
mvn test
|
||||
```
|
||||
|
||||
15 tests, about three minutes, mostly the performance sweep.
|
||||
|
||||
Run one demonstration at a time:
|
||||
|
||||
```console
|
||||
mvn test -Dtest=IdempotentProducerTest
|
||||
mvn test -Dtest=TransactionMarkerOffsetTest
|
||||
mvn test -Dtest=ZombieFencingTest
|
||||
mvn test -Dtest=DatabaseBoundaryTest
|
||||
mvn test -Dtest=TransactionCostTest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The demonstrations
|
||||
|
||||
| Test | Shows | Key measured result |
|
||||
|---|---|---|
|
||||
| [`IdempotentProducerTest`](src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) | Idempotence verified **from the broker** via `describeProducers`, and the config that silently disables it | default producer → 1 tracked producer state; `acks=1` → **0** |
|
||||
| [`TransactionMarkerOffsetTest`](src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java) | Transaction markers, aborted records, `read_committed` vs `read_uncommitted`, offset arithmetic | 5 records → `endOffset` 6; offset span overstates record count by **100%** |
|
||||
| [`ReadProcessWriteTest`](src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) | The Spring read-process-write loop, and rollback + redelivery | listener ran **2x**, `read_committed` consumer sees **1** |
|
||||
| [`ZombieFencingTest`](src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) | What `transactional.id` is for, and how a random one destroys the guarantee | shared id → `ProducerFencedException`; random id → **both commit** |
|
||||
| [`DatabaseBoundaryTest`](src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) | Kafka EOS does not cover your database, and the fix | naive listener → **2 rows**; idempotent listener → **1 row** |
|
||||
| [`TransactionCostTest`](src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) | What transactions cost, and the setting that controls it | 1 record/tx = **711 rec/s**; 100/tx = **78,770 rec/s** |
|
||||
|
||||
## The documentation
|
||||
|
||||
| Chapter | Covers |
|
||||
|---|---|
|
||||
| [1. Boot 4 setup](docs/01-boot4-setup.md) | The `spring-boot-starter-kafka` trap, the one property that enables EOS, testing without Docker |
|
||||
| [2. Idempotence](docs/02-idempotence.md) | PIDs and sequence numbers, the `acks=1` silent downgrade, the constraints |
|
||||
| [3. Transactions and markers](docs/03-transactions-and-markers.md) | `sendOffsetsToTransaction`, isolation levels, control records, the Last Stable Offset |
|
||||
| [4. Fencing](docs/04-fencing.md) | `transactional.id`, epochs, EOSMode V2, KIP-890 on Kafka 4.x |
|
||||
| [5. The database boundary](docs/05-database-boundary.md) | Why there is no 2PC, deprecated `ChainedKafkaTransactionManager`, dedupe keys, the outbox pattern |
|
||||
| [6. Performance](docs/06-performance.md) | Measured cost per transaction size, and why `max.poll.records` is the knob |
|
||||
| [7. Corner cases](docs/07-corner-cases.md) | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams |
|
||||
|
||||
---
|
||||
|
||||
## Six findings you will not read elsewhere
|
||||
|
||||
1. **Boot 4 moved Kafka auto-configuration out of `spring-boot-autoconfigure`.** Depend on
|
||||
`spring-kafka` alone and you get a `NoSuchBeanDefinitionException` for `KafkaTemplate` that looks
|
||||
like a generics problem. Use `spring-boot-starter-kafka`.
|
||||
→ [docs/01](docs/01-boot4-setup.md)
|
||||
2. **`acks=1` silently disables idempotence — proven from the broker, not the config.**
|
||||
`Admin.describeProducers()` shows 1 tracked producer state for a default producer and **0** for an
|
||||
`acks=1` producer. Setting `enable.idempotence=true` explicitly turns that silent downgrade into
|
||||
a startup failure.
|
||||
→ [docs/02](docs/02-idempotence.md)
|
||||
3. **Transactions do *not* inflate consumer lag.** This widely repeated claim is wrong: a drained
|
||||
`read_committed` consumer reaches exactly zero lag, because its position advances over markers
|
||||
too. What markers *do* break is counting records by offset arithmetic — measured here at **100%**
|
||||
overstatement.
|
||||
→ [docs/03](docs/03-transactions-and-markers.md)
|
||||
4. **A random `transactional.id` per start-up silently defeats fencing.** Measured: two producers
|
||||
with different UUIDs both commit, and nothing is fenced. Everything appears to work while
|
||||
delivering exactly the duplicate processing you enabled transactions to prevent.
|
||||
→ [docs/04](docs/04-fencing.md)
|
||||
5. **Aborted records reach the log only if they were already flushed.** `abortTransaction()`
|
||||
discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends
|
||||
on timing. Rely on `read_committed`, not on either behaviour.
|
||||
→ [docs/03](docs/03-transactions-and-markers.md) §3.6
|
||||
6. **One record per transaction is ~111x slower than one hundred.** The cost is per commit, not per
|
||||
record, and the setting that controls it in Spring is `max.poll.records` — not anything named
|
||||
"transaction".
|
||||
→ [docs/06](docs/06-performance.md)
|
||||
|
||||
---
|
||||
|
||||
## Reference machine
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads |
|
||||
| RAM | 15.3 GB |
|
||||
| OS | Windows 11 Home |
|
||||
| JDK | `java 25.0.3+9-LTS-195` |
|
||||
| Kafka | in-JVM `EmbeddedKafkaKraftBroker`, kafka-clients 4.2.1, `transaction.version=2` |
|
||||
|
||||
A laptop and a single-node broker. The absolute throughput figures are illustrative; the
|
||||
relationships are what generalise. See [docs/06](docs/06-performance.md) §6.3 for what to trust.
|
||||
|
||||
---
|
||||
|
||||
## Licence
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
Reference in New Issue
Block a user