Ankur
e23ca359eb
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.
Demonstrations
_01_idempotence verifies idempotence from the BROKER via Admin.describeProducers rather
than by reading config back: a default producer registers one producer
state, an acks=1 producer registers zero. Also covers the ConfigException
you get only when idempotence is requested explicitly.
_02_transactions transaction markers consuming offsets, aborted records surviving in the
log, read_committed vs read_uncommitted, and a correction: markers do NOT
inflate consumer lag (a drained consumer reaches zero), they break
counting records by offset arithmetic (measured 100% overstatement).
_03_read_process_write
the Spring loop with container-managed transactions; a listener that
fails after sending runs twice and is committed once.
_04_fencing shared transactional.id fences the zombie; a random per-startup id lets
both producers commit and silently defeats the guarantee entirely.
_05_database Kafka EOS does not extend to a database. Naive listener writes 2 rows for
1 message; the same listener keyed on topic-partition-offset writes 1.
_06_performance cost per transaction size, median of 5 interleaved rounds.
Docs
Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
outbox pattern, performance, and corner cases.
Measured highlights
acks=1 producer 0 broker-tracked producer states (idempotence silently off)
5 records in one tx endOffset 6; offset span overstates record count by 100%
1 record per tx 711 records/s
100 records per tx 78,770 records/s (~111x)
112 lines
6.1 KiB
Markdown
112 lines
6.1 KiB
Markdown
# Exactly-once with Spring Kafka on Spring Boot 4
|
|
|
|
Companion repository for **[Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions](https://ankurm.com/spring-kafka-exactly-once-boot-4/)**
|
|
on [ankurm.com](https://ankurm.com).
|
|
|
|
The blog post covers the concepts most people need. **This repository covers everything** — every
|
|
demonstration, every failure mode, every corner case, as runnable tests with real captured output.
|
|
|
|
Verified against **Spring Boot 4.1.0**, **Spring Kafka 4.1.0**, **kafka-clients 4.2.1**, **JDK 25.0.3**.
|
|
Raw output of the full suite is in [`results/full-run.txt`](results/full-run.txt).
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker.
|
|
|
|
```console
|
|
git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git
|
|
cd spring-kafka-exactly-once-boot4
|
|
mvn test
|
|
```
|
|
|
|
15 tests, about three minutes, mostly the performance sweep.
|
|
|
|
Run one demonstration at a time:
|
|
|
|
```console
|
|
mvn test -Dtest=IdempotentProducerTest
|
|
mvn test -Dtest=TransactionMarkerOffsetTest
|
|
mvn test -Dtest=ZombieFencingTest
|
|
mvn test -Dtest=DatabaseBoundaryTest
|
|
mvn test -Dtest=TransactionCostTest
|
|
```
|
|
|
|
---
|
|
|
|
## The demonstrations
|
|
|
|
| Test | Shows | Key measured result |
|
|
|---|---|---|
|
|
| [`IdempotentProducerTest`](src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) | Idempotence verified **from the broker** via `describeProducers`, and the config that silently disables it | default producer → 1 tracked producer state; `acks=1` → **0** |
|
|
| [`TransactionMarkerOffsetTest`](src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java) | Transaction markers, aborted records, `read_committed` vs `read_uncommitted`, offset arithmetic | 5 records → `endOffset` 6; offset span overstates record count by **100%** |
|
|
| [`ReadProcessWriteTest`](src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) | The Spring read-process-write loop, and rollback + redelivery | listener ran **2x**, `read_committed` consumer sees **1** |
|
|
| [`ZombieFencingTest`](src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) | What `transactional.id` is for, and how a random one destroys the guarantee | shared id → `ProducerFencedException`; random id → **both commit** |
|
|
| [`DatabaseBoundaryTest`](src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) | Kafka EOS does not cover your database, and the fix | naive listener → **2 rows**; idempotent listener → **1 row** |
|
|
| [`TransactionCostTest`](src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) | What transactions cost, and the setting that controls it | 1 record/tx = **711 rec/s**; 100/tx = **78,770 rec/s** |
|
|
|
|
## The documentation
|
|
|
|
| Chapter | Covers |
|
|
|---|---|
|
|
| [1. Boot 4 setup](docs/01-boot4-setup.md) | The `spring-boot-starter-kafka` trap, the one property that enables EOS, testing without Docker |
|
|
| [2. Idempotence](docs/02-idempotence.md) | PIDs and sequence numbers, the `acks=1` silent downgrade, the constraints |
|
|
| [3. Transactions and markers](docs/03-transactions-and-markers.md) | `sendOffsetsToTransaction`, isolation levels, control records, the Last Stable Offset |
|
|
| [4. Fencing](docs/04-fencing.md) | `transactional.id`, epochs, EOSMode V2, KIP-890 on Kafka 4.x |
|
|
| [5. The database boundary](docs/05-database-boundary.md) | Why there is no 2PC, deprecated `ChainedKafkaTransactionManager`, dedupe keys, the outbox pattern |
|
|
| [6. Performance](docs/06-performance.md) | Measured cost per transaction size, and why `max.poll.records` is the knob |
|
|
| [7. Corner cases](docs/07-corner-cases.md) | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams |
|
|
|
|
---
|
|
|
|
## Six findings you will not read elsewhere
|
|
|
|
1. **Boot 4 moved Kafka auto-configuration out of `spring-boot-autoconfigure`.** Depend on
|
|
`spring-kafka` alone and you get a `NoSuchBeanDefinitionException` for `KafkaTemplate` that looks
|
|
like a generics problem. Use `spring-boot-starter-kafka`.
|
|
→ [docs/01](docs/01-boot4-setup.md)
|
|
2. **`acks=1` silently disables idempotence — proven from the broker, not the config.**
|
|
`Admin.describeProducers()` shows 1 tracked producer state for a default producer and **0** for an
|
|
`acks=1` producer. Setting `enable.idempotence=true` explicitly turns that silent downgrade into
|
|
a startup failure.
|
|
→ [docs/02](docs/02-idempotence.md)
|
|
3. **Transactions do *not* inflate consumer lag.** This widely repeated claim is wrong: a drained
|
|
`read_committed` consumer reaches exactly zero lag, because its position advances over markers
|
|
too. What markers *do* break is counting records by offset arithmetic — measured here at **100%**
|
|
overstatement.
|
|
→ [docs/03](docs/03-transactions-and-markers.md)
|
|
4. **A random `transactional.id` per start-up silently defeats fencing.** Measured: two producers
|
|
with different UUIDs both commit, and nothing is fenced. Everything appears to work while
|
|
delivering exactly the duplicate processing you enabled transactions to prevent.
|
|
→ [docs/04](docs/04-fencing.md)
|
|
5. **Aborted records reach the log only if they were already flushed.** `abortTransaction()`
|
|
discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends
|
|
on timing. Rely on `read_committed`, not on either behaviour.
|
|
→ [docs/03](docs/03-transactions-and-markers.md) §3.6
|
|
6. **One record per transaction is ~111x slower than one hundred.** The cost is per commit, not per
|
|
record, and the setting that controls it in Spring is `max.poll.records` — not anything named
|
|
"transaction".
|
|
→ [docs/06](docs/06-performance.md)
|
|
|
|
---
|
|
|
|
## Reference machine
|
|
|
|
| | |
|
|
|---|---|
|
|
| CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads |
|
|
| RAM | 15.3 GB |
|
|
| OS | Windows 11 Home |
|
|
| JDK | `java 25.0.3+9-LTS-195` |
|
|
| Kafka | in-JVM `EmbeddedKafkaKraftBroker`, kafka-clients 4.2.1, `transaction.version=2` |
|
|
|
|
A laptop and a single-node broker. The absolute throughput figures are illustrative; the
|
|
relationships are what generalise. See [docs/06](docs/06-performance.md) §6.3 for what to trust.
|
|
|
|
---
|
|
|
|
## Licence
|
|
|
|
MIT. See [LICENSE](LICENSE).
|