# Exactly-once with Spring Kafka on Spring Boot 4 Companion repository for **[Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions](https://ankurm.com/spring-kafka-exactly-once-boot-4/)** on [ankurm.com](https://ankurm.com). The blog post covers the concepts most people need. **This repository covers everything** — every demonstration, every failure mode, every corner case, as runnable tests with real captured output. Verified against **Spring Boot 4.1.0**, **Spring Kafka 4.1.0**, **kafka-clients 4.2.1**, **JDK 25.0.3**. Raw output of the full suite is in [`results/full-run.txt`](results/full-run.txt). --- ## Quick start No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker. ```console git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git cd spring-kafka-exactly-once-boot4 mvn test ``` 15 tests, about three minutes, mostly the performance sweep. Run one demonstration at a time: ```console mvn test -Dtest=IdempotentProducerTest mvn test -Dtest=TransactionMarkerOffsetTest mvn test -Dtest=ZombieFencingTest mvn test -Dtest=DatabaseBoundaryTest mvn test -Dtest=TransactionCostTest ``` --- ## The demonstrations | Test | Shows | Key measured result | |---|---|---| | [`IdempotentProducerTest`](src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) | Idempotence verified **from the broker** via `describeProducers`, and the config that silently disables it | default producer → 1 tracked producer state; `acks=1` → **0** | | [`TransactionMarkerOffsetTest`](src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java) | Transaction markers, aborted records, `read_committed` vs `read_uncommitted`, offset arithmetic | 5 records → `endOffset` 6; offset span overstates record count by **100%** | | [`ReadProcessWriteTest`](src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) | The Spring read-process-write loop, and rollback + redelivery | listener ran **2x**, `read_committed` consumer sees **1** | | [`ZombieFencingTest`](src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) | What `transactional.id` is for, and how a random one destroys the guarantee | shared id → `ProducerFencedException`; random id → **both commit** | | [`DatabaseBoundaryTest`](src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) | Kafka EOS does not cover your database, and the fix | naive listener → **2 rows**; idempotent listener → **1 row** | | [`TransactionCostTest`](src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) | What transactions cost, and the setting that controls it | 1 record/tx = **711 rec/s**; 100/tx = **78,770 rec/s** | ## The documentation | Chapter | Covers | |---|---| | [1. Boot 4 setup](docs/01-boot4-setup.md) | The `spring-boot-starter-kafka` trap, the one property that enables EOS, testing without Docker | | [2. Idempotence](docs/02-idempotence.md) | PIDs and sequence numbers, the `acks=1` silent downgrade, the constraints | | [3. Transactions and markers](docs/03-transactions-and-markers.md) | `sendOffsetsToTransaction`, isolation levels, control records, the Last Stable Offset | | [4. Fencing](docs/04-fencing.md) | `transactional.id`, epochs, EOSMode V2, KIP-890 on Kafka 4.x | | [5. The database boundary](docs/05-database-boundary.md) | Why there is no 2PC, deprecated `ChainedKafkaTransactionManager`, dedupe keys, the outbox pattern | | [6. Performance](docs/06-performance.md) | Measured cost per transaction size, and why `max.poll.records` is the knob | | [7. Corner cases](docs/07-corner-cases.md) | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams | --- ## Six findings you will not read elsewhere 1. **Boot 4 moved Kafka auto-configuration out of `spring-boot-autoconfigure`.** Depend on `spring-kafka` alone and you get a `NoSuchBeanDefinitionException` for `KafkaTemplate` that looks like a generics problem. Use `spring-boot-starter-kafka`. → [docs/01](docs/01-boot4-setup.md) 2. **`acks=1` silently disables idempotence — proven from the broker, not the config.** `Admin.describeProducers()` shows 1 tracked producer state for a default producer and **0** for an `acks=1` producer. Setting `enable.idempotence=true` explicitly turns that silent downgrade into a startup failure. → [docs/02](docs/02-idempotence.md) 3. **Transactions do *not* inflate consumer lag.** This widely repeated claim is wrong: a drained `read_committed` consumer reaches exactly zero lag, because its position advances over markers too. What markers *do* break is counting records by offset arithmetic — measured here at **100%** overstatement. → [docs/03](docs/03-transactions-and-markers.md) 4. **A random `transactional.id` per start-up silently defeats fencing.** Measured: two producers with different UUIDs both commit, and nothing is fenced. Everything appears to work while delivering exactly the duplicate processing you enabled transactions to prevent. → [docs/04](docs/04-fencing.md) 5. **Aborted records reach the log only if they were already flushed.** `abortTransaction()` discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends on timing. Rely on `read_committed`, not on either behaviour. → [docs/03](docs/03-transactions-and-markers.md) §3.6 6. **One record per transaction is ~111x slower than one hundred.** The cost is per commit, not per record, and the setting that controls it in Spring is `max.poll.records` — not anything named "transaction". → [docs/06](docs/06-performance.md) --- ## Reference machine | | | |---|---| | CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads | | RAM | 15.3 GB | | OS | Windows 11 Home | | JDK | `java 25.0.3+9-LTS-195` | | Kafka | in-JVM `EmbeddedKafkaKraftBroker`, kafka-clients 4.2.1, `transaction.version=2` | A laptop and a single-node broker. The absolute throughput figures are illustrative; the relationships are what generalise. See [docs/06](docs/06-performance.md) §6.3 for what to trust. --- ## Licence MIT. See [LICENSE](LICENSE).