Ankur
e23ca359eb
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.
Demonstrations
_01_idempotence verifies idempotence from the BROKER via Admin.describeProducers rather
than by reading config back: a default producer registers one producer
state, an acks=1 producer registers zero. Also covers the ConfigException
you get only when idempotence is requested explicitly.
_02_transactions transaction markers consuming offsets, aborted records surviving in the
log, read_committed vs read_uncommitted, and a correction: markers do NOT
inflate consumer lag (a drained consumer reaches zero), they break
counting records by offset arithmetic (measured 100% overstatement).
_03_read_process_write
the Spring loop with container-managed transactions; a listener that
fails after sending runs twice and is committed once.
_04_fencing shared transactional.id fences the zombie; a random per-startup id lets
both producers commit and silently defeats the guarantee entirely.
_05_database Kafka EOS does not extend to a database. Naive listener writes 2 rows for
1 message; the same listener keyed on topic-partition-offset writes 1.
_06_performance cost per transaction size, median of 5 interleaved rounds.
Docs
Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
outbox pattern, performance, and corner cases.
Measured highlights
acks=1 producer 0 broker-tracked producer states (idempotence silently off)
5 records in one tx endOffset 6; offset span overstates record count by 100%
1 record per tx 711 records/s
100 records per tx 78,770 records/s (~111x)
Exactly-once with Spring Kafka on Spring Boot 4
Companion repository for Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions on ankurm.com.
The blog post covers the concepts most people need. This repository covers everything — every demonstration, every failure mode, every corner case, as runnable tests with real captured output.
Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3.
Raw output of the full suite is in results/full-run.txt.
Quick start
No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker.
git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git
cd spring-kafka-exactly-once-boot4
mvn test
15 tests, about three minutes, mostly the performance sweep.
Run one demonstration at a time:
mvn test -Dtest=IdempotentProducerTest
mvn test -Dtest=TransactionMarkerOffsetTest
mvn test -Dtest=ZombieFencingTest
mvn test -Dtest=DatabaseBoundaryTest
mvn test -Dtest=TransactionCostTest
The demonstrations
| Test | Shows | Key measured result |
|---|---|---|
IdempotentProducerTest |
Idempotence verified from the broker via describeProducers, and the config that silently disables it |
default producer → 1 tracked producer state; acks=1 → 0 |
TransactionMarkerOffsetTest |
Transaction markers, aborted records, read_committed vs read_uncommitted, offset arithmetic |
5 records → endOffset 6; offset span overstates record count by 100% |
ReadProcessWriteTest |
The Spring read-process-write loop, and rollback + redelivery | listener ran 2x, read_committed consumer sees 1 |
ZombieFencingTest |
What transactional.id is for, and how a random one destroys the guarantee |
shared id → ProducerFencedException; random id → both commit |
DatabaseBoundaryTest |
Kafka EOS does not cover your database, and the fix | naive listener → 2 rows; idempotent listener → 1 row |
TransactionCostTest |
What transactions cost, and the setting that controls it | 1 record/tx = 711 rec/s; 100/tx = 78,770 rec/s |
The documentation
| Chapter | Covers |
|---|---|
| 1. Boot 4 setup | The spring-boot-starter-kafka trap, the one property that enables EOS, testing without Docker |
| 2. Idempotence | PIDs and sequence numbers, the acks=1 silent downgrade, the constraints |
| 3. Transactions and markers | sendOffsetsToTransaction, isolation levels, control records, the Last Stable Offset |
| 4. Fencing | transactional.id, epochs, EOSMode V2, KIP-890 on Kafka 4.x |
| 5. The database boundary | Why there is no 2PC, deprecated ChainedKafkaTransactionManager, dedupe keys, the outbox pattern |
| 6. Performance | Measured cost per transaction size, and why max.poll.records is the knob |
| 7. Corner cases | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams |
Six findings you will not read elsewhere
- Boot 4 moved Kafka auto-configuration out of
spring-boot-autoconfigure. Depend onspring-kafkaalone and you get aNoSuchBeanDefinitionExceptionforKafkaTemplatethat looks like a generics problem. Usespring-boot-starter-kafka. → docs/01 acks=1silently disables idempotence — proven from the broker, not the config.Admin.describeProducers()shows 1 tracked producer state for a default producer and 0 for anacks=1producer. Settingenable.idempotence=trueexplicitly turns that silent downgrade into a startup failure. → docs/02- Transactions do not inflate consumer lag. This widely repeated claim is wrong: a drained
read_committedconsumer reaches exactly zero lag, because its position advances over markers too. What markers do break is counting records by offset arithmetic — measured here at 100% overstatement. → docs/03 - A random
transactional.idper start-up silently defeats fencing. Measured: two producers with different UUIDs both commit, and nothing is fenced. Everything appears to work while delivering exactly the duplicate processing you enabled transactions to prevent. → docs/04 - Aborted records reach the log only if they were already flushed.
abortTransaction()discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends on timing. Rely onread_committed, not on either behaviour. → docs/03 §3.6 - One record per transaction is ~111x slower than one hundred. The cost is per commit, not per
record, and the setting that controls it in Spring is
max.poll.records— not anything named "transaction". → docs/06
Reference machine
| CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads |
| RAM | 15.3 GB |
| OS | Windows 11 Home |
| JDK | java 25.0.3+9-LTS-195 |
| Kafka | in-JVM EmbeddedKafkaKraftBroker, kafka-clients 4.2.1, transaction.version=2 |
A laptop and a single-node broker. The absolute throughput figures are illustrative; the relationships are what generalise. See docs/06 §6.3 for what to trust.
Licence
MIT. See LICENSE.