asmhatre/spring-kafka-exactly-once-boot4
1
0
Fork 0
Code Activity

Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output

Browse Source 
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.

Demonstrations
  _01_idempotence   verifies idempotence from the BROKER via Admin.describeProducers rather
                    than by reading config back: a default producer registers one producer
                    state, an acks=1 producer registers zero. Also covers the ConfigException
                    you get only when idempotence is requested explicitly.
  _02_transactions  transaction markers consuming offsets, aborted records surviving in the
                    log, read_committed vs read_uncommitted, and a correction: markers do NOT
                    inflate consumer lag (a drained consumer reaches zero), they break
                    counting records by offset arithmetic (measured 100% overstatement).
  _03_read_process_write
                    the Spring loop with container-managed transactions; a listener that
                    fails after sending runs twice and is committed once.
  _04_fencing       shared transactional.id fences the zombie; a random per-startup id lets
                    both producers commit and silently defeats the guarantee entirely.
  _05_database      Kafka EOS does not extend to a database. Naive listener writes 2 rows for
                    1 message; the same listener keyed on topic-partition-offset writes 1.
  _06_performance   cost per transaction size, median of 5 interleaved rounds.

Docs
  Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
  idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
  outbox pattern, performance, and corner cases.

Measured highlights
  acks=1 producer          0 broker-tracked producer states (idempotence silently off)
  5 records in one tx      endOffset 6; offset span overstates record count by 100%
  1 record  per tx            711 records/s
  100 records per tx       78,770 records/s  (~111x)
This commit is contained in:
Ankur
2026-07-31 22:27:35 +05:30
commit e23ca359eb
26 changed files with 3635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
docs/02-idempotence.md Normal file
View File

@@ -0,0 +1,105 @@
# 2. Idempotent producers
> Runnable companion: [`IdempotentProducerTest`](../src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java)
## 2.1 The problem it solves
A producer sends a record. The broker writes it. The acknowledgement is lost on the way back — a
network blip, a leader election, a GC pause. The producer retries. The broker now holds the record
**twice**, and no amount of consumer-side care can undo that: the duplicate is a real, distinct
record with its own offset and its own content.
This is not exotic. It is the normal consequence of retries on an unreliable network, and retries
are on by default because without them you drop records instead.
## 2.2 How it works
The producer obtains a **producer ID (PID)** from the broker and stamps every record batch with a
**per-partition sequence number**. The broker tracks the last accepted sequence per PID per
partition and rejects a repeat. A retry of a batch the broker already has is silently dropped
instead of appended.
The guarantee has a precise scope: **exactly-once delivery per partition, for the lifetime of one
producer session.** It does not survive a producer restart (a new PID is issued), and it says
nothing about ordering across partitions.
## 2.3 It has been the default since Kafka 3.0
```java
ProducerConfig.configDef().defaultValues().get("enable.idempotence") // true
ProducerConfig.configDef().defaultValues().get("acks") // all
```
You can verify it from the broker's side rather than trusting the config, which is what the test in
this repository does:
```java
admin.describeProducers(List.of(partition))
.partitionResult(partition).get().activeProducers();
```
A producer with idempotence on appears there with a real producer ID, epoch and last sequence
number. A non-idempotent producer creates no such state, because there is nothing to de-duplicate
against. Measured output:
```
-- default producer --
producerId=0 producerEpoch=0 lastSequence=2
producer states tracked : 1
-- acks=1 producer --
producer states tracked : 0
```
## 2.4 The silent downgrade
`acks` and `enable.idempotence` interact, and — this is the important part — they interact
**differently depending on whether you set idempotence explicitly**:
| Configuration | Result |
|---|---|
| `acks=1` only | Idempotence **silently disabled**. No warning, no exception. |
| `acks=1` + `enable.idempotence=true` | `ConfigException` at construction. |
```
ConfigException: Must set acks to all in order to use the idempotent producer.
Otherwise we cannot guarantee idempotence.
```
The first row is the dangerous one and it is extremely common: someone set `acks=1` for latency, in
a properties file, years ago, and the system has been quietly at-least-once ever since. Nothing in
the logs says so.
**Therefore: set `enable.idempotence=true` explicitly even though it is the default.** Not because
it changes behaviour, but because it converts a silent downgrade into a startup failure. That is
the only reason, and it is a good one.
## 2.5 The other constraints
| Setting | Requirement | Fails how |
|---|---|---|
| `acks` | must be `all` | loudly if idempotence explicit, silently otherwise |
| `retries` | must be `> 0` | `ConfigException` |
| `max.in.flight.requests.per.connection` | must be `<= 5` | `ConfigException` |
Five in-flight requests is safe *with* idempotence because sequence numbers let the broker reorder
within that window. **Without** idempotence, `max.in.flight > 1` silently breaks ordering on retry:
a failed batch that is retried can land after a batch that was sent later. That is a second,
independent reason not to disable idempotence.
## 2.6 What idempotence is not
It is **not** exactly-once processing. It protects one producer session writing to one partition.
It does nothing about:
- a producer restarting and re-sending from an application-level buffer,
- a consumer processing a record twice after a rebalance,
- offsets and output records committing atomically,
- anything your listener writes to a database.
For those you need transactions ([03](03-transactions-and-markers.md)), fencing
([04](04-fencing.md)) and idempotent side effects ([05](05-database-boundary.md)).
---
Next: [03-transactions-and-markers.md](03-transactions-and-markers.md).