asmhatre/spring-kafka-exactly-once-boot4
1
0
Fork 0
Code Activity

Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output

Browse Source 
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.

Demonstrations
  _01_idempotence   verifies idempotence from the BROKER via Admin.describeProducers rather
                    than by reading config back: a default producer registers one producer
                    state, an acks=1 producer registers zero. Also covers the ConfigException
                    you get only when idempotence is requested explicitly.
  _02_transactions  transaction markers consuming offsets, aborted records surviving in the
                    log, read_committed vs read_uncommitted, and a correction: markers do NOT
                    inflate consumer lag (a drained consumer reaches zero), they break
                    counting records by offset arithmetic (measured 100% overstatement).
  _03_read_process_write
                    the Spring loop with container-managed transactions; a listener that
                    fails after sending runs twice and is committed once.
  _04_fencing       shared transactional.id fences the zombie; a random per-startup id lets
                    both producers commit and silently defeats the guarantee entirely.
  _05_database      Kafka EOS does not extend to a database. Naive listener writes 2 rows for
                    1 message; the same listener keyed on topic-partition-offset writes 1.
  _06_performance   cost per transaction size, median of 5 interleaved rounds.

Docs
  Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
  idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
  outbox pattern, performance, and corner cases.

Measured highlights
  acks=1 producer          0 broker-tracked producer states (idempotence silently off)
  5 records in one tx      endOffset 6; offset span overstates record count by 100%
  1 record  per tx            711 records/s
  100 records per tx       78,770 records/s  (~111x)
This commit is contained in:
Ankur
2026-07-31 22:27:35 +05:30
commit e23ca359eb
26 changed files with 3635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
docs/06-performance.md Normal file
View File

@@ -0,0 +1,96 @@
# 6. What transactions actually cost
> Runnable companion: [`TransactionCostTest`](../src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java)
## 6.1 The useful version of "transactions are slow"
The cost is **per transaction, not per record**.
Committing requires a round trip to the transaction coordinator plus a durable marker write into
every partition the transaction touched, plus acknowledgement of those writes. That is a fixed cost.
It does not care whether the transaction contained one record or ten thousand.
So the question is never "should I use transactions" in isolation. It is **"how many records go in
each one"**.
## 6.2 Measured
10,000 records of 256 bytes, single-node in-JVM broker, median of 5 interleaved rounds,
JDK 25.0.3 / Kafka 4.2.1 on an AMD Ryzen 5 5600U laptop:
| Configuration | records/s | round-to-round spread | vs previous row |
|---|---:|---:|---:|
| no transaction (idempotent only) | 801,880 | 26% | — |
| 1 record / transaction | **711** | 11% | — |
| 10 records / transaction | 8,452 | 12% | **11.9x** |
| 100 records / transaction | 78,770 | 55% | **9.3x** |
| 1,000 records / transaction | 330,969 | 77% | 4.2x |
| 10,000 records / transaction | 878,071 | 32% | 2.7x |
**One record per transaction is roughly 111x slower than one hundred**, and about three orders of
magnitude below the non-transactional baseline.
Each 10x increase in transaction size buys close to 10x throughput while commit cost dominates, then
flattens as per-record cost takes over — the knee is somewhere around 1,000 records here.
## 6.3 How to read these numbers honestly
They come from a laptop and a single-node in-JVM broker, so the absolute throughput is not
meaningful. Two specific caveats:
- The **non-transactional baseline is noisy** (26% spread, and it drifted 30% across the run).
Sending 10,000 records with no commit is fast enough that the measurement is dominated by flush
timing and OS scheduling. Ratios against the baseline are indicative only.
- The **transactional measurements are stable** because each is dominated by a fixed number of
commits. The comparisons *between* transaction sizes are the trustworthy part, and they are the
point.
On a real cluster the per-commit cost is *larger* — real network, real replication, real disk — so
the penalty at small transaction sizes is worse than shown here and batching matters more.
The test itself went through three revisions before producing numbers worth quoting; the reasons are
documented in its source, and they are the usual ones: insufficient warm-up, measurement order
accruing warm-up to whichever configuration ran last, and one path paying for a metadata fetch its
competitor had already done. If you adapt it, keep the interleaving and the drift report.
## 6.4 The setting that controls this in Spring
**`max.poll.records`.** Not anything named "transaction".
With a `KafkaTransactionManager`, the container's transaction boundary is *the batch returned by
`poll()`*. So the number of records per transaction is exactly `max.poll.records` (default 500),
capped by how many are actually available.
```yaml
spring:
kafka:
consumer:
max-poll-records: 500 # == records per transaction
```
Raising it is the single most effective throughput fix for a transactional listener. The price is
that a failure anywhere in the batch rolls back and redoes the whole batch, so you are trading
throughput against the cost of redoing work — and against `max.poll.interval.ms`, since the whole
batch must be processed within it.
## 6.5 The other costs, which are not throughput
- **Latency.** A `read_committed` consumer cannot read past the Last Stable Offset, so downstream
consumers see nothing until the transaction commits. Larger transactions mean higher end-to-end
latency. This is a direct trade against §6.4.
- **Storage.** Every transaction writes a marker per partition. High-frequency small transactions on
many partitions produce a surprising amount of control-record overhead.
- **Redelivery amplification.** One poisoned record in a batch of 500 rolls back all 500.
- **Coordinator load.** Every transaction is state on a broker, in `__transaction_state`.
## 6.6 Rules of thumb
- Do not use one transaction per record. It is three orders of magnitude off the achievable rate.
- Start at the default `max.poll.records=500` and tune from measurements, not from feeling.
- If latency matters more than throughput, lower it deliberately and know what you are buying.
- If you do not need atomic offset+output commits, do not use transactions at all — an idempotent
producer plus an idempotent consumer is cheaper and simpler, and covers most systems.
---
Next: [07-corner-cases.md](07-corner-cases.md).