asmhatre/spring-kafka-exactly-once-boot4
1
0
Fork 0
Code Activity

Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output

Browse Source 
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.

Demonstrations
  _01_idempotence   verifies idempotence from the BROKER via Admin.describeProducers rather
                    than by reading config back: a default producer registers one producer
                    state, an acks=1 producer registers zero. Also covers the ConfigException
                    you get only when idempotence is requested explicitly.
  _02_transactions  transaction markers consuming offsets, aborted records surviving in the
                    log, read_committed vs read_uncommitted, and a correction: markers do NOT
                    inflate consumer lag (a drained consumer reaches zero), they break
                    counting records by offset arithmetic (measured 100% overstatement).
  _03_read_process_write
                    the Spring loop with container-managed transactions; a listener that
                    fails after sending runs twice and is committed once.
  _04_fencing       shared transactional.id fences the zombie; a random per-startup id lets
                    both producers commit and silently defeats the guarantee entirely.
  _05_database      Kafka EOS does not extend to a database. Naive listener writes 2 rows for
                    1 message; the same listener keyed on topic-partition-offset writes 1.
  _06_performance   cost per transaction size, median of 5 interleaved rounds.

Docs
  Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
  idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
  outbox pattern, performance, and corner cases.

Measured highlights
  acks=1 producer          0 broker-tracked producer states (idempotence silently off)
  5 records in one tx      endOffset 6; offset span overstates record count by 100%
  1 record  per tx            711 records/s
  100 records per tx       78,770 records/s  (~111x)
This commit is contained in:
Ankur
2026-07-31 22:27:35 +05:30
commit e23ca359eb
26 changed files with 3635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
docs/01-boot4-setup.md Normal file
View File

@@ -0,0 +1,121 @@
# 1. Setting up exactly-once on Spring Boot 4
> Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3.
## 1.1 The dependency trap that will cost you an hour
On Spring Boot 3 this was enough:
```xml
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
```
…because `KafkaAutoConfiguration` lived in the monolithic `spring-boot-autoconfigure` jar that every
Boot application already had on the classpath.
**Boot 4 split auto-configuration into per-technology modules.** Kafka's now lives in
`org.springframework.boot:spring-boot-kafka`. Depend on `spring-kafka` alone and you get the
library but *no* auto-configuration: no `ProducerFactory`, no `KafkaTemplate`, no
`KafkaTransactionManager`.
The failure looks like this, and it looks like a generics problem, which it is not:
```
Error creating bean with name 'orderProcessor': Unsatisfied dependency expressed through
constructor parameter 0: No qualifying bean of type
'org.springframework.kafka.core.KafkaTemplate<java.lang.String, java.lang.String>' available
```
Use the starter:
```xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
```
which pulls in both `spring-kafka` and `spring-boot-kafka`.
## 1.2 The configuration
The whole exactly-once switch is one property:
```yaml
spring:
kafka:
producer:
transaction-id-prefix: order-processor-tx-
```
Spring Boot sees it, auto-configures a `KafkaTransactionManager`, and attaches it to every listener
container. From that point the container begins a Kafka transaction before invoking your listener,
writes the consumer offsets *into* that transaction, and commits — or aborts everything if your
listener throws.
Two more properties matter and neither is on by default:
```yaml
spring:
kafka:
consumer:
isolation-level: read_committed # default is read_uncommitted!
enable-auto-commit: false
```
`isolation-level` is the one people forget. Producing transactionally while consuming
`read_uncommitted` gives you all of the cost of transactions and none of the benefit — your
consumers will happily read records from transactions that were aborted. Proven in
[`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java).
## 1.3 Config worth stating even though it is already the default
```yaml
acks: all
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 5
```
All three are defaults on Kafka 3.0+. Stating them explicitly is still worth it, because
`enable.idempotence: true` converts a *silent* downgrade into a *startup failure* if anyone later
sets `acks=1` for latency. See [02-idempotence.md](02-idempotence.md) — this is the single most
common way a system that believes it is exactly-once is not.
## 1.4 Testing without Docker
Kafka 4.x removed ZooKeeper entirely, so `spring-kafka-test` now only ships
`EmbeddedKafkaKraftBroker`. It runs a real broker in-JVM:
```java
@SpringBootTest(classes = Application.class)
@EmbeddedKafka(
partitions = 1,
topics = {"orders.incoming", "orders.validated"},
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class MyTest { }
```
**Those three `brokerProperties` are mandatory for any transactional test.** The transaction state
log defaults to 3 replicas, which a single-broker embedded cluster cannot provide. Without them
every transactional test fails with a `COORDINATOR_NOT_AVAILABLE` that gives no hint as to why.
Two further limitations of the KRaft embedded broker:
- **You cannot pin ports.** `KafkaClusterTestKit` does not support it, so the `ports` attribute of
`@EmbeddedKafka` is ignored. Read the address from `EmbeddedKafkaBroker.getBrokersAsString()` or
the `spring.embedded.kafka.brokers` system property.
- Broker start-up costs a couple of seconds per distinct configuration. Spring's context caching
will reuse a broker across test classes whose configuration matches exactly, so keeping
`@EmbeddedKafka` annotations identical across classes is a real speed-up.
---
Next: [02-idempotence.md](02-idempotence.md).

105
docs/02-idempotence.md Normal file
View File

@@ -0,0 +1,105 @@
# 2. Idempotent producers
> Runnable companion: [`IdempotentProducerTest`](../src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java)
## 2.1 The problem it solves
A producer sends a record. The broker writes it. The acknowledgement is lost on the way back — a
network blip, a leader election, a GC pause. The producer retries. The broker now holds the record
**twice**, and no amount of consumer-side care can undo that: the duplicate is a real, distinct
record with its own offset and its own content.
This is not exotic. It is the normal consequence of retries on an unreliable network, and retries
are on by default because without them you drop records instead.
## 2.2 How it works
The producer obtains a **producer ID (PID)** from the broker and stamps every record batch with a
**per-partition sequence number**. The broker tracks the last accepted sequence per PID per
partition and rejects a repeat. A retry of a batch the broker already has is silently dropped
instead of appended.
The guarantee has a precise scope: **exactly-once delivery per partition, for the lifetime of one
producer session.** It does not survive a producer restart (a new PID is issued), and it says
nothing about ordering across partitions.
## 2.3 It has been the default since Kafka 3.0
```java
ProducerConfig.configDef().defaultValues().get("enable.idempotence") // true
ProducerConfig.configDef().defaultValues().get("acks") // all
```
You can verify it from the broker's side rather than trusting the config, which is what the test in
this repository does:
```java
admin.describeProducers(List.of(partition))
.partitionResult(partition).get().activeProducers();
```
A producer with idempotence on appears there with a real producer ID, epoch and last sequence
number. A non-idempotent producer creates no such state, because there is nothing to de-duplicate
against. Measured output:
```
-- default producer --
producerId=0 producerEpoch=0 lastSequence=2
producer states tracked : 1
-- acks=1 producer --
producer states tracked : 0
```
## 2.4 The silent downgrade
`acks` and `enable.idempotence` interact, and — this is the important part — they interact
**differently depending on whether you set idempotence explicitly**:
| Configuration | Result |
|---|---|
| `acks=1` only | Idempotence **silently disabled**. No warning, no exception. |
| `acks=1` + `enable.idempotence=true` | `ConfigException` at construction. |
```
ConfigException: Must set acks to all in order to use the idempotent producer.
Otherwise we cannot guarantee idempotence.
```
The first row is the dangerous one and it is extremely common: someone set `acks=1` for latency, in
a properties file, years ago, and the system has been quietly at-least-once ever since. Nothing in
the logs says so.
**Therefore: set `enable.idempotence=true` explicitly even though it is the default.** Not because
it changes behaviour, but because it converts a silent downgrade into a startup failure. That is
the only reason, and it is a good one.
## 2.5 The other constraints
| Setting | Requirement | Fails how |
|---|---|---|
| `acks` | must be `all` | loudly if idempotence explicit, silently otherwise |
| `retries` | must be `> 0` | `ConfigException` |
| `max.in.flight.requests.per.connection` | must be `<= 5` | `ConfigException` |
Five in-flight requests is safe *with* idempotence because sequence numbers let the broker reorder
within that window. **Without** idempotence, `max.in.flight > 1` silently breaks ordering on retry:
a failed batch that is retried can land after a batch that was sent later. That is a second,
independent reason not to disable idempotence.
## 2.6 What idempotence is not
It is **not** exactly-once processing. It protects one producer session writing to one partition.
It does nothing about:
- a producer restarting and re-sending from an application-level buffer,
- a consumer processing a record twice after a rebalance,
- offsets and output records committing atomically,
- anything your listener writes to a database.
For those you need transactions ([03](03-transactions-and-markers.md)), fencing
([04](04-fencing.md)) and idempotent side effects ([05](05-database-boundary.md)).
---
Next: [03-transactions-and-markers.md](03-transactions-and-markers.md).

138
docs/03-transactions-and-markers.md Normal file
View File

@@ -0,0 +1,138 @@
# 3. Transactions, isolation, and the markers nobody mentions
> Runnable companion: [`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java),
> [`ReadProcessWriteTest`](../src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java)
## 3.1 What a transaction adds over idempotence
Idempotence protects one producer session writing to one partition. A **transaction** makes a set of
writes atomic across partitions — *and*, crucially, lets a consumer offset commit be one of those
writes.
That last part is the whole trick behind read-process-write. The offset for the input record is
written **into the same transaction** as the output record, via `sendOffsetsToTransaction`. There is
no window between "produced the output" and "committed the offset" in which a crash produces a
duplicate, because there is no between.
Setting `transactional.id` on a producer also forces `enable.idempotence=true` — transactions are
built on top of idempotence, not as an alternative to it.
## 3.2 In Spring, you write none of that
```yaml
spring.kafka.producer.transaction-id-prefix: order-processor-tx-
spring.kafka.consumer.isolation-level: read_committed
```
```java
@KafkaListener(topics = "orders.incoming", groupId = "order-processor")
public void process(String order) {
kafkaTemplate.send("orders.validated", order.toUpperCase());
}
```
The container begins the transaction before the listener runs, enlists the `KafkaTemplate` send,
sends the offsets into the transaction, and commits — or aborts everything if the method throws.
The listener body is deliberately ordinary. **The guarantee is entirely in the configuration.**
## 3.3 `isolation.level` defaults to the wrong thing
`read_uncommitted` is the default. A consumer reading a transactional topic with the default
configuration will happily deliver records from transactions that were **aborted**.
Measured, on one partition, after committing 5 records and aborting 3:
```
read_committed : 5 records [committed-0 .. committed-4]
read_uncommitted : 8 records [committed-0 .. committed-4, aborted-0, aborted-1, aborted-2]
```
`isolation.level` is a **consumer-side filter**, not a broker-side delete. The aborted records are
physically in the log with permanent offsets and they consume disk. Producing transactionally
without setting `read_committed` on every consumer gives you the cost of transactions and none of
the benefit.
## 3.4 Transaction markers consume offsets
When a transaction commits or aborts, Kafka writes a **control record** — a transaction marker —
into every partition it touched. Control records are real log entries that **consume an offset**,
but are never delivered to your consumer.
```
records sent : 5
endOffset : 6 <- records occupy 0..4; the commit marker takes 5
last delivered offset : 4
```
Three consequences:
**Offsets are not contiguous.** Send five records in a transaction and the next record is at offset
6, not 5. Any code assuming `offset + 1` is the next record is wrong. In the Spring
read-process-write test, three messages processed in three transactions landed at offsets **0, 2 and
4** — a marker between each.
**Counting records by offset arithmetic is broken.** After 5 committed records, 3 aborted records
and 2 markers:
```
beginningOffset : 0
endOffset : 10
end - beginning : 10 <- looks like the record count
records actually retrievable: 5
overstatement : 100%
```
Retention estimates, "how many events between these offsets", partition-size dashboards and tests
that assert on offsets are all wrong by however much your marker and abort rate happens to be.
**Consumer lag, however, is fine.** This is worth stating because the opposite is widely repeated.
A drained `read_committed` consumer's *position* advances over markers and aborted records exactly
as it advances over delivered ones, so `endOffset - committedOffset` reaches zero:
```
partition end offset : 10
consumer committed offset : 10
lag : 0
```
The claim that transactions inflate consumer lag does not survive contact with a broker. The claim
that they break offset arithmetic does.
## 3.5 The Last Stable Offset, and why transaction duration is a consumer-latency setting
A `read_committed` consumer can only read up to the **Last Stable Offset** — the offset before the
first still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every
`read_committed` consumer on that partition stalls for 30 seconds, *including for records committed
by other producers in the meantime*.
So `transaction.timeout.ms` and how long you keep a transaction open are consumer-latency knobs, not
just producer concerns. Long-running transactions are the most common cause of "our consumers
randomly freeze for exactly N seconds".
## 3.6 Aborted records only reach the log if they were flushed
A detail that surprised this repository's own test. `KafkaTemplate.send()` is asynchronous. If the
transaction aborts while the record is still sitting in the producer's accumulator,
`abortTransaction()` **discards the un-flushed batch** and the record never reaches the broker at
all.
So aborted records occupy permanent offsets only when they were already flushed — which happens
under load, on large batches, or when you block on the send future. Both behaviours are correct and
neither is something to rely on. Rely on `read_committed`.
## 3.7 Exactly-once means committed once, not executed once
The clearest demonstration in this repository: a listener that fails *after* sending its output.
```
times the listener ran for 'delta' : 2
DELTA records visible (read_committed) : 1
```
The work happened twice. One result was committed. **That is what exactly-once means in Kafka, and
it has never meant that your code runs once.** Every side effect outside Kafka happened twice —
see [05-database-boundary.md](05-database-boundary.md).
---
Next: [04-fencing.md](04-fencing.md).

91
docs/04-fencing.md Normal file
View File

@@ -0,0 +1,91 @@
# 4. Zombie fencing and `transactional.id`
> Runnable companion: [`ZombieFencingTest`](../src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java)
## 4.1 The zombie
A processor reads, transforms and writes. It stalls — a long GC pause, a network partition, a
container the orchestrator has decided is dead. A replacement starts, the group rebalances, the new
instance owns the partition.
Then the old instance wakes up. It holds a partially completed transaction and does not know it was
replaced. If it were allowed to commit, the same input would be processed twice and both results
would be committed. That is the zombie.
## 4.2 The defence
`transactional.id` is a **stable, durable identity** the transaction coordinator remembers, along
with an **epoch**. Every `initTransactions()` for an existing id bumps the epoch and invalidates
anything holding the old one.
Measured:
```
Producer A: initTransactions, beginTransaction, send (transaction OPEN)
Producer B: initTransactions with the SAME id (epoch bumped)
Producer A: commitTransaction
-> REJECTED with ProducerFencedException
Producer B: commits normally
```
The zombie's work is discarded **by the broker**, not by any code you wrote.
## 4.3 The two ways to get it wrong
### Random per start-up — fencing never happens
```java
// Catastrophic, and everything appears to work
props.put(TRANSACTIONAL_ID_CONFIG, "processor-" + UUID.randomUUID());
```
Measured: producer A opens a transaction, producer B starts with a fresh UUID and commits, then A
commits too. **Nothing is fenced. Both commit.** The coordinator saw two unrelated identities and
had no reason to fence either.
This is the most damaging Kafka transactions misconfiguration precisely because it is invisible:
transactions commit, tests pass, and you have exactly the duplicate-processing problem you turned
transactions on to prevent.
### Shared across instances — everything fences everything
Every instance bumps every other instance's epoch continuously, and the group thrashes with
`ProducerFencedException`s.
## 4.4 Getting it right in Spring
```yaml
spring.kafka.producer.transaction-id-prefix: order-processor-tx-
```
Spring appends a suffix per producer. The prefix must be **stable across restarts** and **unique per
application instance** — typically the application name plus a stable instance identifier
(a StatefulSet ordinal, a pod index, a configured node id), *not* a random value and *not* a value
shared by every replica.
Since Spring Kafka 3.2, `TransactionIdSuffixStrategy` lets you control suffix allocation, which
matters when producers are cached and reused across partitions.
> With `EOSMode.V2` — the only mode since Spring Kafka 3.0 — the producer no longer needs a
> `transactional.id` tied to the specific input partition. V2 (KIP-447) uses consumer group metadata
> for fencing, so one producer per instance is sufficient. Older guidance about
> `transactional.id` per topic-partition describes the removed `EOSMode.ALPHA` and does not apply.
## 4.5 Kafka 4.x: KIP-890 changes which exception you see
[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890) strengthened the transaction
protocol against hanging transactions. Under `transaction.version=2` — the default on Kafka 4.x, and
what the embedded broker in this repository reports at start-up — the **producer epoch is bumped on
every transaction**, not only at `initTransactions()`.
Practical effect: the exception on fencing may be `InvalidProducerEpochException` rather than the
`ProducerFencedException` that older articles promise. Catch both, or catch neither and let the
container restart the producer, which is what Spring does.
A hanging transaction is worth understanding because its blast radius is large: it holds the Last
Stable Offset back, which stalls every `read_committed` consumer on the partition and prevents log
compaction. KIP-890 exists to make that class of bug impossible rather than merely rare.
---
Next: [05-database-boundary.md](05-database-boundary.md).

112
docs/05-database-boundary.md Normal file
View File

@@ -0,0 +1,112 @@
# 5. The boundary: Kafka, your database, and the thing that cannot be done
> Runnable companion: [`DatabaseBoundaryTest`](../src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java)
## 5.1 State it plainly
A Kafka transaction is atomic across **Kafka partitions and Kafka consumer offsets**. That is the
entire scope. It cannot include:
- a database write,
- an HTTP call to another service,
- a file,
- an email, an SMS, a push notification,
- a cache invalidation.
There is no two-phase commit between Kafka and your database, and there is not going to be. Kafka's
transaction coordinator has no protocol for enlisting a foreign resource manager.
So: **a listener that writes to a database is at-least-once with respect to that database, no matter
what your Kafka configuration says.** The redelivery that makes Kafka's guarantee work is exactly
the thing that duplicates your row.
## 5.2 What `ChainedKafkaTransactionManager` was, and why it is deprecated
Spring used to ship `ChainedKafkaTransactionManager` for this, and it is deprecated along with its
parent `ChainedTransactionManager`. The reason is worth understanding rather than just obeying:
chaining transaction managers is **best-effort ordering, not distributed coordination**. It commits
one, then the other. If the process dies in between, they disagree, and nothing reconciles them.
It narrows the window. It does not close it. A narrowed window is not a guarantee, and treating it
as one is worse than knowing you have none — because you will skip the idempotency work.
## 5.3 What to do instead
### Option A — idempotent writes (usually right)
Give the work a natural identity and enforce it with a `UNIQUE` constraint. Then a duplicate
attempt is a no-op regardless of *why* it duplicated.
```java
String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset);
if (!repository.existsByDedupeKey(dedupeKey)) {
repository.save(new ProcessedOrder(dedupeKey, payload));
}
```
`topic-partition-offset` is a good default key: Kafka guarantees it is unique and stable, it is on
every consumer record, and it requires no coordination or payload changes. An event ID carried in
the message is better when the same logical event can arrive on more than one topic or be replayed
from a different offset.
Do the check-and-insert inside the database transaction and let the `UNIQUE` constraint be the
real enforcement — the `exists` check is an optimisation, not the guarantee, because two concurrent
attempts can both pass it.
Measured in `DatabaseBoundaryTest`: with a naive insert the listener ran twice and wrote **2 rows**
for one message; with the dedupe key it ran twice and wrote **1**.
### Option B — the transactional outbox (when you cannot be idempotent)
Invert the problem. The database is the source of truth:
1. In **one local database transaction**, write your business change *and* a row into an `outbox`
table.
2. A separate relay — a poller, or change-data-capture such as Debezium — reads the outbox and
publishes to Kafka.
3. The relay is at-least-once, so consumers must tolerate duplicates, which idempotent consumers
already do.
This gives genuine atomicity for the part that matters (business state and the intent to publish),
using only a local transaction. The cost is a table, a relay, and publish latency.
### Option C — make the listener's side effect naturally idempotent
Sometimes the work is already idempotent and nobody noticed: `UPDATE ... SET status='PAID'`,
`PUT /resource/123`, an upsert keyed by business ID, a `SET` in Redis. If the operation is
absorbing rather than accumulating, redelivery is harmless. Prefer designing for this over
detecting duplicates.
## 5.4 Ordering, if you use `@Transactional` anyway
With a `KafkaTransactionManager` on the container and `@Transactional` on the listener method, you
get:
```
container begins Kafka transaction
interceptor begins DB transaction
listener body runs
DB transaction COMMITS <-- first
Kafka transaction COMMITS <-- second
```
The DB commits first, deliberately: if the Kafka commit then fails, the offset is not advanced, the
record is redelivered, and an idempotent DB write absorbs it. Had Kafka committed first, a DB
failure would lose the write entirely with the offset already advanced.
So `@Transactional` is still worth having — it makes the DB write roll back when your listener
throws, which is the common case. It just does not make the two systems atomic, and the crash
window between the two commits is real. Idempotency is what covers it.
## 5.5 The rule
> **Exactly-once processing = at-least-once delivery + idempotent side effects.**
>
> Kafka transactions give you the delivery half. You build the other half.
Anyone who tells you Kafka gives you end-to-end exactly-once across your whole system is describing
Kafka-to-Kafka and has not thought about the database.
---
Next: [06-performance.md](06-performance.md).

96
docs/06-performance.md Normal file
View File

@@ -0,0 +1,96 @@
# 6. What transactions actually cost
> Runnable companion: [`TransactionCostTest`](../src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java)
## 6.1 The useful version of "transactions are slow"
The cost is **per transaction, not per record**.
Committing requires a round trip to the transaction coordinator plus a durable marker write into
every partition the transaction touched, plus acknowledgement of those writes. That is a fixed cost.
It does not care whether the transaction contained one record or ten thousand.
So the question is never "should I use transactions" in isolation. It is **"how many records go in
each one"**.
## 6.2 Measured
10,000 records of 256 bytes, single-node in-JVM broker, median of 5 interleaved rounds,
JDK 25.0.3 / Kafka 4.2.1 on an AMD Ryzen 5 5600U laptop:
| Configuration | records/s | round-to-round spread | vs previous row |
|---|---:|---:|---:|
| no transaction (idempotent only) | 801,880 | 26% | — |
| 1 record / transaction | **711** | 11% | — |
| 10 records / transaction | 8,452 | 12% | **11.9x** |
| 100 records / transaction | 78,770 | 55% | **9.3x** |
| 1,000 records / transaction | 330,969 | 77% | 4.2x |
| 10,000 records / transaction | 878,071 | 32% | 2.7x |
**One record per transaction is roughly 111x slower than one hundred**, and about three orders of
magnitude below the non-transactional baseline.
Each 10x increase in transaction size buys close to 10x throughput while commit cost dominates, then
flattens as per-record cost takes over — the knee is somewhere around 1,000 records here.
## 6.3 How to read these numbers honestly
They come from a laptop and a single-node in-JVM broker, so the absolute throughput is not
meaningful. Two specific caveats:
- The **non-transactional baseline is noisy** (26% spread, and it drifted 30% across the run).
Sending 10,000 records with no commit is fast enough that the measurement is dominated by flush
timing and OS scheduling. Ratios against the baseline are indicative only.
- The **transactional measurements are stable** because each is dominated by a fixed number of
commits. The comparisons *between* transaction sizes are the trustworthy part, and they are the
point.
On a real cluster the per-commit cost is *larger* — real network, real replication, real disk — so
the penalty at small transaction sizes is worse than shown here and batching matters more.
The test itself went through three revisions before producing numbers worth quoting; the reasons are
documented in its source, and they are the usual ones: insufficient warm-up, measurement order
accruing warm-up to whichever configuration ran last, and one path paying for a metadata fetch its
competitor had already done. If you adapt it, keep the interleaving and the drift report.
## 6.4 The setting that controls this in Spring
**`max.poll.records`.** Not anything named "transaction".
With a `KafkaTransactionManager`, the container's transaction boundary is *the batch returned by
`poll()`*. So the number of records per transaction is exactly `max.poll.records` (default 500),
capped by how many are actually available.
```yaml
spring:
kafka:
consumer:
max-poll-records: 500 # == records per transaction
```
Raising it is the single most effective throughput fix for a transactional listener. The price is
that a failure anywhere in the batch rolls back and redoes the whole batch, so you are trading
throughput against the cost of redoing work — and against `max.poll.interval.ms`, since the whole
batch must be processed within it.
## 6.5 The other costs, which are not throughput
- **Latency.** A `read_committed` consumer cannot read past the Last Stable Offset, so downstream
consumers see nothing until the transaction commits. Larger transactions mean higher end-to-end
latency. This is a direct trade against §6.4.
- **Storage.** Every transaction writes a marker per partition. High-frequency small transactions on
many partitions produce a surprising amount of control-record overhead.
- **Redelivery amplification.** One poisoned record in a batch of 500 rolls back all 500.
- **Coordinator load.** Every transaction is state on a broker, in `__transaction_state`.
## 6.6 Rules of thumb
- Do not use one transaction per record. It is three orders of magnitude off the achievable rate.
- Start at the default `max.poll.records=500` and tune from measurements, not from feeling.
- If latency matters more than throughput, lower it deliberately and know what you are buying.
- If you do not need atomic offset+output commits, do not use transactions at all — an idempotent
producer plus an idempotent consumer is cheaper and simpler, and covers most systems.
---
Next: [07-corner-cases.md](07-corner-cases.md).

109
docs/07-corner-cases.md Normal file
View File

@@ -0,0 +1,109 @@
# 7. Corner cases and things that bite
The blog post covers the main concepts. This chapter is the long tail — behaviours that do not fit a
narrative but do show up in production.
## 7.1 `transaction.timeout.ms` vs `max.poll.interval.ms`
The most common "it worked in dev" failure. Your listener must finish a whole poll batch, commit,
and get back to `poll()` inside **both** limits:
| Setting | Default | Owned by |
|---|---|---|
| `transaction.timeout.ms` | 60,000 (Spring sets it; Kafka's producer default is 60s) | producer |
| `transaction.max.timeout.ms` | 900,000 | **broker** — caps the above |
| `max.poll.interval.ms` | 300,000 | consumer |
If processing exceeds `transaction.timeout.ms`, the coordinator aborts your transaction and your
commit fails with `ProducerFencedException` / `InvalidProducerEpochException`. If it exceeds
`max.poll.interval.ms`, the consumer is ejected from the group and the partitions rebalance —
usually producing both symptoms at once and a very confusing log.
Rule: `transaction.timeout.ms` should comfortably exceed the worst-case time to process one batch
of `max.poll.records`, and the broker's `transaction.max.timeout.ms` must be at least as large or
producer creation fails outright.
## 7.2 A long transaction stalls other people's consumers
Covered in [03](03-transactions-and-markers.md) §3.5 and worth repeating because it is
counter-intuitive: a `read_committed` consumer cannot read past the Last Stable Offset, which is
held back by the *oldest open transaction on the partition* — regardless of which producer opened
it. One slow producer degrades every consumer on that partition.
## 7.3 `@KafkaListener` with manual acknowledgment does nothing useful
With a `KafkaTransactionManager` present, `AckMode.MANUAL` / `MANUAL_IMMEDIATE` and calls to
`Acknowledgment.acknowledge()` are irrelevant: the offset commit happens inside the transaction, at
the end of the batch. Leaving manual ack code in place is harmless but misleading — future readers
will believe it controls something.
## 7.4 `executeInTransaction` is not the same as being in a listener transaction
```java
kafkaTemplate.executeInTransaction(t -> t.send(TOPIC, value));
```
This starts a **local** producer transaction. It is the right way to produce transactionally from
outside a listener (a REST controller, a scheduled job, a test). Inside a listener that already has
a container-managed transaction, do **not** call it — plain `send()` enlists in the existing
transaction automatically. Nesting produces `IllegalStateException` or, worse, a second independent
transaction that commits separately.
## 7.5 Error handling interacts with transactions
`DefaultErrorHandler` retries in-process by default and then, if configured, publishes to a dead
letter topic via `DeadLetterPublishingRecoverer`. Two things to know:
- The DLT publish happens **inside the same transaction** when a `KafkaTransactionManager` is
present, so "record moved to DLT and offset advanced" is atomic. That is the behaviour you want,
and it is why a DLT is the correct terminal state for a poison record under EOS.
- Retries consume `max.poll.interval.ms`. A back-off of 10 attempts × 30 s will eject the consumer
from the group long before it gives up. Use `DefaultErrorHandler` with a bounded back-off and let
the DLT absorb the rest.
## 7.6 Producers are pooled, and `transactional.id` suffixes are allocated from that pool
Spring caches producers. With transactions, each concurrent transaction needs its own
`transactional.id`, so Spring allocates `prefix + suffix`. `TransactionIdSuffixStrategy`
(Spring Kafka 3.2+) controls that allocation; the default reuses suffixes as producers return to the
cache. If you see unexpected fencing under high concurrency, look here before looking at your code.
## 7.7 Consumer group `isolation.level` cannot be changed per-record
It is a consumer-level setting. If some records on a topic must be visible before commit and others
after, you need two topics, not two isolation levels.
## 7.8 Compacted topics and transactions
Transaction markers interact with log compaction: a marker cannot be cleaned until the transaction
it belongs to is complete, and a **hanging** transaction therefore blocks compaction of the whole
partition indefinitely. This was one of the motivations for
[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890). If a compacted topic stops
shrinking, check for open transactions before checking your compaction settings.
## 7.9 Exactly-once across two Kafka clusters is not a thing
A transaction is scoped to one cluster's transaction coordinator. MirrorMaker 2 and most replication
tooling are at-least-once across clusters. A cross-cluster pipeline is at-least-once end to end no
matter what each side does internally.
## 7.10 Kafka Streams does this for you
If your topology is genuinely read-process-write over Kafka only, `processing.guarantee=exactly_once_v2`
in Kafka Streams gives you all of the above with none of the wiring, including state store changelogs
in the same transaction. Reach for Spring Kafka transactions when you need to interleave non-Kafka
work or you are not building a streaming topology — not because Streams cannot do it.
## 7.11 Things that are true and often assumed not to be
- **Idempotence and transactions do not require a special broker configuration** beyond replication
factors adequate for `__transaction_state`.
- **`enable.idempotence` is on by default** (Kafka 3.0+) — see [02](02-idempotence.md).
- **A transactional producer can send to topics it never mentioned up front.** Partitions are added
to the transaction dynamically as you send.
- **Reading your own writes** inside a transaction is not possible — a `read_committed` consumer
will not see them until commit, including in the same process.
---
Back to the [README](../README.md).