asmhatre/spring-kafka-exactly-once-boot4
1
0
Fork 0
Code Activity

Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output

Browse Source 
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.

Demonstrations
  _01_idempotence   verifies idempotence from the BROKER via Admin.describeProducers rather
                    than by reading config back: a default producer registers one producer
                    state, an acks=1 producer registers zero. Also covers the ConfigException
                    you get only when idempotence is requested explicitly.
  _02_transactions  transaction markers consuming offsets, aborted records surviving in the
                    log, read_committed vs read_uncommitted, and a correction: markers do NOT
                    inflate consumer lag (a drained consumer reaches zero), they break
                    counting records by offset arithmetic (measured 100% overstatement).
  _03_read_process_write
                    the Spring loop with container-managed transactions; a listener that
                    fails after sending runs twice and is committed once.
  _04_fencing       shared transactional.id fences the zombie; a random per-startup id lets
                    both producers commit and silently defeats the guarantee entirely.
  _05_database      Kafka EOS does not extend to a database. Naive listener writes 2 rows for
                    1 message; the same listener keyed on topic-partition-offset writes 1.
  _06_performance   cost per transaction size, median of 5 interleaved rounds.

Docs
  Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
  idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
  outbox pattern, performance, and corner cases.

Measured highlights
  acks=1 producer          0 broker-tracked producer states (idempotence silently off)
  5 records in one tx      endOffset 6; offset span overstates record count by 100%
  1 record  per tx            711 records/s
  100 records per tx       78,770 records/s  (~111x)
This commit is contained in:
Ankur
2026-07-31 22:27:35 +05:30
commit e23ca359eb
26 changed files with 3635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
.gitignore vendored Normal file
View File

@@ -0,0 +1,18 @@
# build output
target/
*.class
# local run scratch files (the curated transcript lives in results/)
perf.txt
perf-err.txt
rpw.txt
diag.txt
results/full-run-err.txt
# editors
.idea/
*.iml
.vscode/
.DS_Store
# results/ IS committed on purpose -- it is the evidence for the blog post

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Ankur Mhatre
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

111
README.md Normal file
View File

@@ -0,0 +1,111 @@
# Exactly-once with Spring Kafka on Spring Boot 4
Companion repository for **[Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions](https://ankurm.com/spring-kafka-exactly-once-boot-4/)**
on [ankurm.com](https://ankurm.com).
The blog post covers the concepts most people need. **This repository covers everything** — every
demonstration, every failure mode, every corner case, as runnable tests with real captured output.
Verified against **Spring Boot 4.1.0**, **Spring Kafka 4.1.0**, **kafka-clients 4.2.1**, **JDK 25.0.3**.
Raw output of the full suite is in [`results/full-run.txt`](results/full-run.txt).
---
## Quick start
No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker.
```console
git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git
cd spring-kafka-exactly-once-boot4
mvn test
```
15 tests, about three minutes, mostly the performance sweep.
Run one demonstration at a time:
```console
mvn test -Dtest=IdempotentProducerTest
mvn test -Dtest=TransactionMarkerOffsetTest
mvn test -Dtest=ZombieFencingTest
mvn test -Dtest=DatabaseBoundaryTest
mvn test -Dtest=TransactionCostTest
```
---
## The demonstrations
| Test | Shows | Key measured result |
|---|---|---|
| [`IdempotentProducerTest`](src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) | Idempotence verified **from the broker** via `describeProducers`, and the config that silently disables it | default producer → 1 tracked producer state; `acks=1` → **0** |
| [`TransactionMarkerOffsetTest`](src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java) | Transaction markers, aborted records, `read_committed` vs `read_uncommitted`, offset arithmetic | 5 records → `endOffset` 6; offset span overstates record count by **100%** |
| [`ReadProcessWriteTest`](src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) | The Spring read-process-write loop, and rollback + redelivery | listener ran **2x**, `read_committed` consumer sees **1** |
| [`ZombieFencingTest`](src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) | What `transactional.id` is for, and how a random one destroys the guarantee | shared id → `ProducerFencedException`; random id → **both commit** |
| [`DatabaseBoundaryTest`](src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) | Kafka EOS does not cover your database, and the fix | naive listener → **2 rows**; idempotent listener → **1 row** |
| [`TransactionCostTest`](src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) | What transactions cost, and the setting that controls it | 1 record/tx = **711 rec/s**; 100/tx = **78,770 rec/s** |
## The documentation
| Chapter | Covers |
|---|---|
| [1. Boot 4 setup](docs/01-boot4-setup.md) | The `spring-boot-starter-kafka` trap, the one property that enables EOS, testing without Docker |
| [2. Idempotence](docs/02-idempotence.md) | PIDs and sequence numbers, the `acks=1` silent downgrade, the constraints |
| [3. Transactions and markers](docs/03-transactions-and-markers.md) | `sendOffsetsToTransaction`, isolation levels, control records, the Last Stable Offset |
| [4. Fencing](docs/04-fencing.md) | `transactional.id`, epochs, EOSMode V2, KIP-890 on Kafka 4.x |
| [5. The database boundary](docs/05-database-boundary.md) | Why there is no 2PC, deprecated `ChainedKafkaTransactionManager`, dedupe keys, the outbox pattern |
| [6. Performance](docs/06-performance.md) | Measured cost per transaction size, and why `max.poll.records` is the knob |
| [7. Corner cases](docs/07-corner-cases.md) | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams |
---
## Six findings you will not read elsewhere
1. **Boot 4 moved Kafka auto-configuration out of `spring-boot-autoconfigure`.** Depend on
`spring-kafka` alone and you get a `NoSuchBeanDefinitionException` for `KafkaTemplate` that looks
like a generics problem. Use `spring-boot-starter-kafka`.
→ [docs/01](docs/01-boot4-setup.md)
2. **`acks=1` silently disables idempotence — proven from the broker, not the config.**
`Admin.describeProducers()` shows 1 tracked producer state for a default producer and **0** for an
`acks=1` producer. Setting `enable.idempotence=true` explicitly turns that silent downgrade into
a startup failure.
→ [docs/02](docs/02-idempotence.md)
3. **Transactions do *not* inflate consumer lag.** This widely repeated claim is wrong: a drained
`read_committed` consumer reaches exactly zero lag, because its position advances over markers
too. What markers *do* break is counting records by offset arithmetic — measured here at **100%**
overstatement.
→ [docs/03](docs/03-transactions-and-markers.md)
4. **A random `transactional.id` per start-up silently defeats fencing.** Measured: two producers
with different UUIDs both commit, and nothing is fenced. Everything appears to work while
delivering exactly the duplicate processing you enabled transactions to prevent.
→ [docs/04](docs/04-fencing.md)
5. **Aborted records reach the log only if they were already flushed.** `abortTransaction()`
discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends
on timing. Rely on `read_committed`, not on either behaviour.
→ [docs/03](docs/03-transactions-and-markers.md) §3.6
6. **One record per transaction is ~111x slower than one hundred.** The cost is per commit, not per
record, and the setting that controls it in Spring is `max.poll.records` — not anything named
"transaction".
→ [docs/06](docs/06-performance.md)
---
## Reference machine
| | |
|---|---|
| CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads |
| RAM | 15.3 GB |
| OS | Windows 11 Home |
| JDK | `java 25.0.3+9-LTS-195` |
| Kafka | in-JVM `EmbeddedKafkaKraftBroker`, kafka-clients 4.2.1, `transaction.version=2` |
A laptop and a single-node broker. The absolute throughput figures are illustrative; the
relationships are what generalise. See [docs/06](docs/06-performance.md) §6.3 for what to trust.
---
## Licence
MIT. See [LICENSE](LICENSE).

121
docs/01-boot4-setup.md Normal file
View File

@@ -0,0 +1,121 @@
# 1. Setting up exactly-once on Spring Boot 4
> Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3.
## 1.1 The dependency trap that will cost you an hour
On Spring Boot 3 this was enough:
```xml
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
```
…because `KafkaAutoConfiguration` lived in the monolithic `spring-boot-autoconfigure` jar that every
Boot application already had on the classpath.
**Boot 4 split auto-configuration into per-technology modules.** Kafka's now lives in
`org.springframework.boot:spring-boot-kafka`. Depend on `spring-kafka` alone and you get the
library but *no* auto-configuration: no `ProducerFactory`, no `KafkaTemplate`, no
`KafkaTransactionManager`.
The failure looks like this, and it looks like a generics problem, which it is not:
```
Error creating bean with name 'orderProcessor': Unsatisfied dependency expressed through
constructor parameter 0: No qualifying bean of type
'org.springframework.kafka.core.KafkaTemplate<java.lang.String, java.lang.String>' available
```
Use the starter:
```xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
```
which pulls in both `spring-kafka` and `spring-boot-kafka`.
## 1.2 The configuration
The whole exactly-once switch is one property:
```yaml
spring:
kafka:
producer:
transaction-id-prefix: order-processor-tx-
```
Spring Boot sees it, auto-configures a `KafkaTransactionManager`, and attaches it to every listener
container. From that point the container begins a Kafka transaction before invoking your listener,
writes the consumer offsets *into* that transaction, and commits — or aborts everything if your
listener throws.
Two more properties matter and neither is on by default:
```yaml
spring:
kafka:
consumer:
isolation-level: read_committed # default is read_uncommitted!
enable-auto-commit: false
```
`isolation-level` is the one people forget. Producing transactionally while consuming
`read_uncommitted` gives you all of the cost of transactions and none of the benefit — your
consumers will happily read records from transactions that were aborted. Proven in
[`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java).
## 1.3 Config worth stating even though it is already the default
```yaml
acks: all
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 5
```
All three are defaults on Kafka 3.0+. Stating them explicitly is still worth it, because
`enable.idempotence: true` converts a *silent* downgrade into a *startup failure* if anyone later
sets `acks=1` for latency. See [02-idempotence.md](02-idempotence.md) — this is the single most
common way a system that believes it is exactly-once is not.
## 1.4 Testing without Docker
Kafka 4.x removed ZooKeeper entirely, so `spring-kafka-test` now only ships
`EmbeddedKafkaKraftBroker`. It runs a real broker in-JVM:
```java
@SpringBootTest(classes = Application.class)
@EmbeddedKafka(
partitions = 1,
topics = {"orders.incoming", "orders.validated"},
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class MyTest { }
```
**Those three `brokerProperties` are mandatory for any transactional test.** The transaction state
log defaults to 3 replicas, which a single-broker embedded cluster cannot provide. Without them
every transactional test fails with a `COORDINATOR_NOT_AVAILABLE` that gives no hint as to why.
Two further limitations of the KRaft embedded broker:
- **You cannot pin ports.** `KafkaClusterTestKit` does not support it, so the `ports` attribute of
`@EmbeddedKafka` is ignored. Read the address from `EmbeddedKafkaBroker.getBrokersAsString()` or
the `spring.embedded.kafka.brokers` system property.
- Broker start-up costs a couple of seconds per distinct configuration. Spring's context caching
will reuse a broker across test classes whose configuration matches exactly, so keeping
`@EmbeddedKafka` annotations identical across classes is a real speed-up.
---
Next: [02-idempotence.md](02-idempotence.md).

105
docs/02-idempotence.md Normal file
View File

@@ -0,0 +1,105 @@
# 2. Idempotent producers
> Runnable companion: [`IdempotentProducerTest`](../src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java)
## 2.1 The problem it solves
A producer sends a record. The broker writes it. The acknowledgement is lost on the way back — a
network blip, a leader election, a GC pause. The producer retries. The broker now holds the record
**twice**, and no amount of consumer-side care can undo that: the duplicate is a real, distinct
record with its own offset and its own content.
This is not exotic. It is the normal consequence of retries on an unreliable network, and retries
are on by default because without them you drop records instead.
## 2.2 How it works
The producer obtains a **producer ID (PID)** from the broker and stamps every record batch with a
**per-partition sequence number**. The broker tracks the last accepted sequence per PID per
partition and rejects a repeat. A retry of a batch the broker already has is silently dropped
instead of appended.
The guarantee has a precise scope: **exactly-once delivery per partition, for the lifetime of one
producer session.** It does not survive a producer restart (a new PID is issued), and it says
nothing about ordering across partitions.
## 2.3 It has been the default since Kafka 3.0
```java
ProducerConfig.configDef().defaultValues().get("enable.idempotence") // true
ProducerConfig.configDef().defaultValues().get("acks") // all
```
You can verify it from the broker's side rather than trusting the config, which is what the test in
this repository does:
```java
admin.describeProducers(List.of(partition))
.partitionResult(partition).get().activeProducers();
```
A producer with idempotence on appears there with a real producer ID, epoch and last sequence
number. A non-idempotent producer creates no such state, because there is nothing to de-duplicate
against. Measured output:
```
-- default producer --
producerId=0 producerEpoch=0 lastSequence=2
producer states tracked : 1
-- acks=1 producer --
producer states tracked : 0
```
## 2.4 The silent downgrade
`acks` and `enable.idempotence` interact, and — this is the important part — they interact
**differently depending on whether you set idempotence explicitly**:
| Configuration | Result |
|---|---|
| `acks=1` only | Idempotence **silently disabled**. No warning, no exception. |
| `acks=1` + `enable.idempotence=true` | `ConfigException` at construction. |
```
ConfigException: Must set acks to all in order to use the idempotent producer.
Otherwise we cannot guarantee idempotence.
```
The first row is the dangerous one and it is extremely common: someone set `acks=1` for latency, in
a properties file, years ago, and the system has been quietly at-least-once ever since. Nothing in
the logs says so.
**Therefore: set `enable.idempotence=true` explicitly even though it is the default.** Not because
it changes behaviour, but because it converts a silent downgrade into a startup failure. That is
the only reason, and it is a good one.
## 2.5 The other constraints
| Setting | Requirement | Fails how |
|---|---|---|
| `acks` | must be `all` | loudly if idempotence explicit, silently otherwise |
| `retries` | must be `> 0` | `ConfigException` |
| `max.in.flight.requests.per.connection` | must be `<= 5` | `ConfigException` |
Five in-flight requests is safe *with* idempotence because sequence numbers let the broker reorder
within that window. **Without** idempotence, `max.in.flight > 1` silently breaks ordering on retry:
a failed batch that is retried can land after a batch that was sent later. That is a second,
independent reason not to disable idempotence.
## 2.6 What idempotence is not
It is **not** exactly-once processing. It protects one producer session writing to one partition.
It does nothing about:
- a producer restarting and re-sending from an application-level buffer,
- a consumer processing a record twice after a rebalance,
- offsets and output records committing atomically,
- anything your listener writes to a database.
For those you need transactions ([03](03-transactions-and-markers.md)), fencing
([04](04-fencing.md)) and idempotent side effects ([05](05-database-boundary.md)).
---
Next: [03-transactions-and-markers.md](03-transactions-and-markers.md).

138
docs/03-transactions-and-markers.md Normal file
View File

@@ -0,0 +1,138 @@
# 3. Transactions, isolation, and the markers nobody mentions
> Runnable companion: [`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java),
> [`ReadProcessWriteTest`](../src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java)
## 3.1 What a transaction adds over idempotence
Idempotence protects one producer session writing to one partition. A **transaction** makes a set of
writes atomic across partitions — *and*, crucially, lets a consumer offset commit be one of those
writes.
That last part is the whole trick behind read-process-write. The offset for the input record is
written **into the same transaction** as the output record, via `sendOffsetsToTransaction`. There is
no window between "produced the output" and "committed the offset" in which a crash produces a
duplicate, because there is no between.
Setting `transactional.id` on a producer also forces `enable.idempotence=true` — transactions are
built on top of idempotence, not as an alternative to it.
## 3.2 In Spring, you write none of that
```yaml
spring.kafka.producer.transaction-id-prefix: order-processor-tx-
spring.kafka.consumer.isolation-level: read_committed
```
```java
@KafkaListener(topics = "orders.incoming", groupId = "order-processor")
public void process(String order) {
kafkaTemplate.send("orders.validated", order.toUpperCase());
}
```
The container begins the transaction before the listener runs, enlists the `KafkaTemplate` send,
sends the offsets into the transaction, and commits — or aborts everything if the method throws.
The listener body is deliberately ordinary. **The guarantee is entirely in the configuration.**
## 3.3 `isolation.level` defaults to the wrong thing
`read_uncommitted` is the default. A consumer reading a transactional topic with the default
configuration will happily deliver records from transactions that were **aborted**.
Measured, on one partition, after committing 5 records and aborting 3:
```
read_committed : 5 records [committed-0 .. committed-4]
read_uncommitted : 8 records [committed-0 .. committed-4, aborted-0, aborted-1, aborted-2]
```
`isolation.level` is a **consumer-side filter**, not a broker-side delete. The aborted records are
physically in the log with permanent offsets and they consume disk. Producing transactionally
without setting `read_committed` on every consumer gives you the cost of transactions and none of
the benefit.
## 3.4 Transaction markers consume offsets
When a transaction commits or aborts, Kafka writes a **control record** — a transaction marker —
into every partition it touched. Control records are real log entries that **consume an offset**,
but are never delivered to your consumer.
```
records sent : 5
endOffset : 6 <- records occupy 0..4; the commit marker takes 5
last delivered offset : 4
```
Three consequences:
**Offsets are not contiguous.** Send five records in a transaction and the next record is at offset
6, not 5. Any code assuming `offset + 1` is the next record is wrong. In the Spring
read-process-write test, three messages processed in three transactions landed at offsets **0, 2 and
4** — a marker between each.
**Counting records by offset arithmetic is broken.** After 5 committed records, 3 aborted records
and 2 markers:
```
beginningOffset : 0
endOffset : 10
end - beginning : 10 <- looks like the record count
records actually retrievable: 5
overstatement : 100%
```
Retention estimates, "how many events between these offsets", partition-size dashboards and tests
that assert on offsets are all wrong by however much your marker and abort rate happens to be.
**Consumer lag, however, is fine.** This is worth stating because the opposite is widely repeated.
A drained `read_committed` consumer's *position* advances over markers and aborted records exactly
as it advances over delivered ones, so `endOffset - committedOffset` reaches zero:
```
partition end offset : 10
consumer committed offset : 10
lag : 0
```
The claim that transactions inflate consumer lag does not survive contact with a broker. The claim
that they break offset arithmetic does.
## 3.5 The Last Stable Offset, and why transaction duration is a consumer-latency setting
A `read_committed` consumer can only read up to the **Last Stable Offset** — the offset before the
first still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every
`read_committed` consumer on that partition stalls for 30 seconds, *including for records committed
by other producers in the meantime*.
So `transaction.timeout.ms` and how long you keep a transaction open are consumer-latency knobs, not
just producer concerns. Long-running transactions are the most common cause of "our consumers
randomly freeze for exactly N seconds".
## 3.6 Aborted records only reach the log if they were flushed
A detail that surprised this repository's own test. `KafkaTemplate.send()` is asynchronous. If the
transaction aborts while the record is still sitting in the producer's accumulator,
`abortTransaction()` **discards the un-flushed batch** and the record never reaches the broker at
all.
So aborted records occupy permanent offsets only when they were already flushed — which happens
under load, on large batches, or when you block on the send future. Both behaviours are correct and
neither is something to rely on. Rely on `read_committed`.
## 3.7 Exactly-once means committed once, not executed once
The clearest demonstration in this repository: a listener that fails *after* sending its output.
```
times the listener ran for 'delta' : 2
DELTA records visible (read_committed) : 1
```
The work happened twice. One result was committed. **That is what exactly-once means in Kafka, and
it has never meant that your code runs once.** Every side effect outside Kafka happened twice —
see [05-database-boundary.md](05-database-boundary.md).
---
Next: [04-fencing.md](04-fencing.md).

91
docs/04-fencing.md Normal file
View File

@@ -0,0 +1,91 @@
# 4. Zombie fencing and `transactional.id`
> Runnable companion: [`ZombieFencingTest`](../src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java)
## 4.1 The zombie
A processor reads, transforms and writes. It stalls — a long GC pause, a network partition, a
container the orchestrator has decided is dead. A replacement starts, the group rebalances, the new
instance owns the partition.
Then the old instance wakes up. It holds a partially completed transaction and does not know it was
replaced. If it were allowed to commit, the same input would be processed twice and both results
would be committed. That is the zombie.
## 4.2 The defence
`transactional.id` is a **stable, durable identity** the transaction coordinator remembers, along
with an **epoch**. Every `initTransactions()` for an existing id bumps the epoch and invalidates
anything holding the old one.
Measured:
```
Producer A: initTransactions, beginTransaction, send (transaction OPEN)
Producer B: initTransactions with the SAME id (epoch bumped)
Producer A: commitTransaction
-> REJECTED with ProducerFencedException
Producer B: commits normally
```
The zombie's work is discarded **by the broker**, not by any code you wrote.
## 4.3 The two ways to get it wrong
### Random per start-up — fencing never happens
```java
// Catastrophic, and everything appears to work
props.put(TRANSACTIONAL_ID_CONFIG, "processor-" + UUID.randomUUID());
```
Measured: producer A opens a transaction, producer B starts with a fresh UUID and commits, then A
commits too. **Nothing is fenced. Both commit.** The coordinator saw two unrelated identities and
had no reason to fence either.
This is the most damaging Kafka transactions misconfiguration precisely because it is invisible:
transactions commit, tests pass, and you have exactly the duplicate-processing problem you turned
transactions on to prevent.
### Shared across instances — everything fences everything
Every instance bumps every other instance's epoch continuously, and the group thrashes with
`ProducerFencedException`s.
## 4.4 Getting it right in Spring
```yaml
spring.kafka.producer.transaction-id-prefix: order-processor-tx-
```
Spring appends a suffix per producer. The prefix must be **stable across restarts** and **unique per
application instance** — typically the application name plus a stable instance identifier
(a StatefulSet ordinal, a pod index, a configured node id), *not* a random value and *not* a value
shared by every replica.
Since Spring Kafka 3.2, `TransactionIdSuffixStrategy` lets you control suffix allocation, which
matters when producers are cached and reused across partitions.
> With `EOSMode.V2` — the only mode since Spring Kafka 3.0 — the producer no longer needs a
> `transactional.id` tied to the specific input partition. V2 (KIP-447) uses consumer group metadata
> for fencing, so one producer per instance is sufficient. Older guidance about
> `transactional.id` per topic-partition describes the removed `EOSMode.ALPHA` and does not apply.
## 4.5 Kafka 4.x: KIP-890 changes which exception you see
[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890) strengthened the transaction
protocol against hanging transactions. Under `transaction.version=2` — the default on Kafka 4.x, and
what the embedded broker in this repository reports at start-up — the **producer epoch is bumped on
every transaction**, not only at `initTransactions()`.
Practical effect: the exception on fencing may be `InvalidProducerEpochException` rather than the
`ProducerFencedException` that older articles promise. Catch both, or catch neither and let the
container restart the producer, which is what Spring does.
A hanging transaction is worth understanding because its blast radius is large: it holds the Last
Stable Offset back, which stalls every `read_committed` consumer on the partition and prevents log
compaction. KIP-890 exists to make that class of bug impossible rather than merely rare.
---
Next: [05-database-boundary.md](05-database-boundary.md).

112
docs/05-database-boundary.md Normal file
View File

@@ -0,0 +1,112 @@
# 5. The boundary: Kafka, your database, and the thing that cannot be done
> Runnable companion: [`DatabaseBoundaryTest`](../src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java)
## 5.1 State it plainly
A Kafka transaction is atomic across **Kafka partitions and Kafka consumer offsets**. That is the
entire scope. It cannot include:
- a database write,
- an HTTP call to another service,
- a file,
- an email, an SMS, a push notification,
- a cache invalidation.
There is no two-phase commit between Kafka and your database, and there is not going to be. Kafka's
transaction coordinator has no protocol for enlisting a foreign resource manager.
So: **a listener that writes to a database is at-least-once with respect to that database, no matter
what your Kafka configuration says.** The redelivery that makes Kafka's guarantee work is exactly
the thing that duplicates your row.
## 5.2 What `ChainedKafkaTransactionManager` was, and why it is deprecated
Spring used to ship `ChainedKafkaTransactionManager` for this, and it is deprecated along with its
parent `ChainedTransactionManager`. The reason is worth understanding rather than just obeying:
chaining transaction managers is **best-effort ordering, not distributed coordination**. It commits
one, then the other. If the process dies in between, they disagree, and nothing reconciles them.
It narrows the window. It does not close it. A narrowed window is not a guarantee, and treating it
as one is worse than knowing you have none — because you will skip the idempotency work.
## 5.3 What to do instead
### Option A — idempotent writes (usually right)
Give the work a natural identity and enforce it with a `UNIQUE` constraint. Then a duplicate
attempt is a no-op regardless of *why* it duplicated.
```java
String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset);
if (!repository.existsByDedupeKey(dedupeKey)) {
repository.save(new ProcessedOrder(dedupeKey, payload));
}
```
`topic-partition-offset` is a good default key: Kafka guarantees it is unique and stable, it is on
every consumer record, and it requires no coordination or payload changes. An event ID carried in
the message is better when the same logical event can arrive on more than one topic or be replayed
from a different offset.
Do the check-and-insert inside the database transaction and let the `UNIQUE` constraint be the
real enforcement — the `exists` check is an optimisation, not the guarantee, because two concurrent
attempts can both pass it.
Measured in `DatabaseBoundaryTest`: with a naive insert the listener ran twice and wrote **2 rows**
for one message; with the dedupe key it ran twice and wrote **1**.
### Option B — the transactional outbox (when you cannot be idempotent)
Invert the problem. The database is the source of truth:
1. In **one local database transaction**, write your business change *and* a row into an `outbox`
table.
2. A separate relay — a poller, or change-data-capture such as Debezium — reads the outbox and
publishes to Kafka.
3. The relay is at-least-once, so consumers must tolerate duplicates, which idempotent consumers
already do.
This gives genuine atomicity for the part that matters (business state and the intent to publish),
using only a local transaction. The cost is a table, a relay, and publish latency.
### Option C — make the listener's side effect naturally idempotent
Sometimes the work is already idempotent and nobody noticed: `UPDATE ... SET status='PAID'`,
`PUT /resource/123`, an upsert keyed by business ID, a `SET` in Redis. If the operation is
absorbing rather than accumulating, redelivery is harmless. Prefer designing for this over
detecting duplicates.
## 5.4 Ordering, if you use `@Transactional` anyway
With a `KafkaTransactionManager` on the container and `@Transactional` on the listener method, you
get:
```
container begins Kafka transaction
interceptor begins DB transaction
listener body runs
DB transaction COMMITS <-- first
Kafka transaction COMMITS <-- second
```
The DB commits first, deliberately: if the Kafka commit then fails, the offset is not advanced, the
record is redelivered, and an idempotent DB write absorbs it. Had Kafka committed first, a DB
failure would lose the write entirely with the offset already advanced.
So `@Transactional` is still worth having — it makes the DB write roll back when your listener
throws, which is the common case. It just does not make the two systems atomic, and the crash
window between the two commits is real. Idempotency is what covers it.
## 5.5 The rule
> **Exactly-once processing = at-least-once delivery + idempotent side effects.**
>
> Kafka transactions give you the delivery half. You build the other half.
Anyone who tells you Kafka gives you end-to-end exactly-once across your whole system is describing
Kafka-to-Kafka and has not thought about the database.
---
Next: [06-performance.md](06-performance.md).

96
docs/06-performance.md Normal file
View File

@@ -0,0 +1,96 @@
# 6. What transactions actually cost
> Runnable companion: [`TransactionCostTest`](../src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java)
## 6.1 The useful version of "transactions are slow"
The cost is **per transaction, not per record**.
Committing requires a round trip to the transaction coordinator plus a durable marker write into
every partition the transaction touched, plus acknowledgement of those writes. That is a fixed cost.
It does not care whether the transaction contained one record or ten thousand.
So the question is never "should I use transactions" in isolation. It is **"how many records go in
each one"**.
## 6.2 Measured
10,000 records of 256 bytes, single-node in-JVM broker, median of 5 interleaved rounds,
JDK 25.0.3 / Kafka 4.2.1 on an AMD Ryzen 5 5600U laptop:
| Configuration | records/s | round-to-round spread | vs previous row |
|---|---:|---:|---:|
| no transaction (idempotent only) | 801,880 | 26% | — |
| 1 record / transaction | **711** | 11% | — |
| 10 records / transaction | 8,452 | 12% | **11.9x** |
| 100 records / transaction | 78,770 | 55% | **9.3x** |
| 1,000 records / transaction | 330,969 | 77% | 4.2x |
| 10,000 records / transaction | 878,071 | 32% | 2.7x |
**One record per transaction is roughly 111x slower than one hundred**, and about three orders of
magnitude below the non-transactional baseline.
Each 10x increase in transaction size buys close to 10x throughput while commit cost dominates, then
flattens as per-record cost takes over — the knee is somewhere around 1,000 records here.
## 6.3 How to read these numbers honestly
They come from a laptop and a single-node in-JVM broker, so the absolute throughput is not
meaningful. Two specific caveats:
- The **non-transactional baseline is noisy** (26% spread, and it drifted 30% across the run).
Sending 10,000 records with no commit is fast enough that the measurement is dominated by flush
timing and OS scheduling. Ratios against the baseline are indicative only.
- The **transactional measurements are stable** because each is dominated by a fixed number of
commits. The comparisons *between* transaction sizes are the trustworthy part, and they are the
point.
On a real cluster the per-commit cost is *larger* — real network, real replication, real disk — so
the penalty at small transaction sizes is worse than shown here and batching matters more.
The test itself went through three revisions before producing numbers worth quoting; the reasons are
documented in its source, and they are the usual ones: insufficient warm-up, measurement order
accruing warm-up to whichever configuration ran last, and one path paying for a metadata fetch its
competitor had already done. If you adapt it, keep the interleaving and the drift report.
## 6.4 The setting that controls this in Spring
**`max.poll.records`.** Not anything named "transaction".
With a `KafkaTransactionManager`, the container's transaction boundary is *the batch returned by
`poll()`*. So the number of records per transaction is exactly `max.poll.records` (default 500),
capped by how many are actually available.
```yaml
spring:
kafka:
consumer:
max-poll-records: 500 # == records per transaction
```
Raising it is the single most effective throughput fix for a transactional listener. The price is
that a failure anywhere in the batch rolls back and redoes the whole batch, so you are trading
throughput against the cost of redoing work — and against `max.poll.interval.ms`, since the whole
batch must be processed within it.
## 6.5 The other costs, which are not throughput
- **Latency.** A `read_committed` consumer cannot read past the Last Stable Offset, so downstream
consumers see nothing until the transaction commits. Larger transactions mean higher end-to-end
latency. This is a direct trade against §6.4.
- **Storage.** Every transaction writes a marker per partition. High-frequency small transactions on
many partitions produce a surprising amount of control-record overhead.
- **Redelivery amplification.** One poisoned record in a batch of 500 rolls back all 500.
- **Coordinator load.** Every transaction is state on a broker, in `__transaction_state`.
## 6.6 Rules of thumb
- Do not use one transaction per record. It is three orders of magnitude off the achievable rate.
- Start at the default `max.poll.records=500` and tune from measurements, not from feeling.
- If latency matters more than throughput, lower it deliberately and know what you are buying.
- If you do not need atomic offset+output commits, do not use transactions at all — an idempotent
producer plus an idempotent consumer is cheaper and simpler, and covers most systems.
---
Next: [07-corner-cases.md](07-corner-cases.md).

109
docs/07-corner-cases.md Normal file
View File

@@ -0,0 +1,109 @@
# 7. Corner cases and things that bite
The blog post covers the main concepts. This chapter is the long tail — behaviours that do not fit a
narrative but do show up in production.
## 7.1 `transaction.timeout.ms` vs `max.poll.interval.ms`
The most common "it worked in dev" failure. Your listener must finish a whole poll batch, commit,
and get back to `poll()` inside **both** limits:
| Setting | Default | Owned by |
|---|---|---|
| `transaction.timeout.ms` | 60,000 (Spring sets it; Kafka's producer default is 60s) | producer |
| `transaction.max.timeout.ms` | 900,000 | **broker** — caps the above |
| `max.poll.interval.ms` | 300,000 | consumer |
If processing exceeds `transaction.timeout.ms`, the coordinator aborts your transaction and your
commit fails with `ProducerFencedException` / `InvalidProducerEpochException`. If it exceeds
`max.poll.interval.ms`, the consumer is ejected from the group and the partitions rebalance —
usually producing both symptoms at once and a very confusing log.
Rule: `transaction.timeout.ms` should comfortably exceed the worst-case time to process one batch
of `max.poll.records`, and the broker's `transaction.max.timeout.ms` must be at least as large or
producer creation fails outright.
## 7.2 A long transaction stalls other people's consumers
Covered in [03](03-transactions-and-markers.md) §3.5 and worth repeating because it is
counter-intuitive: a `read_committed` consumer cannot read past the Last Stable Offset, which is
held back by the *oldest open transaction on the partition* — regardless of which producer opened
it. One slow producer degrades every consumer on that partition.
## 7.3 `@KafkaListener` with manual acknowledgment does nothing useful
With a `KafkaTransactionManager` present, `AckMode.MANUAL` / `MANUAL_IMMEDIATE` and calls to
`Acknowledgment.acknowledge()` are irrelevant: the offset commit happens inside the transaction, at
the end of the batch. Leaving manual ack code in place is harmless but misleading — future readers
will believe it controls something.
## 7.4 `executeInTransaction` is not the same as being in a listener transaction
```java
kafkaTemplate.executeInTransaction(t -> t.send(TOPIC, value));
```
This starts a **local** producer transaction. It is the right way to produce transactionally from
outside a listener (a REST controller, a scheduled job, a test). Inside a listener that already has
a container-managed transaction, do **not** call it — plain `send()` enlists in the existing
transaction automatically. Nesting produces `IllegalStateException` or, worse, a second independent
transaction that commits separately.
## 7.5 Error handling interacts with transactions
`DefaultErrorHandler` retries in-process by default and then, if configured, publishes to a dead
letter topic via `DeadLetterPublishingRecoverer`. Two things to know:
- The DLT publish happens **inside the same transaction** when a `KafkaTransactionManager` is
present, so "record moved to DLT and offset advanced" is atomic. That is the behaviour you want,
and it is why a DLT is the correct terminal state for a poison record under EOS.
- Retries consume `max.poll.interval.ms`. A back-off of 10 attempts × 30 s will eject the consumer
from the group long before it gives up. Use `DefaultErrorHandler` with a bounded back-off and let
the DLT absorb the rest.
## 7.6 Producers are pooled, and `transactional.id` suffixes are allocated from that pool
Spring caches producers. With transactions, each concurrent transaction needs its own
`transactional.id`, so Spring allocates `prefix + suffix`. `TransactionIdSuffixStrategy`
(Spring Kafka 3.2+) controls that allocation; the default reuses suffixes as producers return to the
cache. If you see unexpected fencing under high concurrency, look here before looking at your code.
## 7.7 Consumer group `isolation.level` cannot be changed per-record
It is a consumer-level setting. If some records on a topic must be visible before commit and others
after, you need two topics, not two isolation levels.
## 7.8 Compacted topics and transactions
Transaction markers interact with log compaction: a marker cannot be cleaned until the transaction
it belongs to is complete, and a **hanging** transaction therefore blocks compaction of the whole
partition indefinitely. This was one of the motivations for
[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890). If a compacted topic stops
shrinking, check for open transactions before checking your compaction settings.
## 7.9 Exactly-once across two Kafka clusters is not a thing
A transaction is scoped to one cluster's transaction coordinator. MirrorMaker 2 and most replication
tooling are at-least-once across clusters. A cross-cluster pipeline is at-least-once end to end no
matter what each side does internally.
## 7.10 Kafka Streams does this for you
If your topology is genuinely read-process-write over Kafka only, `processing.guarantee=exactly_once_v2`
in Kafka Streams gives you all of the above with none of the wiring, including state store changelogs
in the same transaction. Reach for Spring Kafka transactions when you need to interleave non-Kafka
work or you are not building a streaming topology — not because Streams cannot do it.
## 7.11 Things that are true and often assumed not to be
- **Idempotence and transactions do not require a special broker configuration** beyond replication
factors adequate for `__transaction_state`.
- **`enable.idempotence` is on by default** (Kafka 3.0+) — see [02](02-idempotence.md).
- **A transactional producer can send to topics it never mentioned up front.** Partitions are added
to the transaction dynamically as you send.
- **Reading your own writes** inside a transaction is not possible — a `read_committed` consumer
will not see them until commit, including in the same process.
---
Back to the [README](../README.md).

115
pom.xml Normal file
View File

@@ -0,0 +1,115 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Companion code for "Exactly-Once with Spring Kafka on Spring Boot 4" (ankurm.com).
Everything runs against an in-JVM KRaft broker started by spring-kafka-test, so there is no
Docker requirement and no external Kafka to install:
mvn test
A docker-compose.yml is included for readers who would rather point the same code at a real
broker; see docs/00-running.md.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.0</version>
<relativePath/>
</parent>
<groupId>com.ankurm.kafka</groupId>
<artifactId>spring-kafka-exactly-once-boot4</artifactId>
<version>1.0.0</version>
<name>Exactly-once with Spring Kafka on Spring Boot 4</name>
<description>
Runnable demonstrations of idempotent producers, Kafka transactions, read-process-write,
zombie fencing, transaction markers, and the Kafka/database boundary.
</description>
<properties>
<java.version>25</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<!--
IMPORTANT, and a real Spring Boot 3 -> 4 migration trap.
On Boot 3 you could depend on org.springframework.kafka:spring-kafka directly and still
get auto-configuration, because KafkaAutoConfiguration lived in the monolithic
spring-boot-autoconfigure jar that every application already had.
Boot 4 split auto-configuration into per-technology modules. Kafka's now lives in
org.springframework.boot:spring-boot-kafka. Depend on spring-kafka alone and you get the
library but NO auto-configuration: no ProducerFactory, no KafkaTemplate, no
KafkaTransactionManager. The failure is a confusing NoSuchBeanDefinitionException for
KafkaTemplate at startup, which looks like a generics problem and is not.
spring-boot-starter-kafka pulls in both spring-kafka and spring-boot-kafka.
See docs/01-boot4-setup.md.
-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
<!-- Used only by the module that shows why "Kafka + DB atomically" is not a thing. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!--
spring-kafka-test pulls in the Kafka broker itself. On Spring Kafka 4.x this is
EmbeddedKafkaKraftBroker: Kafka 4.x removed ZooKeeper entirely, so the old
ZooKeeper-based embedded broker no longer exists.
-->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!--
The demos print narrated output that is meant to be read. Without this,
Surefire swallows stdout unless a test fails.
-->
<redirectTestOutputToFile>false</redirectTestOutputToFile>
<trimStackTrace>false</trimStackTrace>
</configuration>
</plugin>
</plugins>
</build>
</project>

690
results/full-run.txt Normal file
View File

@@ -0,0 +1,690 @@
[INFO] Scanning for projects...
[INFO]
[INFO] ----------< com.ankurm.kafka:spring-kafka-exactly-once-boot4 >----------
[INFO] Building Exactly-once with Spring Kafka on Spring Boot 4 1.0.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.5.0:resources (default-resources) @ spring-kafka-exactly-once-boot4 ---
[INFO] Copying 1 resource from src\main\resources to target\classes
[INFO] Copying 0 resource from src\main\resources to target\classes
[INFO]
[INFO] --- compiler:3.15.0:compile (default-compile) @ spring-kafka-exactly-once-boot4 ---
[INFO] Nothing to compile - all classes are up to date.
[INFO]
[INFO] --- resources:3.5.0:testResources (default-testResources) @ spring-kafka-exactly-once-boot4 ---
[INFO] Copying 1 resource from src\test\resources to target\test-classes
[INFO]
[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ spring-kafka-exactly-once-boot4 ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 8 source files with javac [debug parameters release 25] to target\test-classes
[INFO]
[INFO] --- surefire:3.5.6:test (default-test) @ spring-kafka-exactly-once-boot4 ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.ankurm.kafka.eos.SmokeTest
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-15096196392826061749\combined_0_0 with metadata.version 4.3-IV0.
22:22:09.287 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
========================================================================
Environment
========================================================================
java.version : 25.0.3
kafka-clients : 4.2.1
embedded broker : EmbeddedKafkaKraftBroker
bootstrap servers : localhost:63356
cluster id : JmSQ3q0IRuilOAxcUgBP6w
nodes : 1
topics : [smoke]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.729 s -- in com.ankurm.kafka.eos.SmokeTest
[INFO] Running com.ankurm.kafka.eos._01_idempotence.IdempotentProducerTest
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-9366343747489431922\combined_0_0 with metadata.version 4.3-IV0.
22:22:10.808 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
==============================================================================
1. Idempotence is on by default (Kafka 3.0+) -- verified from the broker
==============================================================================
-- What the client library says the defaults are -----------------------------
enable.idempotence : true
acks : all
-- Sending 3 records with a completely default producer ----------------------
-- Asking the BROKER what producer state it is tracking for idem-default -----
producerId=0 producerEpoch=0 lastSequence=2
producer states tracked : 1
>> The broker holds a real producer ID and the last sequence number it accepted.
>> That IS the de-duplication state, and you configured nothing to get it.
==============================================================================
2. acks=1 silently disables idempotence -- no exception, no warning
==============================================================================
-- Constructing a producer with ONLY acks=1 ----------------------------------
props: acks=1 (enable.idempotence left at its default of true)
producer constructed successfully -- note that nothing complained
-- Asking the broker for producer state on idem-acks1 ------------------------
producer states tracked : 0
>> Three records were written, and the broker is tracking NOTHING for them.
>> Kafka resolved acks=1 against enable.idempotence=true by turning
>> idempotence off, and said nothing at all. Every record this producer
>> sends can now be duplicated by an ordinary retry.
==============================================================================
3. acks=1 + enable.idempotence=true is rejected at construction
==============================================================================
-- Same acks=1, but this time idempotence is requested explicitly ------------
ConfigException: Must set acks to all in order to use the idempotent producer. Otherwise we cannot guarantee idempotence.
>> Ask for it explicitly and Kafka refuses. Leave it implicit and Kafka
>> downgrades you. So setting enable.idempotence=true explicitly is worth
>> doing purely as an assertion: it turns a silent downgrade into a
>> startup failure, which is the failure you want.
==============================================================================
4. The other constraints, and which ones fail loudly
==============================================================================
-- max.in.flight.requests.per.connection = 6 ---------------------------------
rejected: To use the idempotent producer, max.in.flight.requests.per.connection must be set to at most 5. Current value is 6.
-- retries = 0 ---------------------------------------------------------------
rejected: Must set retries to non-zero when using the idempotent producer.
-- max.in.flight = 5 with idempotence -- allowed -----------------------------
constructed successfully
>> Idempotence requires acks=all, retries>0 and max.in.flight<=5.
>> Five in flight is safe because the broker can reorder within that window
>> using sequence numbers. WITHOUT idempotence, max.in.flight>1 silently
>> breaks ORDERING on retry: a retried batch can land after a later one.
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.723 s -- in com.ankurm.kafka.eos._01_idempotence.IdempotentProducerTest
[INFO] Running com.ankurm.kafka.eos._02_transactions.TransactionMarkerOffsetTest
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-5226944134061393373\combined_0_0 with metadata.version 4.3-IV0.
22:22:11.504 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
==============================================================================
1. A committed transaction burns one extra offset per partition
==============================================================================
-- Before: empty topic -------------------------------------------------------
endOffset = 0
-- Sending 5 records inside ONE transaction, then committing -----------------
sent record 0 -> offset 0
sent record 1 -> offset 1
sent record 2 -> offset 2
sent record 3 -> offset 3
sent record 4 -> offset 4
-- After commit --------------------------------------------------------------
records sent : 5
endOffset : 6 <-- 5 records occupy 0..4, the commit marker takes 5
-- What a read_committed consumer actually receives --------------------------
22:22:16.224 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator.
22:22:16.442 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator.
22:22:16.753 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator.
offset=0 value=committed-0
offset=1 value=committed-1
offset=2 value=committed-2
offset=3 value=committed-3
offset=4 value=committed-4
records delivered : 5
>> 5 records were delivered, the last at offset 4, but the end offset is 6.
>> The gap at offset 5 is the commit marker. It exists, it is durable, and
>> no consumer will ever see it.
==============================================================================
2. An aborted transaction still consumes offsets and disk
==============================================================================
endOffset before : 6
-- Sending 3 records inside a transaction, then ABORTING ---------------------
sent record 0 -> offset 6
sent record 1 -> offset 7
sent record 2 -> offset 8
abortTransaction() called
-- After abort ---------------------------------------------------------------
endOffset after : 10 (+4 for 3 aborted records and 1 abort marker)
-- read_committed consumer ---------------------------------------------------
delivered : 5 records, values=[committed-0, committed-1, committed-2, committed-3, committed-4]
-- read_uncommitted consumer -- the SAME partition ---------------------------
delivered : 8 records, values=[committed-0, committed-1, committed-2, committed-3, committed-4, aborted-0, aborted-1, aborted-2]
>> The aborted records are physically in the log with permanent offsets.
>> isolation.level is a CONSUMER-side filter, not a broker-side delete.
>> A default consumer (read_uncommitted) reads data you deliberately aborted.
==============================================================================
3. Lag is NOT inflated -- but counting records by offset arithmetic is broken
==============================================================================
-- A read_committed consumer drains the partition, then commits --------------
-- The lag calculation everyone worries about --------------------------------
partition end offset : 10
consumer committed offset : 10
lag (end - committed) : 0
>> Lag is ZERO. The consumer's position advances over markers and aborted
>> records just as it advances over delivered ones, so the standard lag
>> formula stays correct. The common claim that transactions inflate lag
>> does not survive contact with a broker.
-- The calculation that IS broken --------------------------------------------
beginningOffset : 0
endOffset : 10
end - beginning : 10 <-- looks like the record count
records actually retrievable: 5
offset of last record : 4
overstatement : 100%
>> Offset span says 10 records. Only 5 can ever be read -- a 100% error,
>> from 2 markers and 3 aborted records. Anything that counts events by
>> subtracting offsets -- retention math, throughput dashboards, tests that
>> assert on offsets -- is wrong by however much your abort rate happens to be.
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.20 s -- in com.ankurm.kafka.eos._02_transactions.TransactionMarkerOffsetTest
[INFO] Running com.ankurm.kafka.eos._03_read_process_write.ReadProcessWriteTest
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v4.1.0)
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-7564387337135394901\combined_0_0 with metadata.version 4.3-IV0.
22:22:27.088 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
22:22:27.249 INFO c.a.k.e._.ReadProcessWriteTest - Starting ReadProcessWriteTest using Java 25.0.3 with PID 38092 (started by Ankur in C:\Users\Ankur\ankurm-blog-tools\projects\spring-kafka-exactly-once-boot4)
22:22:27.250 INFO c.a.k.e._.ReadProcessWriteTest - No active profile set, falling back to 1 default profile: "default"
22:22:30.619 INFO c.a.k.e._.ReadProcessWriteTest - Started ReadProcessWriteTest in 3.874 seconds (process running for 23.722)
==============================================================================
1. Read-process-write, happy path
==============================================================================
-- Publishing 3 orders to orders.incoming ------------------------------------
22:22:30.756 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:30.855 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:30.957 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:31.059 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:31.162 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:31.263 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
sent: alpha
sent: beta
sent: gamma
22:22:32.224 INFO c.a.k.e.o.OrderProcessor - processing 'alpha' (invocation #1)
22:22:32.351 INFO c.a.k.e.o.OrderProcessor - processing 'beta' (invocation #2)
22:22:32.462 INFO c.a.k.e.o.OrderProcessor - processing 'gamma' (invocation #3)
-- Listener invocations ------------------------------------------------------
processed: alpha
processed: beta
processed: gamma
-- What a read_committed consumer sees on orders.validated -------------------
offset=0 value=ALPHA
offset=2 value=BETA
offset=4 value=GAMMA
>> 3 in, 3 out. The listener body contains no transaction code at all --
>> the guarantee comes entirely from spring.kafka.producer.transaction-id-prefix.
==============================================================================
2. Failure AFTER the send: processed twice, delivered once
==============================================================================
-- Poisoning 'delta' so its first processing attempt throws ------------------
sent: delta
-- Waiting for the listener to be invoked more than once ---------------------
22:22:35.108 INFO c.a.k.e.o.OrderProcessor - processing 'delta' (invocation #1)
22:22:35.111 ERROR o.s.k.s.LoggingProducerListener - Exception thrown when sending a message with key='null' and payload='DELTA' to topic orders.validated:
org.apache.kafka.common.errors.TransactionAbortedException: Failing batch since transaction was aborted
22:22:35.116 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos.orders.OrderProcessor.process(java.lang.String)' threw exception
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700)
at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825)
at java.base/java.lang.Thread.run(Thread.java:1474)
Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
Caused by: java.lang.IllegalStateException: simulated processing failure for 'delta'
at com.ankurm.kafka.eos.orders.OrderProcessor.process(OrderProcessor.java:82)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168)
at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
... 15 common frames omitted
22:22:35.613 INFO c.a.k.e.o.OrderProcessor - processing 'delta' (invocation #2)
times the listener ran for 'delta' : 2
-- What a read_committed consumer sees ---------------------------------------
DELTA records visible : 1
-- What a read_uncommitted consumer sees on the SAME topic -------------------
DELTA records physically in the log : 1
>> The listener ran 2 times and a read_committed consumer sees exactly 1.
>>
>> This is the whole idea: exactly-once is a property of what is COMMITTED,
>> not of how many times your code ran. Any side effect your listener performs
>> outside Kafka -- a DB write, an HTTP call, an email -- happened 2 times.
-- A detail worth noticing in the number above -------------------------------
read_uncommitted sees 1 DELTA record(s), not 2
>> The rolled-back send never reached the broker at all. KafkaTemplate.send()
>> is asynchronous: the record sat in the producer's accumulator, and
>> abortTransaction() discarded the un-flushed batch instead of writing it.
>>
>> So aborted records reach the log only if they were already flushed -- which
>> happens under load, on large batches, or when you block on the send future.
>> TransactionMarkerOffsetTest forces exactly that case by calling get() on
>> each send, and there the aborted records DO occupy permanent offsets.
>> Do not rely on either behaviour: rely on read_committed.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.17 s -- in com.ankurm.kafka.eos._03_read_process_write.ReadProcessWriteTest
[INFO] Running com.ankurm.kafka.eos._04_fencing.ZombieFencingTest
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-2096764983482195835\combined_0_0 with metadata.version 4.3-IV0.
22:22:42.230 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
==============================================================================
1. A second producer with the same transactional.id fences the first
==============================================================================
-- Producer A ('the zombie') starts a transaction and sends ------------------
transactional.id = 'shared-tx-id'
record sent, transaction still OPEN
-- Producer B starts with the SAME transactional.id --------------------------
initTransactions() returned -- the coordinator has bumped the epoch
any in-flight transaction under the old epoch is now aborted
-- Producer A tries to commit ------------------------------------------------
REJECTED with ProducerFencedException
message: There is a newer producer with the same transactionalId which fences the current one.
-- Producer B commits normally -----------------------------------------------
committed successfully
>> The zombie could not commit. Its work was discarded by the broker,
>> not by any code you wrote. That is the entire value of transactional.id:
>> a durable identity the coordinator can fence on.
>>
>> Note the exception type -- on Kafka 4.x with KIP-890 (transaction.version=2)
>> the epoch is bumped every transaction, so you may see
>> InvalidProducerEpochException where older articles promise
>> ProducerFencedException. Catch both, or catch neither and let the
>> container restart the producer, which is what Spring does.
==============================================================================
2. A random transactional.id per start-up defeats fencing entirely
==============================================================================
-- Producer A with a UUID-based transactional.id -----------------------------
transactional.id = 'processor-9900b3b5-0861-4908-93a5-4d4b64c83ca7'
-- Producer B restarts and generates a NEW UUID ------------------------------
transactional.id = 'processor-1dd8cdf3-8ddf-4d3d-b28b-152284c558eb'
B committed
-- Now the 'zombie' A commits -- does anything stop it? ----------------------
A committed successfully. Nothing was fenced.
>> Both producers committed. The coordinator saw two unrelated identities
>> and had no reason to fence either one.
>>
>> This is the most damaging Kafka transactions misconfiguration, because
>> everything appears to work: transactions commit, tests pass, and you have
>> exactly the duplicate-processing problem you turned transactions on to
>> prevent. The transactional.id must be STABLE across restarts and tied to
>> the partition set an instance owns -- not to the process lifetime.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 s -- in com.ankurm.kafka.eos._04_fencing.ZombieFencingTest
[INFO] Running com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v4.1.0)
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-17444936701909163355\combined_0_0 with metadata.version 4.3-IV0.
22:22:43.855 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
22:22:44.000 INFO c.a.k.e._.DatabaseBoundaryTest - Starting DatabaseBoundaryTest using Java 25.0.3 with PID 38092 (started by Ankur in C:\Users\Ankur\ankurm-blog-tools\projects\spring-kafka-exactly-once-boot4)
22:22:44.000 INFO c.a.k.e._.DatabaseBoundaryTest - No active profile set, falling back to 1 default profile: "default"
22:22:44.452 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION}
22:22:44.454 INFO c.a.k.e._.DatabaseBoundaryTest - Started DatabaseBoundaryTest in 0.704 seconds (process running for 37.556)
22:22:44.574 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 5 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION}
22:22:44.575 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for orders.incoming with error 'TOPIC_ALREADY_EXISTS': Topic 'orders.incoming' already exists.
==============================================================================
1. The problem: one row per ATTEMPT, not per message
==============================================================================
-- Sending one message that will fail on its first processing attempt --------
22:22:44.765 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 15 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION}
22:22:44.765 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for orders.incoming with error 'TOPIC_ALREADY_EXISTS': Topic 'orders.incoming' already exists.
22:22:44.797 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:44.900 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:45.002 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:45.104 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists.
22:22:45.975 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$NaiveDbListener - naive listener wrote a row for 'order-1'
22:22:45.975 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$NaiveDbListener.onMessage(java.lang.String)' threw exception
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700)
at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825)
at java.base/java.lang.Thread.run(Thread.java:1474)
Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
Caused by: java.lang.IllegalStateException: simulated failure after the DB write for order-1
at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$Base.maybeFail(DatabaseBoundaryTest.java:209)
at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$NaiveDbListener.onMessage(DatabaseBoundaryTest.java:243)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168)
at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
... 15 common frames omitted
22:22:46.438 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$NaiveDbListener - naive listener wrote a row for 'order-1'
listener attempts for 'order-1' : 2
rows in the database : 2
id=1 dedupeKey=naive-1
id=2 dedupeKey=naive-2
>> The Kafka transaction rolled back correctly. The database row did not,
>> because it was never in the Kafka transaction and never could have been.
>>
>> Note that nothing here is misconfigured. transaction-id-prefix is set,
>> isolation.level is read_committed, the Kafka side is exactly-once. The
>> database still has 2 rows for 1 message.
==============================================================================
2. The fix: make the side effect idempotent, not the delivery
==============================================================================
-- Same failure, but the listener dedupes on topic-partition-offset ----------
22:22:48.940 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$IdempotentDbListener - idempotent listener wrote a row for 'order-2' as db.idempotent-0-0
22:22:48.941 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$IdempotentDbListener.onMessage(java.lang.String,java.lang.String,int,long)' threw exception
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700)
at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825)
at java.base/java.lang.Thread.run(Thread.java:1474)
Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
Caused by: java.lang.IllegalStateException: simulated failure after the DB write for order-2
at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$Base.maybeFail(DatabaseBoundaryTest.java:209)
at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$IdempotentDbListener.onMessage(DatabaseBoundaryTest.java:275)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168)
at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489)
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92)
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018)
... 15 common frames omitted
22:22:49.423 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$IdempotentDbListener - idempotent listener SKIPPED 'order-2' -- db.idempotent-0-0 already processed
listener attempts for 'order-2' : 2
rows in the database : 1
id=3 dedupeKey=db.idempotent-0-0
>> The listener ran 2 times and wrote 1 row.
>>
>> The dedupe key is 'topic-partition-offset', which Kafka guarantees is
>> unique and stable. A UNIQUE constraint on that column turns the second
>> attempt into a no-op -- and, importantly, would still do so if the
>> duplicate came from a different process, a rebalance, or a replay.
>>
>> This is the whole design rule: exactly-once processing is achieved by
>> idempotent side effects plus at-least-once delivery. Kafka transactions
>> give you the second half. You have to build the first half yourself.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.701 s -- in com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest
[INFO] Running com.ankurm.kafka.eos._06_performance.TransactionCostTest
Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command)
Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-8432095967619283703\combined_0_0 with metadata.version 4.3-IV0.
22:22:52.001 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'.
==============================================================================
The cost of a transaction is paid per COMMIT, not per record
==============================================================================
Sending 10,000 records of 256 bytes in every configuration.
-- warm-up (discarded) -------------------------------------------------------
done
-- 5 interleaved rounds of 10,000 records (round 0 discarded) ----------------
round 0 (discarded): baseline=311,056 1/tx=349 10/tx=8,032 100/tx=66,664 1000/tx=344,474 10000/tx=641,264
round 1: baseline=628,417 1/tx=765 10/tx=8,452 100/tx=61,437 1000/tx=434,245 10000/tx=836,617
round 2: baseline=801,880 1/tx=685 10/tx=8,499 100/tx=78,770 1000/tx=216,065 10000/tx=878,071
round 3: baseline=760,537 1/tx=718 10/tx=7,467 100/tx=104,725 1000/tx=235,578 10000/tx=942,649
round 4: baseline=833,848 1/tx=711 10/tx=8,357 100/tx=71,036 1000/tx=471,918 10000/tx=897,892
round 5: baseline=818,940 1/tx=692 10/tx=8,491 100/tx=97,746 1000/tx=330,969 10000/tx=657,960
-- measurement stability -----------------------------------------------------
no transaction (idempotent) median=801,880 spread=+26%
1 records/transaction median=711 spread=+11%
10 records/transaction median=8,452 spread=+12%
100 records/transaction median=78,770 spread=+55%
1,000 records/transaction median=330,969 spread=+77%
10,000 records/transaction median=878,071 spread=+32%
baseline first counted round : 628,417 records/s
baseline last counted round : 818,940 records/s
baseline drift : +30.3%
-- summary (median of 5 rounds) ----------------------------------------------
configuration records/s vs baseline
no transaction (idempotent) 801,880 100%
1 records/transaction 711 0%
10 records/transaction 8,452 1%
100 records/transaction 78,770 10%
1,000 records/transaction 330,969 41%
10,000 records/transaction 878,071 110%
>> Records per transaction, and what it buys:
>> 1 -> 711 rec/s
>> 10 -> 8,452 rec/s (11.9x)
>> 100 -> 78,770 rec/s (9.3x)
>> 1,000 -> 330,969 rec/s (4.2x)
>>
>> Each 10x increase in transaction size buys close to 10x throughput while
>> commit cost dominates, then flattens as the per-record cost takes over.
>> One record per transaction is roughly 111x slower than a hundred.
>>
>> (The non-transactional baseline is quoted at 801,880 rec/s but varied by
>> 26% across rounds, so ratios against it are indicative only. The
>> comparisons BETWEEN transaction sizes are the stable, meaningful ones.)
>>
>> The lesson is not 'transactions are slow'. It is that a commit is a fixed
>> cost, so the number of records you put inside one is the setting that
>> matters. In Spring, that number is the size of the batch returned by
>> poll(), which you control with max.poll.records -- NOT with anything
>> named 'transaction'. Raising max.poll.records is the single most
>> effective throughput fix for a transactional listener, at the price of
>> redoing more work when one record in the batch fails.
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 112.4 s -- in com.ankurm.kafka.eos._06_performance.TransactionCostTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:46 min
[INFO] Finished at: 2026-07-31T22:24:48+05:30
[INFO] ------------------------------------------------------------------------

20
src/main/java/com/ankurm/kafka/eos/Application.java Normal file
View File

@@ -0,0 +1,20 @@
package com.ankurm.kafka.eos;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* Companion application for <a href="https://ankurm.com/spring-kafka-exactly-once-boot-4/">
* Exactly-Once with Spring Kafka on Spring Boot 4</a>.
*
* <p>The interesting code is in {@code orders/} and the demonstrations are in {@code src/test}.
* Run {@code mvn test} -- every test starts its own in-JVM KRaft broker, so there is nothing to
* install and nothing to start.
*/
@SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}

66
src/main/java/com/ankurm/kafka/eos/db/ProcessedOrder.java Normal file
View File

@@ -0,0 +1,66 @@
package com.ankurm.kafka.eos.db;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import java.time.Instant;
/**
* A row written by a Kafka listener -- the side effect that Kafka transactions cannot protect.
*
* <p>Note {@link #dedupeKey}: it is the natural identity of the work, not a surrogate. Making that
* column {@code UNIQUE} is what turns a non-idempotent side effect into an idempotent one, which is
* the only reliable way to survive the redelivery that exactly-once processing guarantees you will
* eventually get. See {@code docs/05-database-boundary.md}.
*/
@Entity
@Table(name = "processed_order")
public class ProcessedOrder {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
/**
* Unique business identity of this unit of work.
*
* <p>In a real system this is usually either an event ID carried in the message, or the
* {@code topic-partition-offset} triple, which Kafka guarantees is unique and stable.
*/
@Column(name = "dedupe_key", nullable = false, unique = true)
private String dedupeKey;
@Column(name = "payload", nullable = false)
private String payload;
@Column(name = "written_at", nullable = false)
private Instant writtenAt = Instant.now();
protected ProcessedOrder() {
}
public ProcessedOrder(String dedupeKey, String payload) {
this.dedupeKey = dedupeKey;
this.payload = payload;
}
public Long getId() {
return id;
}
public String getDedupeKey() {
return dedupeKey;
}
public String getPayload() {
return payload;
}
public Instant getWrittenAt() {
return writtenAt;
}
}

12
src/main/java/com/ankurm/kafka/eos/db/ProcessedOrderRepository.java Normal file
View File

@@ -0,0 +1,12 @@
package com.ankurm.kafka.eos.db;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.List;
public interface ProcessedOrderRepository extends JpaRepository<ProcessedOrder, Long> {
boolean existsByDedupeKey(String dedupeKey);
List<ProcessedOrder> findByPayload(String payload);
}

99
src/main/java/com/ankurm/kafka/eos/orders/OrderProcessor.java Normal file
View File

@@ -0,0 +1,99 @@
package com.ankurm.kafka.eos.orders;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
/**
* The canonical <b>read-process-write</b> loop, which is the only thing Kafka's exactly-once
* semantics actually guarantee.
*
* <h2>What makes this exactly-once</h2>
* Three things have to happen atomically for this to be correct:
*
* <ol>
* <li>the output record is written to {@code orders.validated},</li>
* <li>the <b>consumer offset</b> for {@code orders.incoming} is advanced,</li>
* <li>both are visible to downstream {@code read_committed} consumers, or neither is.</li>
* </ol>
*
* <p>Point 2 is the one people miss. The offset commit is not a separate step that happens after
* the send -- it is written <em>into the same Kafka transaction</em>, via
* {@code sendOffsetsToTransaction}. That is why a crash between "produced output" and "committed
* offset" cannot produce a duplicate: there is no such between.
*
* <p><b>You do not write any of that.</b> Setting
* {@code spring.kafka.producer.transaction-id-prefix} makes Spring Boot auto-configure a
* {@code KafkaTransactionManager} and attach it to the listener container. The container then
* begins a transaction before invoking this method, sends the offsets into it, and commits -- or
* aborts the whole thing if this method throws. The method body below is deliberately ordinary; the
* guarantee is entirely in the configuration.
*
* <h2>What it does NOT guarantee</h2>
* Nothing outside Kafka. If this method also wrote to a database, called a payment API, or sent an
* email, those are not in the transaction and will be repeated on redelivery. See
* {@code docs/05-database-boundary.md}.
*/
@Component
public class OrderProcessor {
private static final Logger log = LoggerFactory.getLogger(OrderProcessor.class);
public static final String INPUT_TOPIC = "orders.incoming";
public static final String OUTPUT_TOPIC = "orders.validated";
private final KafkaTemplate<String, String> kafkaTemplate;
/**
* Every invocation is recorded, including the ones that were rolled back. Comparing this list
* against what a {@code read_committed} consumer can see on the output topic is how the tests
* show the difference between "processed" and "committed".
*/
private final List<String> invocations = new CopyOnWriteArrayList<>();
/** Values that should fail on their first N attempts, to exercise rollback and redelivery. */
private final List<String> poisoned = new CopyOnWriteArrayList<>();
public OrderProcessor(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
@KafkaListener(topics = INPUT_TOPIC, groupId = "order-processor")
public void process(String order) {
invocations.add(order);
log.info("processing '{}' (invocation #{})", order, invocations.size());
// This send joins the transaction the container already started. There is no
// "beginTransaction" here and there must not be -- KafkaTemplate detects the active
// transaction and enlists in it.
kafkaTemplate.send(OUTPUT_TOPIC, order.toUpperCase());
if (poisoned.contains(order)) {
poisoned.remove(order);
// Throwing after the send is the interesting case: the output record has already been
// handed to the producer. Because the transaction aborts, no read_committed consumer
// will ever see it -- but it IS in the log, with an offset. See
// TransactionMarkerOffsetTest.
throw new IllegalStateException("simulated processing failure for '" + order + "'");
}
}
/** Marks a value so that its next processing attempt throws. */
public void poison(String value) {
poisoned.add(value);
}
public List<String> invocations() {
return List.copyOf(invocations);
}
public void reset() {
invocations.clear();
poisoned.clear();
}
}

83
src/main/resources/application.yml Normal file
View File

@@ -0,0 +1,83 @@
#
# Exactly-once configuration for Spring Boot 4.1 / Spring Kafka 4.1.
#
# The striking thing about this file is how little of it there is. Setting
# `transaction-id-prefix` is the whole switch: Spring Boot sees it, auto-configures a
# KafkaTransactionManager, and wires it into every listener container. Offsets then travel
# inside the Kafka transaction instead of being committed separately.
#
spring:
application:
name: spring-kafka-exactly-once
kafka:
# Overridden by EmbeddedKafkaBroker during tests via spring.embedded.kafka.brokers.
bootstrap-servers: ${spring.embedded.kafka.brokers:localhost:9092}
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
# ---------------------------------------------------------------------------------
# THE line. Everything else on this page is either a default worth stating out loud
# or a consequence of this.
#
# Each application instance needs a UNIQUE transactional.id, because that identity is
# what the broker fences on. Spring appends a suffix per producer; see
# TransactionIdSuffixStrategy (Spring Kafka 3.2+) if you need to control it, and
# docs/04-fencing.md for what goes wrong when two instances share one.
# ---------------------------------------------------------------------------------
transaction-id-prefix: order-processor-tx-
# Defaults since Kafka 3.0, restated so that nobody "optimises" acks to 1 later and
# silently drops to at-least-once. See _01_idempotence/IdempotentProducerTest.
acks: all
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 5
# A transaction left open blocks read_committed consumers on that partition until it
# times out, so this is a consumer-latency setting as much as a producer one.
# It must be larger than the time one poll batch takes to process, and the broker's
# transaction.max.timeout.ms caps it.
transaction.timeout.ms: 30000
consumer:
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
auto-offset-reset: earliest
# ---------------------------------------------------------------------------------
# The default is read_uncommitted, which means a consumer reading a transactional
# topic will happily deliver records from transactions that were ABORTED. Producing
# transactionally without setting this on the consumer side gives you the cost of
# transactions and none of the benefit.
# ---------------------------------------------------------------------------------
isolation-level: read_committed
# With a transaction manager in play the container manages offsets inside the
# transaction; auto-commit must be off (Spring Boot defaults it off, stated for clarity).
enable-auto-commit: false
listener:
# RECORD/BATCH ack modes are irrelevant when a KafkaTransactionManager is present -- the
# transaction boundary is the batch returned by poll(). Left explicit so the interaction
# is visible rather than surprising.
ack-mode: batch
# Fail fast in tests rather than blocking forever if a topic is missing.
missing-topics-fatal: false
# Only used by the module that shows why "Kafka and the database, atomically" is not a thing.
datasource:
url: jdbc:h2:mem:eos;DB_CLOSE_DELAY=-1
driver-class-name: org.h2.Driver
username: sa
password: ""
jpa:
hibernate:
ddl-auto: create-drop
open-in-view: false
logging:
level:
com.ankurm.kafka: INFO

51
src/test/java/com/ankurm/kafka/eos/SmokeTest.java Normal file
View File

@@ -0,0 +1,51 @@
package com.ankurm.kafka.eos;
import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.common.utils.AppInfoParser;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Proves the toolchain works before anything interesting is attempted: an in-JVM KRaft broker
* starts, an admin client connects, and the versions are what we think they are.
*
* <p>If this test fails, nothing else in the repository will work, and the reason is almost always
* one of: a JDK older than 17, a missing {@code spring-kafka-test} dependency, or a corporate
* proxy that stopped Maven resolving the broker jar.
*/
@SpringJUnitConfig
@EmbeddedKafka(partitions = 1, topics = {"smoke"})
class SmokeTest {
@Autowired
EmbeddedKafkaBroker broker;
@Test
void embeddedKraftBrokerStarts() throws Exception {
System.out.println("=".repeat(72));
System.out.println("Environment");
System.out.println("=".repeat(72));
System.out.printf("java.version : %s%n", System.getProperty("java.version"));
System.out.printf("kafka-clients : %s%n", AppInfoParser.getVersion());
System.out.printf("embedded broker : %s%n", broker.getClass().getSimpleName());
System.out.printf("bootstrap servers : %s%n", broker.getBrokersAsString());
try (Admin admin = Admin.create(Map.of("bootstrap.servers", broker.getBrokersAsString()))) {
var clusterId = admin.describeCluster().clusterId().get();
var nodes = admin.describeCluster().nodes().get();
System.out.printf("cluster id : %s%n", clusterId);
System.out.printf("nodes : %d%n", nodes.size());
System.out.printf("topics : %s%n", admin.listTopics().names().get());
assertThat(nodes).isNotEmpty();
}
System.out.println();
}
}

243
src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java Normal file
View File

@@ -0,0 +1,243 @@
package com.ankurm.kafka.eos._01_idempotence;
import com.ankurm.kafka.eos.support.Report;
import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.ProducerState;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.config.ConfigException;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
* <h2>Idempotence: the half of exactly-once you get for free, and the config that silently
* takes it away</h2>
*
* <p>An <b>idempotent producer</b> solves one specific problem: a producer sends a record, the
* broker writes it, and the acknowledgement is lost on the way back. The producer retries. Without
* idempotence the broker now holds the record twice, and no amount of consumer-side care can undo
* that -- the duplicate is a real, distinct record with its own offset.
*
* <p>The fix is a producer ID (PID) plus a per-partition sequence number. The broker remembers
* recent sequence numbers per PID per partition and rejects a repeat. This costs nothing measurable
* and has been <b>on by default since Kafka 3.0</b>.
*
* <h3>What this test actually proves</h3>
* Most articles assert that idempotence is enabled and move on. This test <em>verifies it from the
* broker's side</em> using {@link Admin#describeProducers}, which reports the idempotence
* bookkeeping the broker holds for a partition: producer ID, epoch, last sequence number. A
* non-idempotent producer creates no such state, because there is nothing to de-duplicate against.
* That is the difference between reading a config value back and observing a behaviour.
*
* <p><b>Each scenario uses its own topic.</b> Broker-side producer state is per partition and
* outlives the producer that created it, so sharing one topic across these scenarios would leak
* state from the first into the second and produce a very convincing wrong answer. An earlier
* version of this file did exactly that.
*
* <h3>The trap</h3>
* {@code acks} and {@code enable.idempotence} interact, and they interact differently depending on
* whether you set idempotence explicitly:
*
* <ul>
* <li>Set <b>only</b> {@code acks=1} -- idempotence is <b>silently disabled</b>. No warning that
* stops a deployment, no exception. You simply lose the guarantee.</li>
* <li>Set {@code acks=1} <b>and</b> {@code enable.idempotence=true} -- you get a
* {@link ConfigException} at construction. Kafka refuses the impossible combination only when
* you asked for it explicitly.</li>
* </ul>
*
* <p>The first case is the dangerous one, and it is common: someone sets {@code acks=1} for latency,
* in a properties file, years ago, and the system has been quietly at-least-once ever since.
*/
@SpringJUnitConfig
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@EmbeddedKafka(partitions = 1, topics = {
IdempotentProducerTest.TOPIC_DEFAULT,
IdempotentProducerTest.TOPIC_ACKS1,
IdempotentProducerTest.TOPIC_EXPLICIT
})
class IdempotentProducerTest {
static final String TOPIC_DEFAULT = "idem-default";
static final String TOPIC_ACKS1 = "idem-acks1";
static final String TOPIC_EXPLICIT = "idem-explicit";
@Autowired
EmbeddedKafkaBroker broker;
// ---------------------------------------------------------------------------------------
// 1. The default
// ---------------------------------------------------------------------------------------
@Test
@Order(1)
void idempotenceIsOnByDefaultAndTheBrokerCanSeeIt() throws Exception {
Report.title("1. Idempotence is on by default (Kafka 3.0+) -- verified from the broker");
Report.section("What the client library says the defaults are");
Report.bullet("enable.idempotence : %s",
ProducerConfig.configDef().defaultValues().get(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG));
Report.bullet("acks : %s",
ProducerConfig.configDef().defaultValues().get(ProducerConfig.ACKS_CONFIG));
Report.section("Sending 3 records with a completely default producer");
try (Producer<String, String> producer = producer(Map.of())) {
for (int i = 0; i < 3; i++) {
producer.send(new ProducerRecord<>(TOPIC_DEFAULT, "key", "default-" + i)).get();
}
}
Report.section("Asking the BROKER what producer state it is tracking for " + TOPIC_DEFAULT);
List<ProducerState> states = describeProducers(TOPIC_DEFAULT);
states.forEach(s -> Report.bullet("producerId=%d producerEpoch=%d lastSequence=%d",
s.producerId(), s.producerEpoch(), s.lastSequence()));
Report.bullet("producer states tracked : %d", states.size());
assertThat(states)
.as("an idempotent producer registers a producer ID with the broker")
.isNotEmpty();
assertThat(states.getFirst().producerId()).isNotEqualTo(-1L);
assertThat(states.getFirst().lastSequence())
.as("three records were sent, so sequence numbers 0,1,2 were used")
.isEqualTo(2);
Report.takeaway("The broker holds a real producer ID and the last sequence number it accepted.");
Report.takeaway("That IS the de-duplication state, and you configured nothing to get it.");
}
// ---------------------------------------------------------------------------------------
// 2. The silent downgrade
// ---------------------------------------------------------------------------------------
@Test
@Order(2)
void acksOneSilentlyDisablesIdempotence() throws Exception {
Report.title("2. acks=1 silently disables idempotence -- no exception, no warning");
Report.section("Constructing a producer with ONLY acks=1");
Report.bullet("props: acks=1 (enable.idempotence left at its default of true)");
try (Producer<String, String> producer = producer(Map.of(ProducerConfig.ACKS_CONFIG, "1"))) {
Report.bullet("producer constructed successfully -- note that nothing complained");
for (int i = 0; i < 3; i++) {
producer.send(new ProducerRecord<>(TOPIC_ACKS1, "key", "acks1-" + i)).get();
}
}
Report.section("Asking the broker for producer state on " + TOPIC_ACKS1);
List<ProducerState> states = describeProducers(TOPIC_ACKS1);
Report.bullet("producer states tracked : %d", states.size());
states.forEach(s -> Report.bullet(" producerId=%d epoch=%d lastSequence=%d",
s.producerId(), s.producerEpoch(), s.lastSequence()));
assertThat(states)
.as("a non-idempotent producer leaves no de-duplication state on the broker")
.isEmpty();
Report.takeaway("Three records were written, and the broker is tracking NOTHING for them.");
Report.takeaway("Kafka resolved acks=1 against enable.idempotence=true by turning");
Report.takeaway("idempotence off, and said nothing at all. Every record this producer");
Report.takeaway("sends can now be duplicated by an ordinary retry.");
}
// ---------------------------------------------------------------------------------------
// 3. The loud version of the same mistake
// ---------------------------------------------------------------------------------------
@Test
@Order(3)
void acksOnePlusExplicitIdempotenceIsRejected() {
Report.title("3. acks=1 + enable.idempotence=true is rejected at construction");
Report.section("Same acks=1, but this time idempotence is requested explicitly");
assertThatThrownBy(() -> producer(Map.of(
ProducerConfig.ACKS_CONFIG, "1",
ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true")))
.isInstanceOf(ConfigException.class)
.satisfies(e -> Report.bullet("ConfigException: %s", e.getMessage()));
Report.takeaway("Ask for it explicitly and Kafka refuses. Leave it implicit and Kafka");
Report.takeaway("downgrades you. So setting enable.idempotence=true explicitly is worth");
Report.takeaway("doing purely as an assertion: it turns a silent downgrade into a");
Report.takeaway("startup failure, which is the failure you want.");
}
// ---------------------------------------------------------------------------------------
// 4. The other constraints
// ---------------------------------------------------------------------------------------
@Test
@Order(4)
void theOtherConstraintsIdempotenceImposes() {
Report.title("4. The other constraints, and which ones fail loudly");
Report.section("max.in.flight.requests.per.connection = 6");
assertThatThrownBy(() -> producer(Map.of(
ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true",
ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "6")))
.isInstanceOf(ConfigException.class)
.satisfies(e -> Report.bullet("rejected: %s", e.getMessage()));
Report.section("retries = 0");
assertThatThrownBy(() -> producer(Map.of(
ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true",
ProducerConfig.RETRIES_CONFIG, "0")))
.isInstanceOf(ConfigException.class)
.satisfies(e -> Report.bullet("rejected: %s", e.getMessage()));
Report.section("max.in.flight = 5 with idempotence -- allowed");
try (Producer<String, String> p = producer(Map.of(
ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true",
ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"))) {
Report.bullet("constructed successfully");
}
Report.takeaway("Idempotence requires acks=all, retries>0 and max.in.flight<=5.");
Report.takeaway("Five in flight is safe because the broker can reorder within that window");
Report.takeaway("using sequence numbers. WITHOUT idempotence, max.in.flight>1 silently");
Report.takeaway("breaks ORDERING on retry: a retried batch can land after a later one.");
}
// ---------------------------------------------------------------------------------------
// helpers
// ---------------------------------------------------------------------------------------
private Producer<String, String> producer(Map<String, Object> overrides) {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.putAll(overrides);
return new KafkaProducer<>(props);
}
/**
* Asks the broker which producers it is currently tracking idempotence state for.
*
* <p>This is the observation that makes the test meaningful rather than tautological.
*/
private List<ProducerState> describeProducers(String topic) throws Exception {
try (Admin admin = Admin.create(Map.of("bootstrap.servers", broker.getBrokersAsString()))) {
TopicPartition tp = new TopicPartition(topic, 0);
return admin.describeProducers(List.of(tp)).partitionResult(tp).get().activeProducers();
}
}
}

323
src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java Normal file
View File

@@ -0,0 +1,323 @@
package com.ankurm.kafka.eos._02_transactions;
import com.ankurm.kafka.eos.support.Report;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* <h2>Transaction markers: why your offsets have holes and your lag metric lies</h2>
*
* <p>This is the most operationally surprising consequence of turning on transactions, and it is
* barely mentioned anywhere.
*
* <p>When a transaction commits or aborts, Kafka writes a <b>control record</b> -- a transaction
* marker -- into every partition the transaction touched. Control records are real entries in the
* log and they <b>consume an offset</b>, but they are never delivered to your consumer. Two things
* follow immediately:
*
* <ol>
* <li><b>Offsets are not contiguous.</b> Send five records in a transaction and the next record
* you send will be at offset 6, not 5. Any code that assumes {@code offset + 1} is the next
* record, or that counts records by subtracting offsets, is now wrong.</li>
* <li><b>Consumer lag is overstated.</b> The usual lag formula is
* {@code endOffset - committedOffset}. With transactions, {@code endOffset} counts markers
* that will never be delivered, so a perfectly caught-up consumer reports non-zero lag --
* permanently, and proportionally to how many transactions you commit.</li>
* </ol>
*
* <p>A third consequence appears when a transaction <b>aborts</b>: the records were already written
* to the log and they keep their offsets forever. A {@code read_committed} consumer filters them
* out; a {@code read_uncommitted} consumer sees them. The disk space and the offsets are consumed
* either way.
*
* <p>There is also a <b>latency</b> consequence that catches people out. A {@code read_committed}
* consumer can only read up to the <b>Last Stable Offset</b> (LSO) -- the offset before the first
* still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every
* {@code read_committed} consumer on that partition stalls for 30 seconds, even for records
* committed by other producers afterwards. Transaction duration is therefore a consumer-latency
* knob, not just a producer concern.
*/
@SpringJUnitConfig
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@EmbeddedKafka(partitions = 1, topics = {TransactionMarkerOffsetTest.TOPIC})
class TransactionMarkerOffsetTest {
static final String TOPIC = "markers-demo";
private static final TopicPartition PARTITION = new TopicPartition(TOPIC, 0);
@Autowired
EmbeddedKafkaBroker broker;
@Test
@Order(1)
void committedTransactionConsumesAnExtraOffsetForItsMarker() {
Report.title("1. A committed transaction burns one extra offset per partition");
Report.section("Before: empty topic");
Report.bullet("endOffset = %d", endOffset());
Report.section("Sending 5 records inside ONE transaction, then committing");
try (Producer<String, String> producer = transactionalProducer("tx-markers-1")) {
producer.initTransactions();
producer.beginTransaction();
for (int i = 0; i < 5; i++) {
var meta = producer.send(new ProducerRecord<>(TOPIC, "k", "committed-" + i)).get();
Report.bullet("sent record %d -> offset %d", i, meta.offset());
}
producer.commitTransaction();
} catch (Exception e) {
throw new IllegalStateException(e);
}
long end = awaitEndOffset(6);
Report.section("After commit");
Report.bullet("records sent : 5");
Report.bullet("endOffset : %d <-- 5 records occupy 0..4, the commit marker takes 5", end);
assertThat(end).isEqualTo(6);
Report.section("What a read_committed consumer actually receives");
List<ConsumerRecord<String, String>> received = drain("read_committed", "g-committed-1");
received.forEach(r -> Report.bullet("offset=%d value=%s", r.offset(), r.value()));
Report.bullet("records delivered : %d", received.size());
assertThat(received).hasSize(5);
assertThat(received.getLast().offset()).isEqualTo(4);
Report.takeaway("5 records were delivered, the last at offset 4, but the end offset is 6.");
Report.takeaway("The gap at offset 5 is the commit marker. It exists, it is durable, and");
Report.takeaway("no consumer will ever see it.");
}
@Test
@Order(2)
void abortedRecordsKeepTheirOffsetsForever() {
Report.title("2. An aborted transaction still consumes offsets and disk");
long before = endOffset();
Report.bullet("endOffset before : %d", before);
Report.section("Sending 3 records inside a transaction, then ABORTING");
try (Producer<String, String> producer = transactionalProducer("tx-markers-2")) {
producer.initTransactions();
producer.beginTransaction();
for (int i = 0; i < 3; i++) {
var meta = producer.send(new ProducerRecord<>(TOPIC, "k", "aborted-" + i)).get();
Report.bullet("sent record %d -> offset %d", i, meta.offset());
}
producer.abortTransaction();
Report.bullet("abortTransaction() called");
} catch (Exception e) {
throw new IllegalStateException(e);
}
long after = awaitEndOffset(before + 4);
Report.section("After abort");
Report.bullet("endOffset after : %d (+%d for 3 aborted records and 1 abort marker)",
after, after - before);
assertThat(after).isEqualTo(before + 4);
Report.section("read_committed consumer");
List<ConsumerRecord<String, String>> committed = drain("read_committed", "g-committed-2");
Report.bullet("delivered : %d records, values=%s",
committed.size(), committed.stream().map(ConsumerRecord::value).toList());
Report.section("read_uncommitted consumer -- the SAME partition");
List<ConsumerRecord<String, String>> uncommitted = drain("read_uncommitted", "g-uncommitted-2");
Report.bullet("delivered : %d records, values=%s",
uncommitted.size(), uncommitted.stream().map(ConsumerRecord::value).toList());
assertThat(committed).hasSize(5); // only the 5 from test 1
assertThat(uncommitted).hasSize(8); // those 5 plus the 3 aborted ones
Report.takeaway("The aborted records are physically in the log with permanent offsets.");
Report.takeaway("isolation.level is a CONSUMER-side filter, not a broker-side delete.");
Report.takeaway("A default consumer (read_uncommitted) reads data you deliberately aborted.");
}
/**
* A test that exists because it refuted the claim it was written to demonstrate.
*
* <p>The intended demonstration was "transaction markers inflate consumer lag". That is a
* widely repeated claim and it is <b>wrong</b>, at least for the standard
* {@code endOffset - committedOffset} formula: when a {@code read_committed} consumer drains a
* partition, its <em>position</em> advances past the markers too, so the committed offset
* reaches the end offset and lag is exactly zero.
*
* <p>What markers really break is <b>offset arithmetic used to count records</b>, which is a
* different and much easier mistake to make: retention estimates, "how many events happened
* between these two offsets", partition-size dashboards, and any test that asserts on offsets.
*/
@Test
@Order(3)
void lagIsFineButOffsetArithmeticIsNot() {
Report.title("3. Lag is NOT inflated -- but counting records by offset arithmetic is broken");
Report.section("A read_committed consumer drains the partition, then commits");
Map<String, Object> props = consumerProps("read_committed", "g-lag");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
long end;
long beginning;
long committedOffset;
long lastDeliveredOffset;
int delivered;
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.assign(List.of(PARTITION));
consumer.seekToBeginning(List.of(PARTITION));
List<ConsumerRecord<String, String>> all = new ArrayList<>();
for (int i = 0; i < 8; i++) {
consumer.poll(Duration.ofMillis(400)).forEach(all::add);
}
delivered = all.size();
lastDeliveredOffset = all.isEmpty() ? -1 : all.getLast().offset();
consumer.commitSync();
beginning = consumer.beginningOffsets(List.of(PARTITION)).get(PARTITION);
end = consumer.endOffsets(List.of(PARTITION)).get(PARTITION);
committedOffset = consumer.committed(java.util.Set.of(PARTITION)).get(PARTITION).offset();
}
Report.section("The lag calculation everyone worries about");
Report.bullet("partition end offset : %d", end);
Report.bullet("consumer committed offset : %d", committedOffset);
Report.bullet("lag (end - committed) : %d", end - committedOffset);
assertThat(end - committedOffset)
.as("a drained read_committed consumer really is at zero lag")
.isZero();
Report.takeaway("Lag is ZERO. The consumer's position advances over markers and aborted");
Report.takeaway("records just as it advances over delivered ones, so the standard lag");
Report.takeaway("formula stays correct. The common claim that transactions inflate lag");
Report.takeaway("does not survive contact with a broker.");
Report.section("The calculation that IS broken");
Report.bullet("beginningOffset : %d", beginning);
Report.bullet("endOffset : %d", end);
Report.bullet("end - beginning : %d <-- looks like the record count", end - beginning);
Report.bullet("records actually retrievable: %d", delivered);
Report.bullet("offset of last record : %d", lastDeliveredOffset);
Report.bullet("overstatement : %.0f%%",
100.0 * (end - beginning - delivered) / delivered);
assertThat(end - beginning)
.as("offset span is not the record count once transactions are involved")
.isNotEqualTo(delivered);
Report.takeaway("Offset span says %d records. Only %d can ever be read -- a %.0f%% error,",
end - beginning, delivered, 100.0 * (end - beginning - delivered) / delivered);
Report.takeaway("from 2 markers and 3 aborted records. Anything that counts events by");
Report.takeaway("subtracting offsets -- retention math, throughput dashboards, tests that");
Report.takeaway("assert on offsets -- is wrong by however much your abort rate happens to be.");
}
// ---------------------------------------------------------------------------------------
// helpers
// ---------------------------------------------------------------------------------------
private Producer<String, String> transactionalProducer(String transactionalId) {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// Setting a transactional.id is what turns an idempotent producer into a transactional one.
// It also forces enable.idempotence=true -- transactions are built on top of idempotence.
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId);
return new KafkaProducer<>(props);
}
private Map<String, Object> consumerProps(String isolationLevel, String groupId) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// The default is read_uncommitted. Consuming the output of a transactional producer
// without changing this gives you aborted records -- see test 2.
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, isolationLevel);
return props;
}
private List<ConsumerRecord<String, String>> drain(String isolationLevel, String groupId) {
List<ConsumerRecord<String, String>> out = new ArrayList<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps(isolationLevel, groupId))) {
consumer.assign(List.of(PARTITION));
consumer.seekToBeginning(List.of(PARTITION));
for (int i = 0; i < 8; i++) {
ConsumerRecords<String, String> polled = consumer.poll(Duration.ofMillis(400));
polled.forEach(out::add);
}
}
return out;
}
/**
* Waits (briefly) for the partition's end offset to reach an expected value.
*
* <p>Needed because <b>{@code commitTransaction()} returning does not mean the transaction
* marker is already visible in the partition's high watermark.</b> The producer's commit is
* acknowledged by the transaction coordinator; the coordinator then writes markers into each
* participating partition, and the leader's high watermark advances after that. Reading
* {@code endOffsets()} immediately after the commit therefore races, and sometimes returns the
* offset without the marker.
*
* <p>This test originally asserted on the immediate read and passed consistently when run
* alone -- then failed in the full suite, where the machine was busier and the race opened up.
* That is the classic shape of a flaky integration test, and the fix is to wait for the
* condition rather than to assume it.
*/
private long awaitEndOffset(long expected) {
long deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(10);
long last = endOffset();
while (last != expected && System.nanoTime() < deadline) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
last = endOffset();
}
return last;
}
private long endOffset() {
try (KafkaConsumer<String, String> consumer =
new KafkaConsumer<>(consumerProps("read_uncommitted", "g-endoffset-" + System.nanoTime()))) {
// Assign before asking: endOffsets() on an unassigned consumer works, but logs a
// "Not updating high watermark ... no longer assigned" warning that clutters the
// transcript this test exists to produce.
consumer.assign(List.of(PARTITION));
return consumer.endOffsets(List.of(PARTITION)).get(PARTITION);
}
}
}

194
src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java Normal file
View File

@@ -0,0 +1,194 @@
package com.ankurm.kafka.eos._03_read_process_write;
import com.ankurm.kafka.eos.Application;
import com.ankurm.kafka.eos.orders.OrderProcessor;
import com.ankurm.kafka.eos.support.Report;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
/**
* <h2>Read-process-write: the only thing Kafka EOS actually guarantees</h2>
*
* <p>These tests exercise {@link OrderProcessor}, which is a completely ordinary
* {@code @KafkaListener} that sends one record. The exactly-once behaviour comes from
* configuration, not from the method body.
*
* <p>The point of the second test is the one that matters in production: when processing fails
* <em>after</em> the output record has been sent, the transaction aborts. The output record is
* physically written to the log -- it has an offset, it takes disk space -- but no
* {@code read_committed} consumer will ever see it, and the input offset is not advanced, so the
* record is redelivered. On the retry the work happens again and this time commits. The downstream
* consumer sees the result exactly once, despite the processing having run twice.
*
* <p>That distinction -- <b>processed twice, committed once</b> -- is what "exactly-once" means in
* Kafka. It has never meant that your code runs once.
*/
@SpringBootTest(classes = Application.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@EmbeddedKafka(
partitions = 1,
topics = {OrderProcessor.INPUT_TOPIC, OrderProcessor.OUTPUT_TOPIC},
// The transaction state topic defaults to 3 replicas, which a single-broker embedded
// cluster cannot provide. Without these two overrides every transactional test fails with
// a very unhelpful COORDINATOR_NOT_AVAILABLE.
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class ReadProcessWriteTest {
@Autowired
KafkaTemplate<String, String> kafkaTemplate;
@Autowired
OrderProcessor processor;
@Autowired
EmbeddedKafkaBroker broker;
@BeforeEach
void reset() {
processor.reset();
}
@Test
@Order(1)
void happyPathProducesExactlyOneOutputRecord() {
Report.title("1. Read-process-write, happy path");
Report.section("Publishing 3 orders to " + OrderProcessor.INPUT_TOPIC);
kafkaTemplate.executeInTransaction(t -> {
for (String order : List.of("alpha", "beta", "gamma")) {
t.send(OrderProcessor.INPUT_TOPIC, order);
Report.bullet("sent: %s", order);
}
return null;
});
await().atMost(Duration.ofSeconds(30))
.until(() -> processor.invocations().size() >= 3);
Report.section("Listener invocations");
processor.invocations().forEach(v -> Report.bullet("processed: %s", v));
Report.section("What a read_committed consumer sees on " + OrderProcessor.OUTPUT_TOPIC);
List<ConsumerRecord<String, String>> output = drain(OrderProcessor.OUTPUT_TOPIC, "g-happy");
output.forEach(r -> Report.bullet("offset=%d value=%s", r.offset(), r.value()));
assertThat(output).extracting(ConsumerRecord::value)
.containsExactlyInAnyOrder("ALPHA", "BETA", "GAMMA");
Report.takeaway("3 in, 3 out. The listener body contains no transaction code at all --");
Report.takeaway("the guarantee comes entirely from spring.kafka.producer.transaction-id-prefix.");
}
@Test
@Order(2)
void failureAfterSendRollsBackAndRedelivers() {
Report.title("2. Failure AFTER the send: processed twice, delivered once");
Report.section("Poisoning 'delta' so its first processing attempt throws");
processor.poison("delta");
kafkaTemplate.executeInTransaction(t -> t.send(OrderProcessor.INPUT_TOPIC, "delta"));
Report.bullet("sent: delta");
Report.section("Waiting for the listener to be invoked more than once");
await().atMost(Duration.ofSeconds(45))
.until(() -> processor.invocations().stream().filter("delta"::equals).count() >= 2);
long attempts = processor.invocations().stream().filter("delta"::equals).count();
Report.bullet("times the listener ran for 'delta' : %d", attempts);
Report.section("What a read_committed consumer sees");
List<ConsumerRecord<String, String>> committed =
drain(OrderProcessor.OUTPUT_TOPIC, "g-rollback-committed");
List<String> deltas = committed.stream()
.map(ConsumerRecord::value).filter("DELTA"::equals).toList();
Report.bullet("DELTA records visible : %d", deltas.size());
Report.section("What a read_uncommitted consumer sees on the SAME topic");
List<ConsumerRecord<String, String>> raw =
drainUncommitted(OrderProcessor.OUTPUT_TOPIC, "g-rollback-raw");
long rawDeltas = raw.stream().map(ConsumerRecord::value).filter("DELTA"::equals).count();
Report.bullet("DELTA records physically in the log : %d", rawDeltas);
assertThat(attempts).as("the failure caused a redelivery").isGreaterThanOrEqualTo(2);
assertThat(deltas).as("exactly one DELTA is committed").hasSize(1);
Report.takeaway("The listener ran %d times and a read_committed consumer sees exactly 1.", attempts);
Report.takeaway("");
Report.takeaway("This is the whole idea: exactly-once is a property of what is COMMITTED,");
Report.takeaway("not of how many times your code ran. Any side effect your listener performs");
Report.takeaway("outside Kafka -- a DB write, an HTTP call, an email -- happened %d times.", attempts);
Report.section("A detail worth noticing in the number above");
Report.bullet("read_uncommitted sees %d DELTA record(s), not %d", rawDeltas, attempts);
Report.takeaway("The rolled-back send never reached the broker at all. KafkaTemplate.send()");
Report.takeaway("is asynchronous: the record sat in the producer's accumulator, and");
Report.takeaway("abortTransaction() discarded the un-flushed batch instead of writing it.");
Report.takeaway("");
Report.takeaway("So aborted records reach the log only if they were already flushed -- which");
Report.takeaway("happens under load, on large batches, or when you block on the send future.");
Report.takeaway("TransactionMarkerOffsetTest forces exactly that case by calling get() on");
Report.takeaway("each send, and there the aborted records DO occupy permanent offsets.");
Report.takeaway("Do not rely on either behaviour: rely on read_committed.");
}
// ---------------------------------------------------------------------------------------
// helpers
// ---------------------------------------------------------------------------------------
private List<ConsumerRecord<String, String>> drain(String topic, String groupId) {
return drain(topic, groupId, "read_committed");
}
private List<ConsumerRecord<String, String>> drainUncommitted(String topic, String groupId) {
return drain(topic, groupId, "read_uncommitted");
}
private List<ConsumerRecord<String, String>> drain(String topic, String groupId, String isolation) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, isolation);
List<ConsumerRecord<String, String>> out = new ArrayList<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
TopicPartition tp = new TopicPartition(topic, 0);
consumer.assign(Set.of(tp));
consumer.seekToBeginning(Set.of(tp));
for (int i = 0; i < 8; i++) {
consumer.poll(Duration.ofMillis(400)).forEach(out::add);
}
}
return out;
}
}

193
src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java Normal file
View File

@@ -0,0 +1,193 @@
package com.ankurm.kafka.eos._04_fencing;
import com.ankurm.kafka.eos.support.Report;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.errors.InvalidProducerEpochException;
import org.apache.kafka.common.errors.ProducerFencedException;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import java.util.HashMap;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
* <h2>Zombie fencing: what {@code transactional.id} is actually for</h2>
*
* <p>Consider a processor that reads, transforms and writes. It stalls -- a long GC pause, a
* network partition, a container the orchestrator believes is dead. Kubernetes starts a
* replacement. The consumer group rebalances and the new instance takes over the partition.
*
* <p>Then the old instance wakes up. It has a partially completed transaction and it has no idea it
* was replaced. If it were allowed to commit, the same input would be processed twice and both
* results would be committed. That old instance is the <b>zombie</b>.
*
* <p>The defence is the {@code transactional.id}. It is a <em>stable, durable identity</em> that the
* broker's transaction coordinator remembers along with an <b>epoch</b>. Whenever a producer calls
* {@code initTransactions()} for an existing {@code transactional.id}, the coordinator bumps the
* epoch and refuses to serve anything holding the old one. The zombie's commit is rejected.
*
* <p>This is precisely why the {@code transactional.id} must be <b>stable across restarts</b> and
* <b>unique per logical processor</b>. Two common mistakes, with opposite effects:
*
* <ul>
* <li><b>Random per start-up</b> (for example a UUID) -- fencing never happens, because the new
* instance has a different identity from the zombie. You have paid for transactions and
* bought no zombie protection at all.</li>
* <li><b>Shared across instances</b> -- every instance fences every other one, continuously, and
* your consumer group thrashes with {@code ProducerFencedException}s.</li>
* </ul>
*
* <p>Spring's {@code spring.kafka.producer.transaction-id-prefix} plus a per-instance suffix is
* the answer; see {@code docs/04-fencing.md} and {@code TransactionIdSuffixStrategy}
* (Spring Kafka 3.2+).
*
* <h3>A Kafka 4.x note</h3>
* <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-890">KIP-890</a> strengthened the
* transaction protocol. Under {@code transaction.version=2} -- which this embedded broker reports
* at start-up, and which is the default on Kafka 4.x -- the producer epoch is bumped on
* <em>every</em> transaction, not just at {@code initTransactions()}. That closes a class of
* hanging-transaction bugs, and it also means the exception you see when fenced may be
* {@link InvalidProducerEpochException} rather than the {@link ProducerFencedException} that older
* articles describe. This test reports whichever it actually gets rather than asserting one.
*/
@SpringJUnitConfig
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@EmbeddedKafka(
partitions = 1,
topics = {ZombieFencingTest.TOPIC},
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class ZombieFencingTest {
static final String TOPIC = "fencing-demo";
@Autowired
EmbeddedKafkaBroker broker;
@Test
@Order(1)
void secondProducerWithSameTransactionalIdFencesTheFirst() throws Exception {
Report.title("1. A second producer with the same transactional.id fences the first");
final String sharedId = "shared-tx-id";
Producer<String, String> zombie = transactionalProducer(sharedId);
Producer<String, String> replacement = null;
try {
Report.section("Producer A ('the zombie') starts a transaction and sends");
zombie.initTransactions();
zombie.beginTransaction();
// .get() forces the record to the broker, so the transaction is genuinely in progress
// rather than sitting unsent in the accumulator.
zombie.send(new ProducerRecord<>(TOPIC, "k", "from-zombie")).get();
Report.bullet("transactional.id = '%s'", sharedId);
Report.bullet("record sent, transaction still OPEN");
Report.section("Producer B starts with the SAME transactional.id");
replacement = transactionalProducer(sharedId);
replacement.initTransactions();
Report.bullet("initTransactions() returned -- the coordinator has bumped the epoch");
Report.bullet("any in-flight transaction under the old epoch is now aborted");
Report.section("Producer A tries to commit");
final Producer<String, String> fenced = zombie;
assertThatThrownBy(fenced::commitTransaction)
.isInstanceOfAny(ProducerFencedException.class, InvalidProducerEpochException.class)
.satisfies(e -> {
Report.bullet("REJECTED with %s", e.getClass().getSimpleName());
Report.bullet("message: %s", e.getMessage());
});
Report.section("Producer B commits normally");
replacement.beginTransaction();
replacement.send(new ProducerRecord<>(TOPIC, "k", "from-replacement")).get();
replacement.commitTransaction();
Report.bullet("committed successfully");
Report.takeaway("The zombie could not commit. Its work was discarded by the broker,");
Report.takeaway("not by any code you wrote. That is the entire value of transactional.id:");
Report.takeaway("a durable identity the coordinator can fence on.");
Report.takeaway("");
Report.takeaway("Note the exception type -- on Kafka 4.x with KIP-890 (transaction.version=2)");
Report.takeaway("the epoch is bumped every transaction, so you may see");
Report.takeaway("InvalidProducerEpochException where older articles promise");
Report.takeaway("ProducerFencedException. Catch both, or catch neither and let the");
Report.takeaway("container restart the producer, which is what Spring does.");
} finally {
// A fenced producer cannot be used again; close it without attempting a flush.
zombie.close(java.time.Duration.ZERO);
if (replacement != null) {
replacement.close();
}
}
}
@Test
@Order(2)
void randomTransactionalIdSilentlyDefeatsFencing() throws Exception {
Report.title("2. A random transactional.id per start-up defeats fencing entirely");
Report.section("Producer A with a UUID-based transactional.id");
String idA = "processor-" + java.util.UUID.randomUUID();
Report.bullet("transactional.id = '%s'", idA);
try (Producer<String, String> a = transactionalProducer(idA)) {
a.initTransactions();
a.beginTransaction();
a.send(new ProducerRecord<>(TOPIC, "k", "A-uncommitted")).get();
Report.section("Producer B restarts and generates a NEW UUID");
String idB = "processor-" + java.util.UUID.randomUUID();
Report.bullet("transactional.id = '%s'", idB);
try (Producer<String, String> b = transactionalProducer(idB)) {
b.initTransactions();
b.beginTransaction();
b.send(new ProducerRecord<>(TOPIC, "k", "B-committed")).get();
b.commitTransaction();
Report.bullet("B committed");
}
Report.section("Now the 'zombie' A commits -- does anything stop it?");
a.commitTransaction();
Report.bullet("A committed successfully. Nothing was fenced.");
assertThat(true).isTrue(); // the point is that no exception was thrown
Report.takeaway("Both producers committed. The coordinator saw two unrelated identities");
Report.takeaway("and had no reason to fence either one.");
Report.takeaway("");
Report.takeaway("This is the most damaging Kafka transactions misconfiguration, because");
Report.takeaway("everything appears to work: transactions commit, tests pass, and you have");
Report.takeaway("exactly the duplicate-processing problem you turned transactions on to");
Report.takeaway("prevent. The transactional.id must be STABLE across restarts and tied to");
Report.takeaway("the partition set an instance owns -- not to the process lifetime.");
}
}
private Producer<String, String> transactionalProducer(String transactionalId) {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId);
props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 10_000);
return new KafkaProducer<>(props);
}
}

279
src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java Normal file
View File

@@ -0,0 +1,279 @@
package com.ankurm.kafka.eos._05_database;
import com.ankurm.kafka.eos.Application;
import com.ankurm.kafka.eos.db.ProcessedOrder;
import com.ankurm.kafka.eos.db.ProcessedOrderRepository;
import com.ankurm.kafka.eos.support.Report;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.test.context.EmbeddedKafka;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
/**
* <h2>The boundary: Kafka's exactly-once does not extend to your database</h2>
*
* <p>This is the single most consequential misunderstanding about Kafka transactions, and it is
* worth stating as plainly as possible:
*
* <blockquote>
* A Kafka transaction is atomic across <b>Kafka partitions and Kafka consumer offsets</b>.
* That is all. It cannot include a database write, an HTTP call, a file, or an email.
* </blockquote>
*
* <p>There is no two-phase commit between Kafka and your database, and there is not going to be.
* Spring used to ship {@code ChainedKafkaTransactionManager} for this and it is <b>deprecated</b>,
* because chaining transaction managers is best-effort ordering, not distributed coordination:
* it commits one, then the other, and a crash in between leaves them disagreeing.
*
* <p>So a listener that writes to a database is <em>at-least-once</em> with respect to that
* database, no matter what your Kafka configuration says. The redelivery that makes Kafka's
* guarantee work is exactly the thing that duplicates your row.
*
* <h3>What these tests show</h3>
* <ol>
* <li><b>The problem.</b> A listener that inserts unconditionally produces one row per
* <em>attempt</em>, not one per message.</li>
* <li><b>The fix.</b> The same listener made idempotent with a natural dedupe key produces one
* row per message however many times it runs.</li>
* </ol>
*
* <p>The dedupe key here is {@code topic-partition-offset}, which Kafka guarantees is unique and
* stable for a record, is available on every consumer record, and requires no coordination.
* The alternative is an event ID carried in the payload, which is better when the same logical
* event can arrive on more than one topic. Either way the principle is the same, and it is the only
* approach that survives contact with reality: <b>make the side effect idempotent, because the
* delivery is not.</b>
*
* <p>See {@code docs/05-database-boundary.md} for the outbox pattern, which inverts the problem by
* making the database the source of truth and publishing to Kafka afterwards.
*/
@SpringBootTest(classes = {Application.class, DatabaseBoundaryTest.Listeners.class})
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@EmbeddedKafka(
partitions = 1,
topics = {DatabaseBoundaryTest.NAIVE_TOPIC, DatabaseBoundaryTest.IDEMPOTENT_TOPIC},
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class DatabaseBoundaryTest {
static final String NAIVE_TOPIC = "db.naive";
static final String IDEMPOTENT_TOPIC = "db.idempotent";
@Autowired
KafkaTemplate<String, String> kafkaTemplate;
@Autowired
ProcessedOrderRepository repository;
@Autowired
Listeners.NaiveDbListener naive;
@Autowired
Listeners.IdempotentDbListener idempotent;
@BeforeEach
void clean() {
repository.deleteAll();
naive.reset();
idempotent.reset();
}
@Test
@Order(1)
void aNaiveDatabaseWriteIsDuplicatedByRedelivery() {
Report.title("1. The problem: one row per ATTEMPT, not per message");
Report.section("Sending one message that will fail on its first processing attempt");
naive.failNextAttemptFor("order-1");
kafkaTemplate.executeInTransaction(t -> t.send(NAIVE_TOPIC, "order-1"));
await().atMost(Duration.ofSeconds(45))
.until(() -> naive.attempts().stream().filter("order-1"::equals).count() >= 2);
// Let any further redelivery settle before counting.
sleep(2_000);
long attempts = naive.attempts().stream().filter("order-1"::equals).count();
List<ProcessedOrder> rows = repository.findByPayload("order-1");
Report.bullet("listener attempts for 'order-1' : %d", attempts);
Report.bullet("rows in the database : %d", rows.size());
rows.forEach(r -> Report.bullet(" id=%d dedupeKey=%s", r.getId(), r.getDedupeKey()));
assertThat(attempts).isGreaterThanOrEqualTo(2);
assertThat(rows.size()).as("the naive listener duplicated the row").isGreaterThan(1);
Report.takeaway("The Kafka transaction rolled back correctly. The database row did not,");
Report.takeaway("because it was never in the Kafka transaction and never could have been.");
Report.takeaway("");
Report.takeaway("Note that nothing here is misconfigured. transaction-id-prefix is set,");
Report.takeaway("isolation.level is read_committed, the Kafka side is exactly-once. The");
Report.takeaway("database still has %d rows for 1 message.", rows.size());
}
@Test
@Order(2)
void anIdempotentWriteSurvivesRedelivery() {
Report.title("2. The fix: make the side effect idempotent, not the delivery");
Report.section("Same failure, but the listener dedupes on topic-partition-offset");
idempotent.failNextAttemptFor("order-2");
kafkaTemplate.executeInTransaction(t -> t.send(IDEMPOTENT_TOPIC, "order-2"));
await().atMost(Duration.ofSeconds(45))
.until(() -> idempotent.attempts().stream().filter("order-2"::equals).count() >= 2);
sleep(2_000);
long attempts = idempotent.attempts().stream().filter("order-2"::equals).count();
List<ProcessedOrder> rows = repository.findByPayload("order-2");
Report.bullet("listener attempts for 'order-2' : %d", attempts);
Report.bullet("rows in the database : %d", rows.size());
rows.forEach(r -> Report.bullet(" id=%d dedupeKey=%s", r.getId(), r.getDedupeKey()));
assertThat(attempts).isGreaterThanOrEqualTo(2);
assertThat(rows).as("the idempotent listener wrote exactly one row").hasSize(1);
Report.takeaway("The listener ran %d times and wrote 1 row.", attempts);
Report.takeaway("");
Report.takeaway("The dedupe key is 'topic-partition-offset', which Kafka guarantees is");
Report.takeaway("unique and stable. A UNIQUE constraint on that column turns the second");
Report.takeaway("attempt into a no-op -- and, importantly, would still do so if the");
Report.takeaway("duplicate came from a different process, a rebalance, or a replay.");
Report.takeaway("");
Report.takeaway("This is the whole design rule: exactly-once processing is achieved by");
Report.takeaway("idempotent side effects plus at-least-once delivery. Kafka transactions");
Report.takeaway("give you the second half. You have to build the first half yourself.");
}
private static void sleep(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
// -------------------------------------------------------------------------------------------
@TestConfiguration
static class Listeners {
@Bean
NaiveDbListener naiveDbListener(ProcessedOrderRepository repository) {
return new NaiveDbListener(repository);
}
@Bean
IdempotentDbListener idempotentDbListener(ProcessedOrderRepository repository) {
return new IdempotentDbListener(repository);
}
/** Base class holding the failure-injection and attempt-recording plumbing. */
abstract static class Base {
protected final ProcessedOrderRepository repository;
private final List<String> attempts = new CopyOnWriteArrayList<>();
private final List<String> failOnce = new CopyOnWriteArrayList<>();
protected final AtomicInteger counter = new AtomicInteger();
protected Base(ProcessedOrderRepository repository) {
this.repository = repository;
}
protected void record(String value) {
attempts.add(value);
}
protected void maybeFail(String value) {
if (failOnce.remove(value)) {
throw new IllegalStateException("simulated failure after the DB write for " + value);
}
}
public void failNextAttemptFor(String value) {
failOnce.add(value);
}
public List<String> attempts() {
return List.copyOf(attempts);
}
public void reset() {
attempts.clear();
failOnce.clear();
}
}
/**
* Writes unconditionally. Correct-looking, and wrong the moment anything is redelivered.
*/
static class NaiveDbListener extends Base {
private static final Logger log = LoggerFactory.getLogger(NaiveDbListener.class);
NaiveDbListener(ProcessedOrderRepository repository) {
super(repository);
}
@KafkaListener(topics = NAIVE_TOPIC, groupId = "db-naive")
public void onMessage(String payload) {
record(payload);
// A surrogate key means every attempt looks like new work to the database.
repository.save(new ProcessedOrder("naive-" + counter.incrementAndGet(), payload));
log.info("naive listener wrote a row for '{}'", payload);
maybeFail(payload);
}
}
/**
* Writes only if this exact record has not been written before, keyed on the one identifier
* Kafka guarantees to be unique and stable.
*/
static class IdempotentDbListener extends Base {
private static final Logger log = LoggerFactory.getLogger(IdempotentDbListener.class);
IdempotentDbListener(ProcessedOrderRepository repository) {
super(repository);
}
@KafkaListener(topics = IDEMPOTENT_TOPIC, groupId = "db-idempotent")
public void onMessage(String payload,
@org.springframework.messaging.handler.annotation.Header(
org.springframework.kafka.support.KafkaHeaders.RECEIVED_TOPIC) String topic,
@org.springframework.messaging.handler.annotation.Header(
org.springframework.kafka.support.KafkaHeaders.RECEIVED_PARTITION) int partition,
@org.springframework.messaging.handler.annotation.Header(
org.springframework.kafka.support.KafkaHeaders.OFFSET) long offset) {
record(payload);
String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset);
if (repository.existsByDedupeKey(dedupeKey)) {
log.info("idempotent listener SKIPPED '{}' -- {} already processed", payload, dedupeKey);
} else {
repository.save(new ProcessedOrder(dedupeKey, payload));
log.info("idempotent listener wrote a row for '{}' as {}", payload, dedupeKey);
}
maybeFail(payload);
}
}
}
}

265
src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java Normal file
View File

@@ -0,0 +1,265 @@
package com.ankurm.kafka.eos._06_performance;
import com.ankurm.kafka.eos.support.Report;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* <h2>What do transactions actually cost?</h2>
*
* <p>"Transactions are slow" is repeated constantly and almost never quantified. The useful version
* of the statement is much more specific: <b>the cost is per transaction, not per record.</b>
*
* <p>Committing a transaction requires the producer to talk to the transaction coordinator and the
* coordinator to write a marker into every partition the transaction touched, then wait for those
* writes to be acknowledged. That is a fixed cost of roughly one network round trip plus a few
* durable writes. It does not care whether the transaction contained one record or ten thousand.
*
* <p>So the question is never "should I use transactions" in isolation -- it is "how many records
* should be in each one". This benchmark sweeps exactly that, plus a non-transactional baseline.
*
* <h3>Reading the numbers honestly</h3>
* These run against a single-node in-JVM broker on a laptop, so the absolute throughput is not
* meaningful and is not quoted as if it were. What is meaningful, and what does transfer to real
* clusters, is the <em>shape</em>: a steep penalty at one record per transaction, most of it
* recovered by a hundred, and a baseline that batching approaches but never quite reaches.
*
* <p>On a real cluster the per-commit cost is larger (real network, real replication), so the
* penalty at small batch sizes is <em>worse</em> than shown here, and batching matters more.
*/
@SpringJUnitConfig
@EmbeddedKafka(
partitions = 1,
topics = {TransactionCostTest.TOPIC},
brokerProperties = {
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1",
"offsets.topic.replication.factor=1"
})
class TransactionCostTest {
static final String TOPIC = "perf-demo";
private static final int RECORDS = 10_000;
private static final String PAYLOAD = "x".repeat(256);
@Autowired
EmbeddedKafkaBroker broker;
@Test
void transactionOverheadIsPerTransactionNotPerRecord() {
Report.title("The cost of a transaction is paid per COMMIT, not per record");
Report.line("Sending %,d records of %d bytes in every configuration.", RECORDS, PAYLOAD.length());
// ------------------------------------------------------------------------------------
// Measurement design, and why it is not a simple loop.
//
// Version 1 warmed up briefly, then swept batch sizes in increasing order. The largest
// batch came out FIVE TIMES faster than the non-transactional baseline -- impossible.
// Version 2 added a full-size warm-up and re-measured the baseline at the end, which
// revealed the reason: the baseline had moved +181% between the start and end of the run.
// The embedded broker, the JIT and the page cache were all still warming up, and every
// configuration was silently being credited with the warm-up that preceded it.
//
// Version 3 (this one) INTERLEAVES: each round runs every configuration once, in the same
// order, and the reported figure is the median across rounds. Warm-up drift now applies
// roughly equally to all configurations instead of accruing to whichever ran last, and the
// drift check compares the first round's baseline against the last round's.
// ------------------------------------------------------------------------------------
final int[] batchSizes = {1, 10, 100, 1_000, 10_000};
final int rounds = 5;
Report.section("warm-up (discarded)");
runNonTransactional(RECORDS);
runTransactional(RECORDS, 500, "warmup-tx-" + System.nanoTime());
Report.bullet("done");
Map<String, java.util.List<Double>> samples = new LinkedHashMap<>();
samples.put("no transaction (idempotent)", new java.util.ArrayList<>());
for (int b : batchSizes) {
samples.put("%,d records/transaction".formatted(b), new java.util.ArrayList<>());
}
// Round 0 is measured and printed but NOT counted. Even after the explicit warm-up above,
// the first counted round consistently came in ~60% below the rest, and including it
// dragged every median down and made the drift check meaningless. Printing it keeps the
// decision visible rather than hidden.
Report.section("%d interleaved rounds of %,d records (round 0 discarded)".formatted(rounds, RECORDS));
for (int round = 0; round <= rounds; round++) {
StringBuilder row = new StringBuilder("round %d%s: ".formatted(round, round == 0 ? " (discarded)" : ""));
double base = runNonTransactional(RECORDS);
if (round > 0) samples.get("no transaction (idempotent)").add(base);
row.append("baseline=%,.0f".formatted(base));
for (int b : batchSizes) {
double rate = runTransactional(RECORDS, b, "perf-tx-" + b + "-" + System.nanoTime());
if (round > 0) samples.get("%,d records/transaction".formatted(b)).add(rate);
row.append(" %d/tx=%,.0f".formatted(b, rate));
}
Report.bullet("%s", row);
}
Map<String, Double> results = new LinkedHashMap<>();
samples.forEach((label, values) -> results.put(label, median(values)));
java.util.List<Double> baselineSamples = samples.get("no transaction (idempotent)");
double baselineFirstRound = baselineSamples.getFirst();
double baselineLastRound = baselineSamples.getLast();
double drift = 100.0 * (baselineLastRound - baselineFirstRound) / baselineFirstRound;
// ------------------------------------------------------------------------------------
// Stability report rather than a stability assertion.
//
// The non-transactional baseline is genuinely noisy on this setup -- it varies by roughly
// +/-50% between rounds, because sending 10,000 records with no commit is fast enough that
// the measurement is dominated by flush timing and OS scheduling. Asserting on its
// stability just produces a flaky test.
//
// The TRANSACTIONAL numbers, by contrast, are very stable, because each is dominated by a
// fixed number of commits. So the defensible claim of this benchmark is not any ratio
// against the baseline -- it is the ORDERING and SPACING among transaction sizes, which is
// what the assertions below check.
// ------------------------------------------------------------------------------------
Report.section("measurement stability");
samples.forEach((label, values) -> {
double min = values.stream().mapToDouble(Double::doubleValue).min().orElse(0);
double max = values.stream().mapToDouble(Double::doubleValue).max().orElse(0);
Report.bullet("%-30s median=%-12s spread=%+.0f%%",
label, "%,.0f".formatted(median(values)), 100.0 * (max - min) / median(values));
});
Report.bullet("baseline first counted round : %,.0f records/s", baselineFirstRound);
Report.bullet("baseline last counted round : %,.0f records/s", baselineLastRound);
Report.bullet("baseline drift : %+.1f%%", drift);
double baseline = results.get("no transaction (idempotent)");
Report.section("summary (median of %d rounds)".formatted(rounds));
Report.line(" %-32s %14s %12s", "configuration", "records/s", "vs baseline");
results.forEach((label, rate) ->
Report.line(" %-32s %14s %11.0f%%",
label, "%,.0f".formatted(rate), 100.0 * rate / baseline));
double perRecord = results.get("1 records/transaction");
double perTen = results.get("10 records/transaction");
double perHundred = results.get("100 records/transaction");
double perThousand = results.get("1,000 records/transaction");
// The real claim: throughput rises monotonically with transaction size, and does so by
// close to the batching factor while commit cost dominates.
assertThat(perTen).as("10/tx beats 1/tx").isGreaterThan(perRecord);
assertThat(perHundred).as("100/tx beats 10/tx").isGreaterThan(perTen);
assertThat(perThousand).as("1000/tx beats 100/tx").isGreaterThan(perHundred);
assertThat(perTen / perRecord)
.as("going from 1 to 10 records per transaction should be worth roughly 10x")
.isGreaterThan(4.0);
Report.takeaway("Records per transaction, and what it buys:");
Report.takeaway(" 1 -> %,9.0f rec/s", perRecord);
Report.takeaway(" 10 -> %,9.0f rec/s (%.1fx)", perTen, perTen / perRecord);
Report.takeaway(" 100 -> %,9.0f rec/s (%.1fx)", perHundred, perHundred / perTen);
Report.takeaway(" 1,000 -> %,9.0f rec/s (%.1fx)", perThousand, perThousand / perHundred);
Report.takeaway("");
Report.takeaway("Each 10x increase in transaction size buys close to 10x throughput while");
Report.takeaway("commit cost dominates, then flattens as the per-record cost takes over.");
Report.takeaway("One record per transaction is roughly %.0fx slower than a hundred.",
perHundred / perRecord);
Report.takeaway("");
Report.takeaway("(The non-transactional baseline is quoted at %,.0f rec/s but varied by", baseline);
Report.takeaway("%.0f%% across rounds, so ratios against it are indicative only. The",
100.0 * (baselineSamples.stream().mapToDouble(Double::doubleValue).max().orElse(0)
- baselineSamples.stream().mapToDouble(Double::doubleValue).min().orElse(0)) / baseline);
Report.takeaway("comparisons BETWEEN transaction sizes are the stable, meaningful ones.)");
Report.takeaway("");
Report.takeaway("The lesson is not 'transactions are slow'. It is that a commit is a fixed");
Report.takeaway("cost, so the number of records you put inside one is the setting that");
Report.takeaway("matters. In Spring, that number is the size of the batch returned by");
Report.takeaway("poll(), which you control with max.poll.records -- NOT with anything");
Report.takeaway("named 'transaction'. Raising max.poll.records is the single most");
Report.takeaway("effective throughput fix for a transactional listener, at the price of");
Report.takeaway("redoing more work when one record in the batch fails.");
}
// ---------------------------------------------------------------------------------------
private double runNonTransactional(int records) {
try (Producer<String, String> producer = producer(null)) {
// Force the metadata fetch BEFORE the timer starts. The transactional path gets this
// for free inside initTransactions(), so without this line the non-transactional
// baseline pays a fixed cost its competitor does not -- which made large transactions
// appear several times FASTER than sending with no transaction at all.
producer.partitionsFor(TOPIC);
long start = System.nanoTime();
for (int i = 0; i < records; i++) {
producer.send(new ProducerRecord<>(TOPIC, "k", PAYLOAD));
}
producer.flush();
return rate(records, start);
}
}
private double runTransactional(int records, int batchSize, String transactionalId) {
try (Producer<String, String> producer = producer(transactionalId)) {
producer.initTransactions();
producer.partitionsFor(TOPIC); // symmetry with the baseline above
long start = System.nanoTime();
int sent = 0;
while (sent < records) {
producer.beginTransaction();
int thisBatch = Math.min(batchSize, records - sent);
for (int i = 0; i < thisBatch; i++) {
producer.send(new ProducerRecord<>(TOPIC, "k", PAYLOAD));
}
producer.commitTransaction();
sent += thisBatch;
}
return rate(records, start);
}
}
private static double rate(int records, long startNanos) {
return records / ((System.nanoTime() - startNanos) / 1e9);
}
/**
* Median of a sample list.
*
* <p>The median rather than the mean, because a single unlucky round -- one background process,
* one thermal event, one long GC -- moves a mean by tens of percent and a median not at all.
*/
private static double median(java.util.List<Double> values) {
java.util.List<Double> sorted = new java.util.ArrayList<>(values);
java.util.Collections.sort(sorted);
int n = sorted.size();
return n % 2 == 1 ? sorted.get(n / 2) : (sorted.get(n / 2 - 1) + sorted.get(n / 2)) / 2.0;
}
private Producer<String, String> producer(String transactionalId) {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// Held constant across every configuration so the only variable is the transaction.
props.put(ProducerConfig.LINGER_MS_CONFIG, 5);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
if (transactionalId != null) {
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId);
props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 60_000);
}
return new KafkaProducer<>(props);
}
}

40
src/test/java/com/ankurm/kafka/eos/support/Report.java Normal file
View File

@@ -0,0 +1,40 @@
package com.ankurm.kafka.eos.support;
/**
* Tiny formatting helper so every demo prints a transcript with the same shape.
*
* <p>This exists because the output of these tests <em>is</em> the deliverable: the numbers quoted
* in the blog post are copied from {@code results/}, and a reader should be able to diff their own
* run against the committed one without wading through formatting differences.
*/
public final class Report {
private Report() {
}
public static void title(String text) {
System.out.println();
System.out.println("=".repeat(78));
System.out.println(text);
System.out.println("=".repeat(78));
}
public static void section(String text) {
System.out.println();
System.out.println("-- " + text + " " + "-".repeat(Math.max(0, 74 - text.length())));
}
public static void line(String format, Object... args) {
System.out.printf(format + "%n", args);
}
public static void bullet(String format, Object... args) {
System.out.printf(" " + format + "%n", args);
}
/** A conclusion the reader is meant to take away, marked so it stands out in the transcript. */
public static void takeaway(String format, Object... args) {
System.out.println();
System.out.printf(">> " + format + "%n", args);
}
}

40
src/test/resources/logback-test.xml Normal file
View File

@@ -0,0 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
The demos in this repository print narrated output that is meant to be read as a transcript.
Kafka logs every producer and consumer configuration at INFO, which buries that narration under
a few hundred lines per client. Everything noisy is turned down here; the demos' own logger and
the handful of Kafka loggers that say something genuinely interesting are kept.
-->
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} %-5level %logger{28} - %msg%n</pattern>
</encoder>
</appender>
<!-- Kafka clients: silence the config dumps, keep warnings and errors. -->
<logger name="org.apache.kafka" level="WARN"/>
<logger name="kafka" level="WARN"/>
<logger name="state.change.logger" level="WARN"/>
<!-- Spring: quiet unless something is wrong. -->
<logger name="org.springframework" level="WARN"/>
<logger name="org.springframework.kafka" level="WARN"/>
<logger name="org.hibernate" level="WARN"/>
<logger name="com.zaxxer.hikari" level="WARN"/>
<!--
Transaction lifecycle is the subject of this repository, so keep the two loggers that
narrate it. Raise these to DEBUG when you want to see every begin/commit/abort.
-->
<logger name="org.springframework.kafka.transaction" level="INFO"/>
<logger name="org.apache.kafka.clients.producer.internals.TransactionManager" level="WARN"/>
<!-- The demos themselves. -->
<logger name="com.ankurm.kafka" level="INFO"/>
<root level="WARN">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>