commit e23ca359ebcca088126846f85119540a2b0a62f8 Author: Ankur Date: Fri Jul 31 22:27:35 2026 +0530 Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker (spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited output of mvn test: 15 tests, 0 failures. Demonstrations _01_idempotence verifies idempotence from the BROKER via Admin.describeProducers rather than by reading config back: a default producer registers one producer state, an acks=1 producer registers zero. Also covers the ConfigException you get only when idempotence is requested explicitly. _02_transactions transaction markers consuming offsets, aborted records surviving in the log, read_committed vs read_uncommitted, and a correction: markers do NOT inflate consumer lag (a drained consumer reaches zero), they break counting records by offset arithmetic (measured 100% overstatement). _03_read_process_write the Spring loop with container-managed transactions; a listener that fails after sending runs twice and is committed once. _04_fencing shared transactional.id fences the zombie; a random per-startup id lets both producers commit and silently defeats the guarantee entirely. _05_database Kafka EOS does not extend to a database. Naive listener writes 2 rows for 1 message; the same listener keyed on topic-partition-offset writes 1. _06_performance cost per transaction size, median of 5 interleaved rounds. Docs Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka), idempotence, transactions and markers, fencing and KIP-890, the database boundary and the outbox pattern, performance, and corner cases. Measured highlights acks=1 producer 0 broker-tracked producer states (idempotence silently off) 5 records in one tx endOffset 6; offset span overstates record count by 100% 1 record per tx 711 records/s 100 records per tx 78,770 records/s (~111x) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ed8fdbd --- /dev/null +++ b/.gitignore @@ -0,0 +1,18 @@ +# build output +target/ +*.class + +# local run scratch files (the curated transcript lives in results/) +perf.txt +perf-err.txt +rpw.txt +diag.txt +results/full-run-err.txt + +# editors +.idea/ +*.iml +.vscode/ +.DS_Store + +# results/ IS committed on purpose -- it is the evidence for the blog post diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..aa5473f --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Ankur Mhatre + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..765ee7f --- /dev/null +++ b/README.md @@ -0,0 +1,111 @@ +# Exactly-once with Spring Kafka on Spring Boot 4 + +Companion repository for **[Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions](https://ankurm.com/spring-kafka-exactly-once-boot-4/)** +on [ankurm.com](https://ankurm.com). + +The blog post covers the concepts most people need. **This repository covers everything** — every +demonstration, every failure mode, every corner case, as runnable tests with real captured output. + +Verified against **Spring Boot 4.1.0**, **Spring Kafka 4.1.0**, **kafka-clients 4.2.1**, **JDK 25.0.3**. +Raw output of the full suite is in [`results/full-run.txt`](results/full-run.txt). + +--- + +## Quick start + +No Docker, no Kafka install. Every test starts its own in-JVM KRaft broker. + +```console +git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git +cd spring-kafka-exactly-once-boot4 +mvn test +``` + +15 tests, about three minutes, mostly the performance sweep. + +Run one demonstration at a time: + +```console +mvn test -Dtest=IdempotentProducerTest +mvn test -Dtest=TransactionMarkerOffsetTest +mvn test -Dtest=ZombieFencingTest +mvn test -Dtest=DatabaseBoundaryTest +mvn test -Dtest=TransactionCostTest +``` + +--- + +## The demonstrations + +| Test | Shows | Key measured result | +|---|---|---| +| [`IdempotentProducerTest`](src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) | Idempotence verified **from the broker** via `describeProducers`, and the config that silently disables it | default producer → 1 tracked producer state; `acks=1` → **0** | +| [`TransactionMarkerOffsetTest`](src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java) | Transaction markers, aborted records, `read_committed` vs `read_uncommitted`, offset arithmetic | 5 records → `endOffset` 6; offset span overstates record count by **100%** | +| [`ReadProcessWriteTest`](src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) | The Spring read-process-write loop, and rollback + redelivery | listener ran **2x**, `read_committed` consumer sees **1** | +| [`ZombieFencingTest`](src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) | What `transactional.id` is for, and how a random one destroys the guarantee | shared id → `ProducerFencedException`; random id → **both commit** | +| [`DatabaseBoundaryTest`](src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) | Kafka EOS does not cover your database, and the fix | naive listener → **2 rows**; idempotent listener → **1 row** | +| [`TransactionCostTest`](src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) | What transactions cost, and the setting that controls it | 1 record/tx = **711 rec/s**; 100/tx = **78,770 rec/s** | + +## The documentation + +| Chapter | Covers | +|---|---| +| [1. Boot 4 setup](docs/01-boot4-setup.md) | The `spring-boot-starter-kafka` trap, the one property that enables EOS, testing without Docker | +| [2. Idempotence](docs/02-idempotence.md) | PIDs and sequence numbers, the `acks=1` silent downgrade, the constraints | +| [3. Transactions and markers](docs/03-transactions-and-markers.md) | `sendOffsetsToTransaction`, isolation levels, control records, the Last Stable Offset | +| [4. Fencing](docs/04-fencing.md) | `transactional.id`, epochs, EOSMode V2, KIP-890 on Kafka 4.x | +| [5. The database boundary](docs/05-database-boundary.md) | Why there is no 2PC, deprecated `ChainedKafkaTransactionManager`, dedupe keys, the outbox pattern | +| [6. Performance](docs/06-performance.md) | Measured cost per transaction size, and why `max.poll.records` is the knob | +| [7. Corner cases](docs/07-corner-cases.md) | Timeouts, error handlers, DLTs, compaction, cross-cluster, Kafka Streams | + +--- + +## Six findings you will not read elsewhere + +1. **Boot 4 moved Kafka auto-configuration out of `spring-boot-autoconfigure`.** Depend on + `spring-kafka` alone and you get a `NoSuchBeanDefinitionException` for `KafkaTemplate` that looks + like a generics problem. Use `spring-boot-starter-kafka`. + → [docs/01](docs/01-boot4-setup.md) +2. **`acks=1` silently disables idempotence — proven from the broker, not the config.** + `Admin.describeProducers()` shows 1 tracked producer state for a default producer and **0** for an + `acks=1` producer. Setting `enable.idempotence=true` explicitly turns that silent downgrade into + a startup failure. + → [docs/02](docs/02-idempotence.md) +3. **Transactions do *not* inflate consumer lag.** This widely repeated claim is wrong: a drained + `read_committed` consumer reaches exactly zero lag, because its position advances over markers + too. What markers *do* break is counting records by offset arithmetic — measured here at **100%** + overstatement. + → [docs/03](docs/03-transactions-and-markers.md) +4. **A random `transactional.id` per start-up silently defeats fencing.** Measured: two producers + with different UUIDs both commit, and nothing is fenced. Everything appears to work while + delivering exactly the duplicate processing you enabled transactions to prevent. + → [docs/04](docs/04-fencing.md) +5. **Aborted records reach the log only if they were already flushed.** `abortTransaction()` + discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends + on timing. Rely on `read_committed`, not on either behaviour. + → [docs/03](docs/03-transactions-and-markers.md) §3.6 +6. **One record per transaction is ~111x slower than one hundred.** The cost is per commit, not per + record, and the setting that controls it in Spring is `max.poll.records` — not anything named + "transaction". + → [docs/06](docs/06-performance.md) + +--- + +## Reference machine + +| | | +|---|---| +| CPU | AMD Ryzen 5 5600U, 6 cores / 12 threads | +| RAM | 15.3 GB | +| OS | Windows 11 Home | +| JDK | `java 25.0.3+9-LTS-195` | +| Kafka | in-JVM `EmbeddedKafkaKraftBroker`, kafka-clients 4.2.1, `transaction.version=2` | + +A laptop and a single-node broker. The absolute throughput figures are illustrative; the +relationships are what generalise. See [docs/06](docs/06-performance.md) §6.3 for what to trust. + +--- + +## Licence + +MIT. See [LICENSE](LICENSE). diff --git a/docs/01-boot4-setup.md b/docs/01-boot4-setup.md new file mode 100644 index 0000000..daf4d07 --- /dev/null +++ b/docs/01-boot4-setup.md @@ -0,0 +1,121 @@ +# 1. Setting up exactly-once on Spring Boot 4 + +> Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. + +## 1.1 The dependency trap that will cost you an hour + +On Spring Boot 3 this was enough: + +```xml + + org.springframework.kafka + spring-kafka + +``` + +…because `KafkaAutoConfiguration` lived in the monolithic `spring-boot-autoconfigure` jar that every +Boot application already had on the classpath. + +**Boot 4 split auto-configuration into per-technology modules.** Kafka's now lives in +`org.springframework.boot:spring-boot-kafka`. Depend on `spring-kafka` alone and you get the +library but *no* auto-configuration: no `ProducerFactory`, no `KafkaTemplate`, no +`KafkaTransactionManager`. + +The failure looks like this, and it looks like a generics problem, which it is not: + +``` +Error creating bean with name 'orderProcessor': Unsatisfied dependency expressed through +constructor parameter 0: No qualifying bean of type +'org.springframework.kafka.core.KafkaTemplate' available +``` + +Use the starter: + +```xml + + org.springframework.boot + spring-boot-starter-kafka + +``` + +which pulls in both `spring-kafka` and `spring-boot-kafka`. + +## 1.2 The configuration + +The whole exactly-once switch is one property: + +```yaml +spring: + kafka: + producer: + transaction-id-prefix: order-processor-tx- +``` + +Spring Boot sees it, auto-configures a `KafkaTransactionManager`, and attaches it to every listener +container. From that point the container begins a Kafka transaction before invoking your listener, +writes the consumer offsets *into* that transaction, and commits — or aborts everything if your +listener throws. + +Two more properties matter and neither is on by default: + +```yaml +spring: + kafka: + consumer: + isolation-level: read_committed # default is read_uncommitted! + enable-auto-commit: false +``` + +`isolation-level` is the one people forget. Producing transactionally while consuming +`read_uncommitted` gives you all of the cost of transactions and none of the benefit — your +consumers will happily read records from transactions that were aborted. Proven in +[`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java). + +## 1.3 Config worth stating even though it is already the default + +```yaml + acks: all + properties: + enable.idempotence: true + max.in.flight.requests.per.connection: 5 +``` + +All three are defaults on Kafka 3.0+. Stating them explicitly is still worth it, because +`enable.idempotence: true` converts a *silent* downgrade into a *startup failure* if anyone later +sets `acks=1` for latency. See [02-idempotence.md](02-idempotence.md) — this is the single most +common way a system that believes it is exactly-once is not. + +## 1.4 Testing without Docker + +Kafka 4.x removed ZooKeeper entirely, so `spring-kafka-test` now only ships +`EmbeddedKafkaKraftBroker`. It runs a real broker in-JVM: + +```java +@SpringBootTest(classes = Application.class) +@EmbeddedKafka( + partitions = 1, + topics = {"orders.incoming", "orders.validated"}, + brokerProperties = { + "transaction.state.log.replication.factor=1", + "transaction.state.log.min.isr=1", + "offsets.topic.replication.factor=1" + }) +class MyTest { } +``` + +**Those three `brokerProperties` are mandatory for any transactional test.** The transaction state +log defaults to 3 replicas, which a single-broker embedded cluster cannot provide. Without them +every transactional test fails with a `COORDINATOR_NOT_AVAILABLE` that gives no hint as to why. + +Two further limitations of the KRaft embedded broker: + +- **You cannot pin ports.** `KafkaClusterTestKit` does not support it, so the `ports` attribute of + `@EmbeddedKafka` is ignored. Read the address from `EmbeddedKafkaBroker.getBrokersAsString()` or + the `spring.embedded.kafka.brokers` system property. +- Broker start-up costs a couple of seconds per distinct configuration. Spring's context caching + will reuse a broker across test classes whose configuration matches exactly, so keeping + `@EmbeddedKafka` annotations identical across classes is a real speed-up. + +--- + +Next: [02-idempotence.md](02-idempotence.md). diff --git a/docs/02-idempotence.md b/docs/02-idempotence.md new file mode 100644 index 0000000..8b867fd --- /dev/null +++ b/docs/02-idempotence.md @@ -0,0 +1,105 @@ +# 2. Idempotent producers + +> Runnable companion: [`IdempotentProducerTest`](../src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java) + +## 2.1 The problem it solves + +A producer sends a record. The broker writes it. The acknowledgement is lost on the way back — a +network blip, a leader election, a GC pause. The producer retries. The broker now holds the record +**twice**, and no amount of consumer-side care can undo that: the duplicate is a real, distinct +record with its own offset and its own content. + +This is not exotic. It is the normal consequence of retries on an unreliable network, and retries +are on by default because without them you drop records instead. + +## 2.2 How it works + +The producer obtains a **producer ID (PID)** from the broker and stamps every record batch with a +**per-partition sequence number**. The broker tracks the last accepted sequence per PID per +partition and rejects a repeat. A retry of a batch the broker already has is silently dropped +instead of appended. + +The guarantee has a precise scope: **exactly-once delivery per partition, for the lifetime of one +producer session.** It does not survive a producer restart (a new PID is issued), and it says +nothing about ordering across partitions. + +## 2.3 It has been the default since Kafka 3.0 + +```java +ProducerConfig.configDef().defaultValues().get("enable.idempotence") // true +ProducerConfig.configDef().defaultValues().get("acks") // all +``` + +You can verify it from the broker's side rather than trusting the config, which is what the test in +this repository does: + +```java +admin.describeProducers(List.of(partition)) + .partitionResult(partition).get().activeProducers(); +``` + +A producer with idempotence on appears there with a real producer ID, epoch and last sequence +number. A non-idempotent producer creates no such state, because there is nothing to de-duplicate +against. Measured output: + +``` +-- default producer -- + producerId=0 producerEpoch=0 lastSequence=2 + producer states tracked : 1 + +-- acks=1 producer -- + producer states tracked : 0 +``` + +## 2.4 The silent downgrade + +`acks` and `enable.idempotence` interact, and — this is the important part — they interact +**differently depending on whether you set idempotence explicitly**: + +| Configuration | Result | +|---|---| +| `acks=1` only | Idempotence **silently disabled**. No warning, no exception. | +| `acks=1` + `enable.idempotence=true` | `ConfigException` at construction. | + +``` +ConfigException: Must set acks to all in order to use the idempotent producer. +Otherwise we cannot guarantee idempotence. +``` + +The first row is the dangerous one and it is extremely common: someone set `acks=1` for latency, in +a properties file, years ago, and the system has been quietly at-least-once ever since. Nothing in +the logs says so. + +**Therefore: set `enable.idempotence=true` explicitly even though it is the default.** Not because +it changes behaviour, but because it converts a silent downgrade into a startup failure. That is +the only reason, and it is a good one. + +## 2.5 The other constraints + +| Setting | Requirement | Fails how | +|---|---|---| +| `acks` | must be `all` | loudly if idempotence explicit, silently otherwise | +| `retries` | must be `> 0` | `ConfigException` | +| `max.in.flight.requests.per.connection` | must be `<= 5` | `ConfigException` | + +Five in-flight requests is safe *with* idempotence because sequence numbers let the broker reorder +within that window. **Without** idempotence, `max.in.flight > 1` silently breaks ordering on retry: +a failed batch that is retried can land after a batch that was sent later. That is a second, +independent reason not to disable idempotence. + +## 2.6 What idempotence is not + +It is **not** exactly-once processing. It protects one producer session writing to one partition. +It does nothing about: + +- a producer restarting and re-sending from an application-level buffer, +- a consumer processing a record twice after a rebalance, +- offsets and output records committing atomically, +- anything your listener writes to a database. + +For those you need transactions ([03](03-transactions-and-markers.md)), fencing +([04](04-fencing.md)) and idempotent side effects ([05](05-database-boundary.md)). + +--- + +Next: [03-transactions-and-markers.md](03-transactions-and-markers.md). diff --git a/docs/03-transactions-and-markers.md b/docs/03-transactions-and-markers.md new file mode 100644 index 0000000..8b4d91a --- /dev/null +++ b/docs/03-transactions-and-markers.md @@ -0,0 +1,138 @@ +# 3. Transactions, isolation, and the markers nobody mentions + +> Runnable companion: [`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java), +> [`ReadProcessWriteTest`](../src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java) + +## 3.1 What a transaction adds over idempotence + +Idempotence protects one producer session writing to one partition. A **transaction** makes a set of +writes atomic across partitions — *and*, crucially, lets a consumer offset commit be one of those +writes. + +That last part is the whole trick behind read-process-write. The offset for the input record is +written **into the same transaction** as the output record, via `sendOffsetsToTransaction`. There is +no window between "produced the output" and "committed the offset" in which a crash produces a +duplicate, because there is no between. + +Setting `transactional.id` on a producer also forces `enable.idempotence=true` — transactions are +built on top of idempotence, not as an alternative to it. + +## 3.2 In Spring, you write none of that + +```yaml +spring.kafka.producer.transaction-id-prefix: order-processor-tx- +spring.kafka.consumer.isolation-level: read_committed +``` + +```java +@KafkaListener(topics = "orders.incoming", groupId = "order-processor") +public void process(String order) { + kafkaTemplate.send("orders.validated", order.toUpperCase()); +} +``` + +The container begins the transaction before the listener runs, enlists the `KafkaTemplate` send, +sends the offsets into the transaction, and commits — or aborts everything if the method throws. +The listener body is deliberately ordinary. **The guarantee is entirely in the configuration.** + +## 3.3 `isolation.level` defaults to the wrong thing + +`read_uncommitted` is the default. A consumer reading a transactional topic with the default +configuration will happily deliver records from transactions that were **aborted**. + +Measured, on one partition, after committing 5 records and aborting 3: + +``` +read_committed : 5 records [committed-0 .. committed-4] +read_uncommitted : 8 records [committed-0 .. committed-4, aborted-0, aborted-1, aborted-2] +``` + +`isolation.level` is a **consumer-side filter**, not a broker-side delete. The aborted records are +physically in the log with permanent offsets and they consume disk. Producing transactionally +without setting `read_committed` on every consumer gives you the cost of transactions and none of +the benefit. + +## 3.4 Transaction markers consume offsets + +When a transaction commits or aborts, Kafka writes a **control record** — a transaction marker — +into every partition it touched. Control records are real log entries that **consume an offset**, +but are never delivered to your consumer. + +``` +records sent : 5 +endOffset : 6 <- records occupy 0..4; the commit marker takes 5 +last delivered offset : 4 +``` + +Three consequences: + +**Offsets are not contiguous.** Send five records in a transaction and the next record is at offset +6, not 5. Any code assuming `offset + 1` is the next record is wrong. In the Spring +read-process-write test, three messages processed in three transactions landed at offsets **0, 2 and +4** — a marker between each. + +**Counting records by offset arithmetic is broken.** After 5 committed records, 3 aborted records +and 2 markers: + +``` +beginningOffset : 0 +endOffset : 10 +end - beginning : 10 <- looks like the record count +records actually retrievable: 5 +overstatement : 100% +``` + +Retention estimates, "how many events between these offsets", partition-size dashboards and tests +that assert on offsets are all wrong by however much your marker and abort rate happens to be. + +**Consumer lag, however, is fine.** This is worth stating because the opposite is widely repeated. +A drained `read_committed` consumer's *position* advances over markers and aborted records exactly +as it advances over delivered ones, so `endOffset - committedOffset` reaches zero: + +``` +partition end offset : 10 +consumer committed offset : 10 +lag : 0 +``` + +The claim that transactions inflate consumer lag does not survive contact with a broker. The claim +that they break offset arithmetic does. + +## 3.5 The Last Stable Offset, and why transaction duration is a consumer-latency setting + +A `read_committed` consumer can only read up to the **Last Stable Offset** — the offset before the +first still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every +`read_committed` consumer on that partition stalls for 30 seconds, *including for records committed +by other producers in the meantime*. + +So `transaction.timeout.ms` and how long you keep a transaction open are consumer-latency knobs, not +just producer concerns. Long-running transactions are the most common cause of "our consumers +randomly freeze for exactly N seconds". + +## 3.6 Aborted records only reach the log if they were flushed + +A detail that surprised this repository's own test. `KafkaTemplate.send()` is asynchronous. If the +transaction aborts while the record is still sitting in the producer's accumulator, +`abortTransaction()` **discards the un-flushed batch** and the record never reaches the broker at +all. + +So aborted records occupy permanent offsets only when they were already flushed — which happens +under load, on large batches, or when you block on the send future. Both behaviours are correct and +neither is something to rely on. Rely on `read_committed`. + +## 3.7 Exactly-once means committed once, not executed once + +The clearest demonstration in this repository: a listener that fails *after* sending its output. + +``` +times the listener ran for 'delta' : 2 +DELTA records visible (read_committed) : 1 +``` + +The work happened twice. One result was committed. **That is what exactly-once means in Kafka, and +it has never meant that your code runs once.** Every side effect outside Kafka happened twice — +see [05-database-boundary.md](05-database-boundary.md). + +--- + +Next: [04-fencing.md](04-fencing.md). diff --git a/docs/04-fencing.md b/docs/04-fencing.md new file mode 100644 index 0000000..7254051 --- /dev/null +++ b/docs/04-fencing.md @@ -0,0 +1,91 @@ +# 4. Zombie fencing and `transactional.id` + +> Runnable companion: [`ZombieFencingTest`](../src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java) + +## 4.1 The zombie + +A processor reads, transforms and writes. It stalls — a long GC pause, a network partition, a +container the orchestrator has decided is dead. A replacement starts, the group rebalances, the new +instance owns the partition. + +Then the old instance wakes up. It holds a partially completed transaction and does not know it was +replaced. If it were allowed to commit, the same input would be processed twice and both results +would be committed. That is the zombie. + +## 4.2 The defence + +`transactional.id` is a **stable, durable identity** the transaction coordinator remembers, along +with an **epoch**. Every `initTransactions()` for an existing id bumps the epoch and invalidates +anything holding the old one. + +Measured: + +``` +Producer A: initTransactions, beginTransaction, send (transaction OPEN) +Producer B: initTransactions with the SAME id (epoch bumped) +Producer A: commitTransaction + -> REJECTED with ProducerFencedException +Producer B: commits normally +``` + +The zombie's work is discarded **by the broker**, not by any code you wrote. + +## 4.3 The two ways to get it wrong + +### Random per start-up — fencing never happens + +```java +// Catastrophic, and everything appears to work +props.put(TRANSACTIONAL_ID_CONFIG, "processor-" + UUID.randomUUID()); +``` + +Measured: producer A opens a transaction, producer B starts with a fresh UUID and commits, then A +commits too. **Nothing is fenced. Both commit.** The coordinator saw two unrelated identities and +had no reason to fence either. + +This is the most damaging Kafka transactions misconfiguration precisely because it is invisible: +transactions commit, tests pass, and you have exactly the duplicate-processing problem you turned +transactions on to prevent. + +### Shared across instances — everything fences everything + +Every instance bumps every other instance's epoch continuously, and the group thrashes with +`ProducerFencedException`s. + +## 4.4 Getting it right in Spring + +```yaml +spring.kafka.producer.transaction-id-prefix: order-processor-tx- +``` + +Spring appends a suffix per producer. The prefix must be **stable across restarts** and **unique per +application instance** — typically the application name plus a stable instance identifier +(a StatefulSet ordinal, a pod index, a configured node id), *not* a random value and *not* a value +shared by every replica. + +Since Spring Kafka 3.2, `TransactionIdSuffixStrategy` lets you control suffix allocation, which +matters when producers are cached and reused across partitions. + +> With `EOSMode.V2` — the only mode since Spring Kafka 3.0 — the producer no longer needs a +> `transactional.id` tied to the specific input partition. V2 (KIP-447) uses consumer group metadata +> for fencing, so one producer per instance is sufficient. Older guidance about +> `transactional.id` per topic-partition describes the removed `EOSMode.ALPHA` and does not apply. + +## 4.5 Kafka 4.x: KIP-890 changes which exception you see + +[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890) strengthened the transaction +protocol against hanging transactions. Under `transaction.version=2` — the default on Kafka 4.x, and +what the embedded broker in this repository reports at start-up — the **producer epoch is bumped on +every transaction**, not only at `initTransactions()`. + +Practical effect: the exception on fencing may be `InvalidProducerEpochException` rather than the +`ProducerFencedException` that older articles promise. Catch both, or catch neither and let the +container restart the producer, which is what Spring does. + +A hanging transaction is worth understanding because its blast radius is large: it holds the Last +Stable Offset back, which stalls every `read_committed` consumer on the partition and prevents log +compaction. KIP-890 exists to make that class of bug impossible rather than merely rare. + +--- + +Next: [05-database-boundary.md](05-database-boundary.md). diff --git a/docs/05-database-boundary.md b/docs/05-database-boundary.md new file mode 100644 index 0000000..a5bfefb --- /dev/null +++ b/docs/05-database-boundary.md @@ -0,0 +1,112 @@ +# 5. The boundary: Kafka, your database, and the thing that cannot be done + +> Runnable companion: [`DatabaseBoundaryTest`](../src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java) + +## 5.1 State it plainly + +A Kafka transaction is atomic across **Kafka partitions and Kafka consumer offsets**. That is the +entire scope. It cannot include: + +- a database write, +- an HTTP call to another service, +- a file, +- an email, an SMS, a push notification, +- a cache invalidation. + +There is no two-phase commit between Kafka and your database, and there is not going to be. Kafka's +transaction coordinator has no protocol for enlisting a foreign resource manager. + +So: **a listener that writes to a database is at-least-once with respect to that database, no matter +what your Kafka configuration says.** The redelivery that makes Kafka's guarantee work is exactly +the thing that duplicates your row. + +## 5.2 What `ChainedKafkaTransactionManager` was, and why it is deprecated + +Spring used to ship `ChainedKafkaTransactionManager` for this, and it is deprecated along with its +parent `ChainedTransactionManager`. The reason is worth understanding rather than just obeying: +chaining transaction managers is **best-effort ordering, not distributed coordination**. It commits +one, then the other. If the process dies in between, they disagree, and nothing reconciles them. + +It narrows the window. It does not close it. A narrowed window is not a guarantee, and treating it +as one is worse than knowing you have none — because you will skip the idempotency work. + +## 5.3 What to do instead + +### Option A — idempotent writes (usually right) + +Give the work a natural identity and enforce it with a `UNIQUE` constraint. Then a duplicate +attempt is a no-op regardless of *why* it duplicated. + +```java +String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset); +if (!repository.existsByDedupeKey(dedupeKey)) { + repository.save(new ProcessedOrder(dedupeKey, payload)); +} +``` + +`topic-partition-offset` is a good default key: Kafka guarantees it is unique and stable, it is on +every consumer record, and it requires no coordination or payload changes. An event ID carried in +the message is better when the same logical event can arrive on more than one topic or be replayed +from a different offset. + +Do the check-and-insert inside the database transaction and let the `UNIQUE` constraint be the +real enforcement — the `exists` check is an optimisation, not the guarantee, because two concurrent +attempts can both pass it. + +Measured in `DatabaseBoundaryTest`: with a naive insert the listener ran twice and wrote **2 rows** +for one message; with the dedupe key it ran twice and wrote **1**. + +### Option B — the transactional outbox (when you cannot be idempotent) + +Invert the problem. The database is the source of truth: + +1. In **one local database transaction**, write your business change *and* a row into an `outbox` + table. +2. A separate relay — a poller, or change-data-capture such as Debezium — reads the outbox and + publishes to Kafka. +3. The relay is at-least-once, so consumers must tolerate duplicates, which idempotent consumers + already do. + +This gives genuine atomicity for the part that matters (business state and the intent to publish), +using only a local transaction. The cost is a table, a relay, and publish latency. + +### Option C — make the listener's side effect naturally idempotent + +Sometimes the work is already idempotent and nobody noticed: `UPDATE ... SET status='PAID'`, +`PUT /resource/123`, an upsert keyed by business ID, a `SET` in Redis. If the operation is +absorbing rather than accumulating, redelivery is harmless. Prefer designing for this over +detecting duplicates. + +## 5.4 Ordering, if you use `@Transactional` anyway + +With a `KafkaTransactionManager` on the container and `@Transactional` on the listener method, you +get: + +``` +container begins Kafka transaction + interceptor begins DB transaction + listener body runs + DB transaction COMMITS <-- first +Kafka transaction COMMITS <-- second +``` + +The DB commits first, deliberately: if the Kafka commit then fails, the offset is not advanced, the +record is redelivered, and an idempotent DB write absorbs it. Had Kafka committed first, a DB +failure would lose the write entirely with the offset already advanced. + +So `@Transactional` is still worth having — it makes the DB write roll back when your listener +throws, which is the common case. It just does not make the two systems atomic, and the crash +window between the two commits is real. Idempotency is what covers it. + +## 5.5 The rule + +> **Exactly-once processing = at-least-once delivery + idempotent side effects.** +> +> Kafka transactions give you the delivery half. You build the other half. + +Anyone who tells you Kafka gives you end-to-end exactly-once across your whole system is describing +Kafka-to-Kafka and has not thought about the database. + +--- + +Next: [06-performance.md](06-performance.md). diff --git a/docs/06-performance.md b/docs/06-performance.md new file mode 100644 index 0000000..9c6c85e --- /dev/null +++ b/docs/06-performance.md @@ -0,0 +1,96 @@ +# 6. What transactions actually cost + +> Runnable companion: [`TransactionCostTest`](../src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java) + +## 6.1 The useful version of "transactions are slow" + +The cost is **per transaction, not per record**. + +Committing requires a round trip to the transaction coordinator plus a durable marker write into +every partition the transaction touched, plus acknowledgement of those writes. That is a fixed cost. +It does not care whether the transaction contained one record or ten thousand. + +So the question is never "should I use transactions" in isolation. It is **"how many records go in +each one"**. + +## 6.2 Measured + +10,000 records of 256 bytes, single-node in-JVM broker, median of 5 interleaved rounds, +JDK 25.0.3 / Kafka 4.2.1 on an AMD Ryzen 5 5600U laptop: + +| Configuration | records/s | round-to-round spread | vs previous row | +|---|---:|---:|---:| +| no transaction (idempotent only) | 801,880 | 26% | — | +| 1 record / transaction | **711** | 11% | — | +| 10 records / transaction | 8,452 | 12% | **11.9x** | +| 100 records / transaction | 78,770 | 55% | **9.3x** | +| 1,000 records / transaction | 330,969 | 77% | 4.2x | +| 10,000 records / transaction | 878,071 | 32% | 2.7x | + +**One record per transaction is roughly 111x slower than one hundred**, and about three orders of +magnitude below the non-transactional baseline. + +Each 10x increase in transaction size buys close to 10x throughput while commit cost dominates, then +flattens as per-record cost takes over — the knee is somewhere around 1,000 records here. + +## 6.3 How to read these numbers honestly + +They come from a laptop and a single-node in-JVM broker, so the absolute throughput is not +meaningful. Two specific caveats: + +- The **non-transactional baseline is noisy** (26% spread, and it drifted 30% across the run). + Sending 10,000 records with no commit is fast enough that the measurement is dominated by flush + timing and OS scheduling. Ratios against the baseline are indicative only. +- The **transactional measurements are stable** because each is dominated by a fixed number of + commits. The comparisons *between* transaction sizes are the trustworthy part, and they are the + point. + +On a real cluster the per-commit cost is *larger* — real network, real replication, real disk — so +the penalty at small transaction sizes is worse than shown here and batching matters more. + +The test itself went through three revisions before producing numbers worth quoting; the reasons are +documented in its source, and they are the usual ones: insufficient warm-up, measurement order +accruing warm-up to whichever configuration ran last, and one path paying for a metadata fetch its +competitor had already done. If you adapt it, keep the interleaving and the drift report. + +## 6.4 The setting that controls this in Spring + +**`max.poll.records`.** Not anything named "transaction". + +With a `KafkaTransactionManager`, the container's transaction boundary is *the batch returned by +`poll()`*. So the number of records per transaction is exactly `max.poll.records` (default 500), +capped by how many are actually available. + +```yaml +spring: + kafka: + consumer: + max-poll-records: 500 # == records per transaction +``` + +Raising it is the single most effective throughput fix for a transactional listener. The price is +that a failure anywhere in the batch rolls back and redoes the whole batch, so you are trading +throughput against the cost of redoing work — and against `max.poll.interval.ms`, since the whole +batch must be processed within it. + +## 6.5 The other costs, which are not throughput + +- **Latency.** A `read_committed` consumer cannot read past the Last Stable Offset, so downstream + consumers see nothing until the transaction commits. Larger transactions mean higher end-to-end + latency. This is a direct trade against §6.4. +- **Storage.** Every transaction writes a marker per partition. High-frequency small transactions on + many partitions produce a surprising amount of control-record overhead. +- **Redelivery amplification.** One poisoned record in a batch of 500 rolls back all 500. +- **Coordinator load.** Every transaction is state on a broker, in `__transaction_state`. + +## 6.6 Rules of thumb + +- Do not use one transaction per record. It is three orders of magnitude off the achievable rate. +- Start at the default `max.poll.records=500` and tune from measurements, not from feeling. +- If latency matters more than throughput, lower it deliberately and know what you are buying. +- If you do not need atomic offset+output commits, do not use transactions at all — an idempotent + producer plus an idempotent consumer is cheaper and simpler, and covers most systems. + +--- + +Next: [07-corner-cases.md](07-corner-cases.md). diff --git a/docs/07-corner-cases.md b/docs/07-corner-cases.md new file mode 100644 index 0000000..6e546a3 --- /dev/null +++ b/docs/07-corner-cases.md @@ -0,0 +1,109 @@ +# 7. Corner cases and things that bite + +The blog post covers the main concepts. This chapter is the long tail — behaviours that do not fit a +narrative but do show up in production. + +## 7.1 `transaction.timeout.ms` vs `max.poll.interval.ms` + +The most common "it worked in dev" failure. Your listener must finish a whole poll batch, commit, +and get back to `poll()` inside **both** limits: + +| Setting | Default | Owned by | +|---|---|---| +| `transaction.timeout.ms` | 60,000 (Spring sets it; Kafka's producer default is 60s) | producer | +| `transaction.max.timeout.ms` | 900,000 | **broker** — caps the above | +| `max.poll.interval.ms` | 300,000 | consumer | + +If processing exceeds `transaction.timeout.ms`, the coordinator aborts your transaction and your +commit fails with `ProducerFencedException` / `InvalidProducerEpochException`. If it exceeds +`max.poll.interval.ms`, the consumer is ejected from the group and the partitions rebalance — +usually producing both symptoms at once and a very confusing log. + +Rule: `transaction.timeout.ms` should comfortably exceed the worst-case time to process one batch +of `max.poll.records`, and the broker's `transaction.max.timeout.ms` must be at least as large or +producer creation fails outright. + +## 7.2 A long transaction stalls other people's consumers + +Covered in [03](03-transactions-and-markers.md) §3.5 and worth repeating because it is +counter-intuitive: a `read_committed` consumer cannot read past the Last Stable Offset, which is +held back by the *oldest open transaction on the partition* — regardless of which producer opened +it. One slow producer degrades every consumer on that partition. + +## 7.3 `@KafkaListener` with manual acknowledgment does nothing useful + +With a `KafkaTransactionManager` present, `AckMode.MANUAL` / `MANUAL_IMMEDIATE` and calls to +`Acknowledgment.acknowledge()` are irrelevant: the offset commit happens inside the transaction, at +the end of the batch. Leaving manual ack code in place is harmless but misleading — future readers +will believe it controls something. + +## 7.4 `executeInTransaction` is not the same as being in a listener transaction + +```java +kafkaTemplate.executeInTransaction(t -> t.send(TOPIC, value)); +``` + +This starts a **local** producer transaction. It is the right way to produce transactionally from +outside a listener (a REST controller, a scheduled job, a test). Inside a listener that already has +a container-managed transaction, do **not** call it — plain `send()` enlists in the existing +transaction automatically. Nesting produces `IllegalStateException` or, worse, a second independent +transaction that commits separately. + +## 7.5 Error handling interacts with transactions + +`DefaultErrorHandler` retries in-process by default and then, if configured, publishes to a dead +letter topic via `DeadLetterPublishingRecoverer`. Two things to know: + +- The DLT publish happens **inside the same transaction** when a `KafkaTransactionManager` is + present, so "record moved to DLT and offset advanced" is atomic. That is the behaviour you want, + and it is why a DLT is the correct terminal state for a poison record under EOS. +- Retries consume `max.poll.interval.ms`. A back-off of 10 attempts × 30 s will eject the consumer + from the group long before it gives up. Use `DefaultErrorHandler` with a bounded back-off and let + the DLT absorb the rest. + +## 7.6 Producers are pooled, and `transactional.id` suffixes are allocated from that pool + +Spring caches producers. With transactions, each concurrent transaction needs its own +`transactional.id`, so Spring allocates `prefix + suffix`. `TransactionIdSuffixStrategy` +(Spring Kafka 3.2+) controls that allocation; the default reuses suffixes as producers return to the +cache. If you see unexpected fencing under high concurrency, look here before looking at your code. + +## 7.7 Consumer group `isolation.level` cannot be changed per-record + +It is a consumer-level setting. If some records on a topic must be visible before commit and others +after, you need two topics, not two isolation levels. + +## 7.8 Compacted topics and transactions + +Transaction markers interact with log compaction: a marker cannot be cleaned until the transaction +it belongs to is complete, and a **hanging** transaction therefore blocks compaction of the whole +partition indefinitely. This was one of the motivations for +[KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890). If a compacted topic stops +shrinking, check for open transactions before checking your compaction settings. + +## 7.9 Exactly-once across two Kafka clusters is not a thing + +A transaction is scoped to one cluster's transaction coordinator. MirrorMaker 2 and most replication +tooling are at-least-once across clusters. A cross-cluster pipeline is at-least-once end to end no +matter what each side does internally. + +## 7.10 Kafka Streams does this for you + +If your topology is genuinely read-process-write over Kafka only, `processing.guarantee=exactly_once_v2` +in Kafka Streams gives you all of the above with none of the wiring, including state store changelogs +in the same transaction. Reach for Spring Kafka transactions when you need to interleave non-Kafka +work or you are not building a streaming topology — not because Streams cannot do it. + +## 7.11 Things that are true and often assumed not to be + +- **Idempotence and transactions do not require a special broker configuration** beyond replication + factors adequate for `__transaction_state`. +- **`enable.idempotence` is on by default** (Kafka 3.0+) — see [02](02-idempotence.md). +- **A transactional producer can send to topics it never mentioned up front.** Partitions are added + to the transaction dynamically as you send. +- **Reading your own writes** inside a transaction is not possible — a `read_committed` consumer + will not see them until commit, including in the same process. + +--- + +Back to the [README](../README.md). diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..ef677e2 --- /dev/null +++ b/pom.xml @@ -0,0 +1,115 @@ + + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.0 + + + + com.ankurm.kafka + spring-kafka-exactly-once-boot4 + 1.0.0 + Exactly-once with Spring Kafka on Spring Boot 4 + + Runnable demonstrations of idempotent producers, Kafka transactions, read-process-write, + zombie fencing, transaction markers, and the Kafka/database boundary. + + + + 25 + UTF-8 + + + + + org.springframework.boot + spring-boot-starter + + + + + org.springframework.boot + spring-boot-starter-kafka + + + + + org.springframework.boot + spring-boot-starter-data-jpa + + + com.h2database + h2 + runtime + + + + org.springframework.boot + spring-boot-starter-test + test + + + + org.springframework.kafka + spring-kafka-test + test + + + org.awaitility + awaitility + test + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + false + false + + + + + diff --git a/results/full-run.txt b/results/full-run.txt new file mode 100644 index 0000000..a7c16e5 --- /dev/null +++ b/results/full-run.txt @@ -0,0 +1,690 @@ +[INFO] Scanning for projects... +[INFO] +[INFO] ----------< com.ankurm.kafka:spring-kafka-exactly-once-boot4 >---------- +[INFO] Building Exactly-once with Spring Kafka on Spring Boot 4 1.0.0 +[INFO] from pom.xml +[INFO] --------------------------------[ jar ]--------------------------------- +[INFO] +[INFO] --- resources:3.5.0:resources (default-resources) @ spring-kafka-exactly-once-boot4 --- +[INFO] Copying 1 resource from src\main\resources to target\classes +[INFO] Copying 0 resource from src\main\resources to target\classes +[INFO] +[INFO] --- compiler:3.15.0:compile (default-compile) @ spring-kafka-exactly-once-boot4 --- +[INFO] Nothing to compile - all classes are up to date. +[INFO] +[INFO] --- resources:3.5.0:testResources (default-testResources) @ spring-kafka-exactly-once-boot4 --- +[INFO] Copying 1 resource from src\test\resources to target\test-classes +[INFO] +[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ spring-kafka-exactly-once-boot4 --- +[INFO] Recompiling the module because of changed source code. +[INFO] Compiling 8 source files with javac [debug parameters release 25] to target\test-classes +[INFO] +[INFO] --- surefire:3.5.6:test (default-test) @ spring-kafka-exactly-once-boot4 --- +[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider +[INFO] +[INFO] ------------------------------------------------------- +[INFO] T E S T S +[INFO] ------------------------------------------------------- +[INFO] Running com.ankurm.kafka.eos.SmokeTest +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-15096196392826061749\combined_0_0 with metadata.version 4.3-IV0. +22:22:09.287 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. +======================================================================== +Environment +======================================================================== +java.version : 25.0.3 +kafka-clients : 4.2.1 +embedded broker : EmbeddedKafkaKraftBroker +bootstrap servers : localhost:63356 +cluster id : JmSQ3q0IRuilOAxcUgBP6w +nodes : 1 +topics : [smoke] + +[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.729 s -- in com.ankurm.kafka.eos.SmokeTest +[INFO] Running com.ankurm.kafka.eos._01_idempotence.IdempotentProducerTest +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-9366343747489431922\combined_0_0 with metadata.version 4.3-IV0. +22:22:10.808 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. + +============================================================================== +1. Idempotence is on by default (Kafka 3.0+) -- verified from the broker +============================================================================== + +-- What the client library says the defaults are ----------------------------- + enable.idempotence : true + acks : all + +-- Sending 3 records with a completely default producer ---------------------- + +-- Asking the BROKER what producer state it is tracking for idem-default ----- + producerId=0 producerEpoch=0 lastSequence=2 + producer states tracked : 1 + +>> The broker holds a real producer ID and the last sequence number it accepted. + +>> That IS the de-duplication state, and you configured nothing to get it. + +============================================================================== +2. acks=1 silently disables idempotence -- no exception, no warning +============================================================================== + +-- Constructing a producer with ONLY acks=1 ---------------------------------- + props: acks=1 (enable.idempotence left at its default of true) + producer constructed successfully -- note that nothing complained + +-- Asking the broker for producer state on idem-acks1 ------------------------ + producer states tracked : 0 + +>> Three records were written, and the broker is tracking NOTHING for them. + +>> Kafka resolved acks=1 against enable.idempotence=true by turning + +>> idempotence off, and said nothing at all. Every record this producer + +>> sends can now be duplicated by an ordinary retry. + +============================================================================== +3. acks=1 + enable.idempotence=true is rejected at construction +============================================================================== + +-- Same acks=1, but this time idempotence is requested explicitly ------------ + ConfigException: Must set acks to all in order to use the idempotent producer. Otherwise we cannot guarantee idempotence. + +>> Ask for it explicitly and Kafka refuses. Leave it implicit and Kafka + +>> downgrades you. So setting enable.idempotence=true explicitly is worth + +>> doing purely as an assertion: it turns a silent downgrade into a + +>> startup failure, which is the failure you want. + +============================================================================== +4. The other constraints, and which ones fail loudly +============================================================================== + +-- max.in.flight.requests.per.connection = 6 --------------------------------- + rejected: To use the idempotent producer, max.in.flight.requests.per.connection must be set to at most 5. Current value is 6. + +-- retries = 0 --------------------------------------------------------------- + rejected: Must set retries to non-zero when using the idempotent producer. + +-- max.in.flight = 5 with idempotence -- allowed ----------------------------- + constructed successfully + +>> Idempotence requires acks=all, retries>0 and max.in.flight<=5. + +>> Five in flight is safe because the broker can reorder within that window + +>> using sequence numbers. WITHOUT idempotence, max.in.flight>1 silently + +>> breaks ORDERING on retry: a retried batch can land after a later one. +[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.723 s -- in com.ankurm.kafka.eos._01_idempotence.IdempotentProducerTest +[INFO] Running com.ankurm.kafka.eos._02_transactions.TransactionMarkerOffsetTest +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-5226944134061393373\combined_0_0 with metadata.version 4.3-IV0. +22:22:11.504 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. + +============================================================================== +1. A committed transaction burns one extra offset per partition +============================================================================== + +-- Before: empty topic ------------------------------------------------------- + endOffset = 0 + +-- Sending 5 records inside ONE transaction, then committing ----------------- + sent record 0 -> offset 0 + sent record 1 -> offset 1 + sent record 2 -> offset 2 + sent record 3 -> offset 3 + sent record 4 -> offset 4 + +-- After commit -------------------------------------------------------------- + records sent : 5 + endOffset : 6 <-- 5 records occupy 0..4, the commit marker takes 5 + +-- What a read_committed consumer actually receives -------------------------- +22:22:16.224 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator. +22:22:16.442 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator. +22:22:16.753 WARN o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-g-committed-1-4, groupId=g-committed-1] Offset commit failed on partition markers-demo-0 at offset 6: This is not the correct coordinator. + offset=0 value=committed-0 + offset=1 value=committed-1 + offset=2 value=committed-2 + offset=3 value=committed-3 + offset=4 value=committed-4 + records delivered : 5 + +>> 5 records were delivered, the last at offset 4, but the end offset is 6. + +>> The gap at offset 5 is the commit marker. It exists, it is durable, and + +>> no consumer will ever see it. + +============================================================================== +2. An aborted transaction still consumes offsets and disk +============================================================================== + endOffset before : 6 + +-- Sending 3 records inside a transaction, then ABORTING --------------------- + sent record 0 -> offset 6 + sent record 1 -> offset 7 + sent record 2 -> offset 8 + abortTransaction() called + +-- After abort --------------------------------------------------------------- + endOffset after : 10 (+4 for 3 aborted records and 1 abort marker) + +-- read_committed consumer --------------------------------------------------- + delivered : 5 records, values=[committed-0, committed-1, committed-2, committed-3, committed-4] + +-- read_uncommitted consumer -- the SAME partition --------------------------- + delivered : 8 records, values=[committed-0, committed-1, committed-2, committed-3, committed-4, aborted-0, aborted-1, aborted-2] + +>> The aborted records are physically in the log with permanent offsets. + +>> isolation.level is a CONSUMER-side filter, not a broker-side delete. + +>> A default consumer (read_uncommitted) reads data you deliberately aborted. + +============================================================================== +3. Lag is NOT inflated -- but counting records by offset arithmetic is broken +============================================================================== + +-- A read_committed consumer drains the partition, then commits -------------- + +-- The lag calculation everyone worries about -------------------------------- + partition end offset : 10 + consumer committed offset : 10 + lag (end - committed) : 0 + +>> Lag is ZERO. The consumer's position advances over markers and aborted + +>> records just as it advances over delivered ones, so the standard lag + +>> formula stays correct. The common claim that transactions inflate lag + +>> does not survive contact with a broker. + +-- The calculation that IS broken -------------------------------------------- + beginningOffset : 0 + endOffset : 10 + end - beginning : 10 <-- looks like the record count + records actually retrievable: 5 + offset of last record : 4 + overstatement : 100% + +>> Offset span says 10 records. Only 5 can ever be read -- a 100% error, + +>> from 2 markers and 3 aborted records. Anything that counts events by + +>> subtracting offsets -- retention math, throughput dashboards, tests that + +>> assert on offsets -- is wrong by however much your abort rate happens to be. +[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.20 s -- in com.ankurm.kafka.eos._02_transactions.TransactionMarkerOffsetTest +[INFO] Running com.ankurm.kafka.eos._03_read_process_write.ReadProcessWriteTest + + . ____ _ __ _ _ + /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ +( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ + \\/ ___)| |_)| | | | | || (_| | ) ) ) ) + ' |____| .__|_| |_|_| |_\__, | / / / / + =========|_|==============|___/=/_/_/_/ + + :: Spring Boot :: (v4.1.0) + +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-7564387337135394901\combined_0_0 with metadata.version 4.3-IV0. +22:22:27.088 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. +22:22:27.249 INFO c.a.k.e._.ReadProcessWriteTest - Starting ReadProcessWriteTest using Java 25.0.3 with PID 38092 (started by Ankur in C:\Users\Ankur\ankurm-blog-tools\projects\spring-kafka-exactly-once-boot4) +22:22:27.250 INFO c.a.k.e._.ReadProcessWriteTest - No active profile set, falling back to 1 default profile: "default" +22:22:30.619 INFO c.a.k.e._.ReadProcessWriteTest - Started ReadProcessWriteTest in 3.874 seconds (process running for 23.722) + +============================================================================== +1. Read-process-write, happy path +============================================================================== + +-- Publishing 3 orders to orders.incoming ------------------------------------ +22:22:30.756 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:30.855 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:30.957 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:31.059 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:31.162 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:31.263 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. + sent: alpha + sent: beta + sent: gamma +22:22:32.224 INFO c.a.k.e.o.OrderProcessor - processing 'alpha' (invocation #1) +22:22:32.351 INFO c.a.k.e.o.OrderProcessor - processing 'beta' (invocation #2) +22:22:32.462 INFO c.a.k.e.o.OrderProcessor - processing 'gamma' (invocation #3) + +-- Listener invocations ------------------------------------------------------ + processed: alpha + processed: beta + processed: gamma + +-- What a read_committed consumer sees on orders.validated ------------------- + offset=0 value=ALPHA + offset=2 value=BETA + offset=4 value=GAMMA + +>> 3 in, 3 out. The listener body contains no transaction code at all -- + +>> the guarantee comes entirely from spring.kafka.producer.transaction-id-prefix. + +============================================================================== +2. Failure AFTER the send: processed twice, delivered once +============================================================================== + +-- Poisoning 'delta' so its first processing attempt throws ------------------ + sent: delta + +-- Waiting for the listener to be invoked more than once --------------------- +22:22:35.108 INFO c.a.k.e.o.OrderProcessor - processing 'delta' (invocation #1) +22:22:35.111 ERROR o.s.k.s.LoggingProducerListener - Exception thrown when sending a message with key='null' and payload='DELTA' to topic orders.validated: +org.apache.kafka.common.errors.TransactionAbortedException: Failing batch since transaction was aborted +22:22:35.116 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back +org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos.orders.OrderProcessor.process(java.lang.String)' threw exception + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700) + at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) + at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137) + at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373) + at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825) + at java.base/java.lang.Thread.run(Thread.java:1474) + Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) +Caused by: java.lang.IllegalStateException: simulated processing failure for 'delta' + at com.ankurm.kafka.eos.orders.OrderProcessor.process(OrderProcessor.java:82) + at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) + at java.base/java.lang.reflect.Method.invoke(Method.java:565) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168) + at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) + at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) + ... 15 common frames omitted +22:22:35.613 INFO c.a.k.e.o.OrderProcessor - processing 'delta' (invocation #2) + times the listener ran for 'delta' : 2 + +-- What a read_committed consumer sees --------------------------------------- + DELTA records visible : 1 + +-- What a read_uncommitted consumer sees on the SAME topic ------------------- + DELTA records physically in the log : 1 + +>> The listener ran 2 times and a read_committed consumer sees exactly 1. + +>> + +>> This is the whole idea: exactly-once is a property of what is COMMITTED, + +>> not of how many times your code ran. Any side effect your listener performs + +>> outside Kafka -- a DB write, an HTTP call, an email -- happened 2 times. + +-- A detail worth noticing in the number above ------------------------------- + read_uncommitted sees 1 DELTA record(s), not 2 + +>> The rolled-back send never reached the broker at all. KafkaTemplate.send() + +>> is asynchronous: the record sat in the producer's accumulator, and + +>> abortTransaction() discarded the un-flushed batch instead of writing it. + +>> + +>> So aborted records reach the log only if they were already flushed -- which + +>> happens under load, on large batches, or when you block on the send future. + +>> TransactionMarkerOffsetTest forces exactly that case by calling get() on + +>> each send, and there the aborted records DO occupy permanent offsets. + +>> Do not rely on either behaviour: rely on read_committed. +[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.17 s -- in com.ankurm.kafka.eos._03_read_process_write.ReadProcessWriteTest +[INFO] Running com.ankurm.kafka.eos._04_fencing.ZombieFencingTest +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-2096764983482195835\combined_0_0 with metadata.version 4.3-IV0. +22:22:42.230 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. + +============================================================================== +1. A second producer with the same transactional.id fences the first +============================================================================== + +-- Producer A ('the zombie') starts a transaction and sends ------------------ + transactional.id = 'shared-tx-id' + record sent, transaction still OPEN + +-- Producer B starts with the SAME transactional.id -------------------------- + initTransactions() returned -- the coordinator has bumped the epoch + any in-flight transaction under the old epoch is now aborted + +-- Producer A tries to commit ------------------------------------------------ + REJECTED with ProducerFencedException + message: There is a newer producer with the same transactionalId which fences the current one. + +-- Producer B commits normally ----------------------------------------------- + committed successfully + +>> The zombie could not commit. Its work was discarded by the broker, + +>> not by any code you wrote. That is the entire value of transactional.id: + +>> a durable identity the coordinator can fence on. + +>> + +>> Note the exception type -- on Kafka 4.x with KIP-890 (transaction.version=2) + +>> the epoch is bumped every transaction, so you may see + +>> InvalidProducerEpochException where older articles promise + +>> ProducerFencedException. Catch both, or catch neither and let the + +>> container restart the producer, which is what Spring does. + +============================================================================== +2. A random transactional.id per start-up defeats fencing entirely +============================================================================== + +-- Producer A with a UUID-based transactional.id ----------------------------- + transactional.id = 'processor-9900b3b5-0861-4908-93a5-4d4b64c83ca7' + +-- Producer B restarts and generates a NEW UUID ------------------------------ + transactional.id = 'processor-1dd8cdf3-8ddf-4d3d-b28b-152284c558eb' + B committed + +-- Now the 'zombie' A commits -- does anything stop it? ---------------------- + A committed successfully. Nothing was fenced. + +>> Both producers committed. The coordinator saw two unrelated identities + +>> and had no reason to fence either one. + +>> + +>> This is the most damaging Kafka transactions misconfiguration, because + +>> everything appears to work: transactions commit, tests pass, and you have + +>> exactly the duplicate-processing problem you turned transactions on to + +>> prevent. The transactional.id must be STABLE across restarts and tied to + +>> the partition set an instance owns -- not to the process lifetime. +[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 s -- in com.ankurm.kafka.eos._04_fencing.ZombieFencingTest +[INFO] Running com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest + + . ____ _ __ _ _ + /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ +( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ + \\/ ___)| |_)| | | | | || (_| | ) ) ) ) + ' |____| .__|_| |_|_| |_\__, | / / / / + =========|_|==============|___/=/_/_/_/ + + :: Spring Boot :: (v4.1.0) + +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-17444936701909163355\combined_0_0 with metadata.version 4.3-IV0. +22:22:43.855 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. +22:22:44.000 INFO c.a.k.e._.DatabaseBoundaryTest - Starting DatabaseBoundaryTest using Java 25.0.3 with PID 38092 (started by Ankur in C:\Users\Ankur\ankurm-blog-tools\projects\spring-kafka-exactly-once-boot4) +22:22:44.000 INFO c.a.k.e._.DatabaseBoundaryTest - No active profile set, falling back to 1 default profile: "default" +22:22:44.452 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION} +22:22:44.454 INFO c.a.k.e._.DatabaseBoundaryTest - Started DatabaseBoundaryTest in 0.704 seconds (process running for 37.556) +22:22:44.574 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 5 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION} +22:22:44.575 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for orders.incoming with error 'TOPIC_ALREADY_EXISTS': Topic 'orders.incoming' already exists. + +============================================================================== +1. The problem: one row per ATTEMPT, not per message +============================================================================== + +-- Sending one message that will fail on its first processing attempt -------- +22:22:44.765 WARN o.a.k.clients.NetworkClient - [Consumer clientId=consumer-order-processor-15, groupId=order-processor] The metadata response from the cluster reported a recoverable issue with correlation id 15 : {orders.incoming=UNKNOWN_TOPIC_OR_PARTITION} +22:22:44.765 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for orders.incoming with error 'TOPIC_ALREADY_EXISTS': Topic 'orders.incoming' already exists. +22:22:44.797 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:44.900 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:45.002 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:45.104 WARN k.s.DefaultAutoTopicCreationManager - Auto topic creation failed for __transaction_state with error 'TOPIC_ALREADY_EXISTS': Topic '__transaction_state' already exists. +22:22:45.975 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$NaiveDbListener - naive listener wrote a row for 'order-1' +22:22:45.975 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back +org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$NaiveDbListener.onMessage(java.lang.String)' threw exception + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700) + at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) + at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137) + at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373) + at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825) + at java.base/java.lang.Thread.run(Thread.java:1474) + Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) +Caused by: java.lang.IllegalStateException: simulated failure after the DB write for order-1 + at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$Base.maybeFail(DatabaseBoundaryTest.java:209) + at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$NaiveDbListener.onMessage(DatabaseBoundaryTest.java:243) + at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) + at java.base/java.lang.reflect.Method.invoke(Method.java:565) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168) + at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) + at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) + ... 15 common frames omitted +22:22:46.438 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$NaiveDbListener - naive listener wrote a row for 'order-1' + listener attempts for 'order-1' : 2 + rows in the database : 2 + id=1 dedupeKey=naive-1 + id=2 dedupeKey=naive-2 + +>> The Kafka transaction rolled back correctly. The database row did not, + +>> because it was never in the Kafka transaction and never could have been. + +>> + +>> Note that nothing here is misconfigured. transaction-id-prefix is set, + +>> isolation.level is read_committed, the Kafka side is exactly-once. The + +>> database still has 2 rows for 1 message. + +============================================================================== +2. The fix: make the side effect idempotent, not the delivery +============================================================================== + +-- Same failure, but the listener dedupes on topic-partition-offset ---------- +22:22:48.940 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$IdempotentDbListener - idempotent listener wrote a row for 'order-2' as db.idempotent-0-0 +22:22:48.941 ERROR o.s.k.l.KafkaMessageListenerContainer - Transaction rolled back +org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$IdempotentDbListener.onMessage(java.lang.String,java.lang.String,int,long)' threw exception + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:3160) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3031) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2997) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2907) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$invokeInTransaction$0(KafkaMessageListenerContainer.java:2700) + at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) + at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137) + at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2696) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2667) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2640) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2274) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1572) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1504) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1373) + at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1825) + at java.base/java.lang.Thread.run(Thread.java:1474) + Suppressed: org.springframework.kafka.listener.ListenerExecutionFailedException: Restored Stack Trace + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:514) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) +Caused by: java.lang.IllegalStateException: simulated failure after the DB write for order-2 + at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$Base.maybeFail(DatabaseBoundaryTest.java:209) + at com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest$Listeners$IdempotentDbListener.onMessage(DatabaseBoundaryTest.java:275) + at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) + at java.base/java.lang.reflect.Method.invoke(Method.java:565) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:168) + at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:48) + at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) + at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:80) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:489) + at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:426) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) + at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:52) + at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:3018) + ... 15 common frames omitted +22:22:49.423 INFO c.a.k.e._.DatabaseBoundaryTest$Listeners$IdempotentDbListener - idempotent listener SKIPPED 'order-2' -- db.idempotent-0-0 already processed + listener attempts for 'order-2' : 2 + rows in the database : 1 + id=3 dedupeKey=db.idempotent-0-0 + +>> The listener ran 2 times and wrote 1 row. + +>> + +>> The dedupe key is 'topic-partition-offset', which Kafka guarantees is + +>> unique and stable. A UNIQUE constraint on that column turns the second + +>> attempt into a no-op -- and, importantly, would still do so if the + +>> duplicate came from a different process, a rebalance, or a replay. + +>> + +>> This is the whole design rule: exactly-once processing is achieved by + +>> idempotent side effects plus at-least-once delivery. Kafka transactions + +>> give you the second half. You have to build the first half yourself. +[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.701 s -- in com.ankurm.kafka.eos._05_database.DatabaseBoundaryTest +[INFO] Running com.ankurm.kafka.eos._06_performance.TransactionCostTest +Bootstrap metadata: BootstrapMetadata(records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', featureLevel=30) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='eligible.leader.replicas.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='group.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='share.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='streams.version', featureLevel=1) at version 0), ApiMessageAndVersion(FeatureLevelRecord(name='transaction.version', featureLevel=2) at version 0)], metadataVersionLevel=30, source=format command) +Formatting metadata directory C:\Users\Ankur\AppData\Local\Temp\kafka-8432095967619283703\combined_0_0 with metadata.version 4.3-IV0. +22:22:52.001 WARN o.a.k.c.QuorumController - [QuorumController id=0] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.3-IV0 from bootstrap source 'testkit'. + +============================================================================== +The cost of a transaction is paid per COMMIT, not per record +============================================================================== +Sending 10,000 records of 256 bytes in every configuration. + +-- warm-up (discarded) ------------------------------------------------------- + done + +-- 5 interleaved rounds of 10,000 records (round 0 discarded) ---------------- + round 0 (discarded): baseline=311,056 1/tx=349 10/tx=8,032 100/tx=66,664 1000/tx=344,474 10000/tx=641,264 + round 1: baseline=628,417 1/tx=765 10/tx=8,452 100/tx=61,437 1000/tx=434,245 10000/tx=836,617 + round 2: baseline=801,880 1/tx=685 10/tx=8,499 100/tx=78,770 1000/tx=216,065 10000/tx=878,071 + round 3: baseline=760,537 1/tx=718 10/tx=7,467 100/tx=104,725 1000/tx=235,578 10000/tx=942,649 + round 4: baseline=833,848 1/tx=711 10/tx=8,357 100/tx=71,036 1000/tx=471,918 10000/tx=897,892 + round 5: baseline=818,940 1/tx=692 10/tx=8,491 100/tx=97,746 1000/tx=330,969 10000/tx=657,960 + +-- measurement stability ----------------------------------------------------- + no transaction (idempotent) median=801,880 spread=+26% + 1 records/transaction median=711 spread=+11% + 10 records/transaction median=8,452 spread=+12% + 100 records/transaction median=78,770 spread=+55% + 1,000 records/transaction median=330,969 spread=+77% + 10,000 records/transaction median=878,071 spread=+32% + baseline first counted round : 628,417 records/s + baseline last counted round : 818,940 records/s + baseline drift : +30.3% + +-- summary (median of 5 rounds) ---------------------------------------------- + configuration records/s vs baseline + no transaction (idempotent) 801,880 100% + 1 records/transaction 711 0% + 10 records/transaction 8,452 1% + 100 records/transaction 78,770 10% + 1,000 records/transaction 330,969 41% + 10,000 records/transaction 878,071 110% + +>> Records per transaction, and what it buys: + +>> 1 -> 711 rec/s + +>> 10 -> 8,452 rec/s (11.9x) + +>> 100 -> 78,770 rec/s (9.3x) + +>> 1,000 -> 330,969 rec/s (4.2x) + +>> + +>> Each 10x increase in transaction size buys close to 10x throughput while + +>> commit cost dominates, then flattens as the per-record cost takes over. + +>> One record per transaction is roughly 111x slower than a hundred. + +>> + +>> (The non-transactional baseline is quoted at 801,880 rec/s but varied by + +>> 26% across rounds, so ratios against it are indicative only. The + +>> comparisons BETWEEN transaction sizes are the stable, meaningful ones.) + +>> + +>> The lesson is not 'transactions are slow'. It is that a commit is a fixed + +>> cost, so the number of records you put inside one is the setting that + +>> matters. In Spring, that number is the size of the batch returned by + +>> poll(), which you control with max.poll.records -- NOT with anything + +>> named 'transaction'. Raising max.poll.records is the single most + +>> effective throughput fix for a transactional listener, at the price of + +>> redoing more work when one record in the batch fails. +[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 112.4 s -- in com.ankurm.kafka.eos._06_performance.TransactionCostTest +[INFO] +[INFO] Results: +[INFO] +[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0 +[INFO] +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 02:46 min +[INFO] Finished at: 2026-07-31T22:24:48+05:30 +[INFO] ------------------------------------------------------------------------ diff --git a/src/main/java/com/ankurm/kafka/eos/Application.java b/src/main/java/com/ankurm/kafka/eos/Application.java new file mode 100644 index 0000000..cf86a0c --- /dev/null +++ b/src/main/java/com/ankurm/kafka/eos/Application.java @@ -0,0 +1,20 @@ +package com.ankurm.kafka.eos; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +/** + * Companion application for + * Exactly-Once with Spring Kafka on Spring Boot 4. + * + *

The interesting code is in {@code orders/} and the demonstrations are in {@code src/test}. + * Run {@code mvn test} -- every test starts its own in-JVM KRaft broker, so there is nothing to + * install and nothing to start. + */ +@SpringBootApplication +public class Application { + + public static void main(String[] args) { + SpringApplication.run(Application.class, args); + } +} diff --git a/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrder.java b/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrder.java new file mode 100644 index 0000000..1319689 --- /dev/null +++ b/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrder.java @@ -0,0 +1,66 @@ +package com.ankurm.kafka.eos.db; + +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.GeneratedValue; +import jakarta.persistence.GenerationType; +import jakarta.persistence.Id; +import jakarta.persistence.Table; + +import java.time.Instant; + +/** + * A row written by a Kafka listener -- the side effect that Kafka transactions cannot protect. + * + *

Note {@link #dedupeKey}: it is the natural identity of the work, not a surrogate. Making that + * column {@code UNIQUE} is what turns a non-idempotent side effect into an idempotent one, which is + * the only reliable way to survive the redelivery that exactly-once processing guarantees you will + * eventually get. See {@code docs/05-database-boundary.md}. + */ +@Entity +@Table(name = "processed_order") +public class ProcessedOrder { + + @Id + @GeneratedValue(strategy = GenerationType.IDENTITY) + private Long id; + + /** + * Unique business identity of this unit of work. + * + *

In a real system this is usually either an event ID carried in the message, or the + * {@code topic-partition-offset} triple, which Kafka guarantees is unique and stable. + */ + @Column(name = "dedupe_key", nullable = false, unique = true) + private String dedupeKey; + + @Column(name = "payload", nullable = false) + private String payload; + + @Column(name = "written_at", nullable = false) + private Instant writtenAt = Instant.now(); + + protected ProcessedOrder() { + } + + public ProcessedOrder(String dedupeKey, String payload) { + this.dedupeKey = dedupeKey; + this.payload = payload; + } + + public Long getId() { + return id; + } + + public String getDedupeKey() { + return dedupeKey; + } + + public String getPayload() { + return payload; + } + + public Instant getWrittenAt() { + return writtenAt; + } +} diff --git a/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrderRepository.java b/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrderRepository.java new file mode 100644 index 0000000..dfa68e3 --- /dev/null +++ b/src/main/java/com/ankurm/kafka/eos/db/ProcessedOrderRepository.java @@ -0,0 +1,12 @@ +package com.ankurm.kafka.eos.db; + +import org.springframework.data.jpa.repository.JpaRepository; + +import java.util.List; + +public interface ProcessedOrderRepository extends JpaRepository { + + boolean existsByDedupeKey(String dedupeKey); + + List findByPayload(String payload); +} diff --git a/src/main/java/com/ankurm/kafka/eos/orders/OrderProcessor.java b/src/main/java/com/ankurm/kafka/eos/orders/OrderProcessor.java new file mode 100644 index 0000000..2d6ccb5 --- /dev/null +++ b/src/main/java/com/ankurm/kafka/eos/orders/OrderProcessor.java @@ -0,0 +1,99 @@ +package com.ankurm.kafka.eos.orders; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.kafka.annotation.KafkaListener; +import org.springframework.kafka.core.KafkaTemplate; +import org.springframework.stereotype.Component; + +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; + +/** + * The canonical read-process-write loop, which is the only thing Kafka's exactly-once + * semantics actually guarantee. + * + *

What makes this exactly-once

+ * Three things have to happen atomically for this to be correct: + * + *
    + *
  1. the output record is written to {@code orders.validated},
    2. + *
  2. the consumer offset for {@code orders.incoming} is advanced,
    3. + *
  3. both are visible to downstream {@code read_committed} consumers, or neither is.
    4. + *
+ * + *

Point 2 is the one people miss. The offset commit is not a separate step that happens after + * the send -- it is written into the same Kafka transaction, via + * {@code sendOffsetsToTransaction}. That is why a crash between "produced output" and "committed + * offset" cannot produce a duplicate: there is no such between. + * + *

You do not write any of that. Setting + * {@code spring.kafka.producer.transaction-id-prefix} makes Spring Boot auto-configure a + * {@code KafkaTransactionManager} and attach it to the listener container. The container then + * begins a transaction before invoking this method, sends the offsets into it, and commits -- or + * aborts the whole thing if this method throws. The method body below is deliberately ordinary; the + * guarantee is entirely in the configuration. + * + *

What it does NOT guarantee

+ * Nothing outside Kafka. If this method also wrote to a database, called a payment API, or sent an + * email, those are not in the transaction and will be repeated on redelivery. See + * {@code docs/05-database-boundary.md}. + */ +@Component +public class OrderProcessor { + + private static final Logger log = LoggerFactory.getLogger(OrderProcessor.class); + + public static final String INPUT_TOPIC = "orders.incoming"; + public static final String OUTPUT_TOPIC = "orders.validated"; + + private final KafkaTemplate kafkaTemplate; + + /** + * Every invocation is recorded, including the ones that were rolled back. Comparing this list + * against what a {@code read_committed} consumer can see on the output topic is how the tests + * show the difference between "processed" and "committed". + */ + private final List invocations = new CopyOnWriteArrayList<>(); + + /** Values that should fail on their first N attempts, to exercise rollback and redelivery. */ + private final List poisoned = new CopyOnWriteArrayList<>(); + + public OrderProcessor(KafkaTemplate kafkaTemplate) { + this.kafkaTemplate = kafkaTemplate; + } + + @KafkaListener(topics = INPUT_TOPIC, groupId = "order-processor") + public void process(String order) { + invocations.add(order); + log.info("processing '{}' (invocation #{})", order, invocations.size()); + + // This send joins the transaction the container already started. There is no + // "beginTransaction" here and there must not be -- KafkaTemplate detects the active + // transaction and enlists in it. + kafkaTemplate.send(OUTPUT_TOPIC, order.toUpperCase()); + + if (poisoned.contains(order)) { + poisoned.remove(order); + // Throwing after the send is the interesting case: the output record has already been + // handed to the producer. Because the transaction aborts, no read_committed consumer + // will ever see it -- but it IS in the log, with an offset. See + // TransactionMarkerOffsetTest. + throw new IllegalStateException("simulated processing failure for '" + order + "'"); + } + } + + /** Marks a value so that its next processing attempt throws. */ + public void poison(String value) { + poisoned.add(value); + } + + public List invocations() { + return List.copyOf(invocations); + } + + public void reset() { + invocations.clear(); + poisoned.clear(); + } +} diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml new file mode 100644 index 0000000..0f01cdc --- /dev/null +++ b/src/main/resources/application.yml @@ -0,0 +1,83 @@ +# +# Exactly-once configuration for Spring Boot 4.1 / Spring Kafka 4.1. +# +# The striking thing about this file is how little of it there is. Setting +# `transaction-id-prefix` is the whole switch: Spring Boot sees it, auto-configures a +# KafkaTransactionManager, and wires it into every listener container. Offsets then travel +# inside the Kafka transaction instead of being committed separately. +# +spring: + application: + name: spring-kafka-exactly-once + + kafka: + # Overridden by EmbeddedKafkaBroker during tests via spring.embedded.kafka.brokers. + bootstrap-servers: ${spring.embedded.kafka.brokers:localhost:9092} + + producer: + key-serializer: org.apache.kafka.common.serialization.StringSerializer + value-serializer: org.apache.kafka.common.serialization.StringSerializer + + # --------------------------------------------------------------------------------- + # THE line. Everything else on this page is either a default worth stating out loud + # or a consequence of this. + # + # Each application instance needs a UNIQUE transactional.id, because that identity is + # what the broker fences on. Spring appends a suffix per producer; see + # TransactionIdSuffixStrategy (Spring Kafka 3.2+) if you need to control it, and + # docs/04-fencing.md for what goes wrong when two instances share one. + # --------------------------------------------------------------------------------- + transaction-id-prefix: order-processor-tx- + + # Defaults since Kafka 3.0, restated so that nobody "optimises" acks to 1 later and + # silently drops to at-least-once. See _01_idempotence/IdempotentProducerTest. + acks: all + properties: + enable.idempotence: true + max.in.flight.requests.per.connection: 5 + + # A transaction left open blocks read_committed consumers on that partition until it + # times out, so this is a consumer-latency setting as much as a producer one. + # It must be larger than the time one poll batch takes to process, and the broker's + # transaction.max.timeout.ms caps it. + transaction.timeout.ms: 30000 + + consumer: + key-deserializer: org.apache.kafka.common.serialization.StringDeserializer + value-deserializer: org.apache.kafka.common.serialization.StringDeserializer + auto-offset-reset: earliest + + # --------------------------------------------------------------------------------- + # The default is read_uncommitted, which means a consumer reading a transactional + # topic will happily deliver records from transactions that were ABORTED. Producing + # transactionally without setting this on the consumer side gives you the cost of + # transactions and none of the benefit. + # --------------------------------------------------------------------------------- + isolation-level: read_committed + + # With a transaction manager in play the container manages offsets inside the + # transaction; auto-commit must be off (Spring Boot defaults it off, stated for clarity). + enable-auto-commit: false + + listener: + # RECORD/BATCH ack modes are irrelevant when a KafkaTransactionManager is present -- the + # transaction boundary is the batch returned by poll(). Left explicit so the interaction + # is visible rather than surprising. + ack-mode: batch + # Fail fast in tests rather than blocking forever if a topic is missing. + missing-topics-fatal: false + + # Only used by the module that shows why "Kafka and the database, atomically" is not a thing. + datasource: + url: jdbc:h2:mem:eos;DB_CLOSE_DELAY=-1 + driver-class-name: org.h2.Driver + username: sa + password: "" + jpa: + hibernate: + ddl-auto: create-drop + open-in-view: false + +logging: + level: + com.ankurm.kafka: INFO diff --git a/src/test/java/com/ankurm/kafka/eos/SmokeTest.java b/src/test/java/com/ankurm/kafka/eos/SmokeTest.java new file mode 100644 index 0000000..a7bd3fe --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/SmokeTest.java @@ -0,0 +1,51 @@ +package com.ankurm.kafka.eos; + +import org.apache.kafka.clients.admin.Admin; +import org.apache.kafka.common.utils.AppInfoParser; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Proves the toolchain works before anything interesting is attempted: an in-JVM KRaft broker + * starts, an admin client connects, and the versions are what we think they are. + * + *

If this test fails, nothing else in the repository will work, and the reason is almost always + * one of: a JDK older than 17, a missing {@code spring-kafka-test} dependency, or a corporate + * proxy that stopped Maven resolving the broker jar. + */ +@SpringJUnitConfig +@EmbeddedKafka(partitions = 1, topics = {"smoke"}) +class SmokeTest { + + @Autowired + EmbeddedKafkaBroker broker; + + @Test + void embeddedKraftBrokerStarts() throws Exception { + System.out.println("=".repeat(72)); + System.out.println("Environment"); + System.out.println("=".repeat(72)); + System.out.printf("java.version : %s%n", System.getProperty("java.version")); + System.out.printf("kafka-clients : %s%n", AppInfoParser.getVersion()); + System.out.printf("embedded broker : %s%n", broker.getClass().getSimpleName()); + System.out.printf("bootstrap servers : %s%n", broker.getBrokersAsString()); + + try (Admin admin = Admin.create(Map.of("bootstrap.servers", broker.getBrokersAsString()))) { + var clusterId = admin.describeCluster().clusterId().get(); + var nodes = admin.describeCluster().nodes().get(); + System.out.printf("cluster id : %s%n", clusterId); + System.out.printf("nodes : %d%n", nodes.size()); + System.out.printf("topics : %s%n", admin.listTopics().names().get()); + + assertThat(nodes).isNotEmpty(); + } + System.out.println(); + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java b/src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java new file mode 100644 index 0000000..e604577 --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_01_idempotence/IdempotentProducerTest.java @@ -0,0 +1,243 @@ +package com.ankurm.kafka.eos._01_idempotence; + +import com.ankurm.kafka.eos.support.Report; +import org.apache.kafka.clients.admin.Admin; +import org.apache.kafka.clients.admin.ProducerState; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.config.ConfigException; +import org.apache.kafka.common.serialization.StringSerializer; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + *

Idempotence: the half of exactly-once you get for free, and the config that silently + * takes it away

+ * + *

An idempotent producer solves one specific problem: a producer sends a record, the + * broker writes it, and the acknowledgement is lost on the way back. The producer retries. Without + * idempotence the broker now holds the record twice, and no amount of consumer-side care can undo + * that -- the duplicate is a real, distinct record with its own offset. + * + *

The fix is a producer ID (PID) plus a per-partition sequence number. The broker remembers + * recent sequence numbers per PID per partition and rejects a repeat. This costs nothing measurable + * and has been on by default since Kafka 3.0. + * + *

What this test actually proves

+ * Most articles assert that idempotence is enabled and move on. This test verifies it from the + * broker's side using {@link Admin#describeProducers}, which reports the idempotence + * bookkeeping the broker holds for a partition: producer ID, epoch, last sequence number. A + * non-idempotent producer creates no such state, because there is nothing to de-duplicate against. + * That is the difference between reading a config value back and observing a behaviour. + * + *

Each scenario uses its own topic. Broker-side producer state is per partition and + * outlives the producer that created it, so sharing one topic across these scenarios would leak + * state from the first into the second and produce a very convincing wrong answer. An earlier + * version of this file did exactly that. + * + *

The trap

+ * {@code acks} and {@code enable.idempotence} interact, and they interact differently depending on + * whether you set idempotence explicitly: + * + *
    + *
  • Set only {@code acks=1} -- idempotence is silently disabled. No warning that + * stops a deployment, no exception. You simply lose the guarantee.
    • + *
  • Set {@code acks=1} and {@code enable.idempotence=true} -- you get a + * {@link ConfigException} at construction. Kafka refuses the impossible combination only when + * you asked for it explicitly.
    • + *
+ * + *

The first case is the dangerous one, and it is common: someone sets {@code acks=1} for latency, + * in a properties file, years ago, and the system has been quietly at-least-once ever since. + */ +@SpringJUnitConfig +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +@EmbeddedKafka(partitions = 1, topics = { + IdempotentProducerTest.TOPIC_DEFAULT, + IdempotentProducerTest.TOPIC_ACKS1, + IdempotentProducerTest.TOPIC_EXPLICIT +}) +class IdempotentProducerTest { + + static final String TOPIC_DEFAULT = "idem-default"; + static final String TOPIC_ACKS1 = "idem-acks1"; + static final String TOPIC_EXPLICIT = "idem-explicit"; + + @Autowired + EmbeddedKafkaBroker broker; + + // --------------------------------------------------------------------------------------- + // 1. The default + // --------------------------------------------------------------------------------------- + + @Test + @Order(1) + void idempotenceIsOnByDefaultAndTheBrokerCanSeeIt() throws Exception { + Report.title("1. Idempotence is on by default (Kafka 3.0+) -- verified from the broker"); + + Report.section("What the client library says the defaults are"); + Report.bullet("enable.idempotence : %s", + ProducerConfig.configDef().defaultValues().get(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG)); + Report.bullet("acks : %s", + ProducerConfig.configDef().defaultValues().get(ProducerConfig.ACKS_CONFIG)); + + Report.section("Sending 3 records with a completely default producer"); + try (Producer producer = producer(Map.of())) { + for (int i = 0; i < 3; i++) { + producer.send(new ProducerRecord<>(TOPIC_DEFAULT, "key", "default-" + i)).get(); + } + } + + Report.section("Asking the BROKER what producer state it is tracking for " + TOPIC_DEFAULT); + List states = describeProducers(TOPIC_DEFAULT); + states.forEach(s -> Report.bullet("producerId=%d producerEpoch=%d lastSequence=%d", + s.producerId(), s.producerEpoch(), s.lastSequence())); + Report.bullet("producer states tracked : %d", states.size()); + + assertThat(states) + .as("an idempotent producer registers a producer ID with the broker") + .isNotEmpty(); + assertThat(states.getFirst().producerId()).isNotEqualTo(-1L); + assertThat(states.getFirst().lastSequence()) + .as("three records were sent, so sequence numbers 0,1,2 were used") + .isEqualTo(2); + + Report.takeaway("The broker holds a real producer ID and the last sequence number it accepted."); + Report.takeaway("That IS the de-duplication state, and you configured nothing to get it."); + } + + // --------------------------------------------------------------------------------------- + // 2. The silent downgrade + // --------------------------------------------------------------------------------------- + + @Test + @Order(2) + void acksOneSilentlyDisablesIdempotence() throws Exception { + Report.title("2. acks=1 silently disables idempotence -- no exception, no warning"); + + Report.section("Constructing a producer with ONLY acks=1"); + Report.bullet("props: acks=1 (enable.idempotence left at its default of true)"); + + try (Producer producer = producer(Map.of(ProducerConfig.ACKS_CONFIG, "1"))) { + Report.bullet("producer constructed successfully -- note that nothing complained"); + for (int i = 0; i < 3; i++) { + producer.send(new ProducerRecord<>(TOPIC_ACKS1, "key", "acks1-" + i)).get(); + } + } + + Report.section("Asking the broker for producer state on " + TOPIC_ACKS1); + List states = describeProducers(TOPIC_ACKS1); + Report.bullet("producer states tracked : %d", states.size()); + states.forEach(s -> Report.bullet(" producerId=%d epoch=%d lastSequence=%d", + s.producerId(), s.producerEpoch(), s.lastSequence())); + + assertThat(states) + .as("a non-idempotent producer leaves no de-duplication state on the broker") + .isEmpty(); + + Report.takeaway("Three records were written, and the broker is tracking NOTHING for them."); + Report.takeaway("Kafka resolved acks=1 against enable.idempotence=true by turning"); + Report.takeaway("idempotence off, and said nothing at all. Every record this producer"); + Report.takeaway("sends can now be duplicated by an ordinary retry."); + } + + // --------------------------------------------------------------------------------------- + // 3. The loud version of the same mistake + // --------------------------------------------------------------------------------------- + + @Test + @Order(3) + void acksOnePlusExplicitIdempotenceIsRejected() { + Report.title("3. acks=1 + enable.idempotence=true is rejected at construction"); + + Report.section("Same acks=1, but this time idempotence is requested explicitly"); + + assertThatThrownBy(() -> producer(Map.of( + ProducerConfig.ACKS_CONFIG, "1", + ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"))) + .isInstanceOf(ConfigException.class) + .satisfies(e -> Report.bullet("ConfigException: %s", e.getMessage())); + + Report.takeaway("Ask for it explicitly and Kafka refuses. Leave it implicit and Kafka"); + Report.takeaway("downgrades you. So setting enable.idempotence=true explicitly is worth"); + Report.takeaway("doing purely as an assertion: it turns a silent downgrade into a"); + Report.takeaway("startup failure, which is the failure you want."); + } + + // --------------------------------------------------------------------------------------- + // 4. The other constraints + // --------------------------------------------------------------------------------------- + + @Test + @Order(4) + void theOtherConstraintsIdempotenceImposes() { + Report.title("4. The other constraints, and which ones fail loudly"); + + Report.section("max.in.flight.requests.per.connection = 6"); + assertThatThrownBy(() -> producer(Map.of( + ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true", + ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "6"))) + .isInstanceOf(ConfigException.class) + .satisfies(e -> Report.bullet("rejected: %s", e.getMessage())); + + Report.section("retries = 0"); + assertThatThrownBy(() -> producer(Map.of( + ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true", + ProducerConfig.RETRIES_CONFIG, "0"))) + .isInstanceOf(ConfigException.class) + .satisfies(e -> Report.bullet("rejected: %s", e.getMessage())); + + Report.section("max.in.flight = 5 with idempotence -- allowed"); + try (Producer p = producer(Map.of( + ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true", + ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"))) { + Report.bullet("constructed successfully"); + } + + Report.takeaway("Idempotence requires acks=all, retries>0 and max.in.flight<=5."); + Report.takeaway("Five in flight is safe because the broker can reorder within that window"); + Report.takeaway("using sequence numbers. WITHOUT idempotence, max.in.flight>1 silently"); + Report.takeaway("breaks ORDERING on retry: a retried batch can land after a later one."); + } + + // --------------------------------------------------------------------------------------- + // helpers + // --------------------------------------------------------------------------------------- + + private Producer producer(Map overrides) { + Map props = new HashMap<>(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.putAll(overrides); + return new KafkaProducer<>(props); + } + + /** + * Asks the broker which producers it is currently tracking idempotence state for. + * + *

This is the observation that makes the test meaningful rather than tautological. + */ + private List describeProducers(String topic) throws Exception { + try (Admin admin = Admin.create(Map.of("bootstrap.servers", broker.getBrokersAsString()))) { + TopicPartition tp = new TopicPartition(topic, 0); + return admin.describeProducers(List.of(tp)).partitionResult(tp).get().activeProducers(); + } + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java b/src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java new file mode 100644 index 0000000..9cd89f1 --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java @@ -0,0 +1,323 @@ +package com.ankurm.kafka.eos._02_transactions; + +import com.ankurm.kafka.eos.support.Report; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + *

Transaction markers: why your offsets have holes and your lag metric lies

+ * + *

This is the most operationally surprising consequence of turning on transactions, and it is + * barely mentioned anywhere. + * + *

When a transaction commits or aborts, Kafka writes a control record -- a transaction + * marker -- into every partition the transaction touched. Control records are real entries in the + * log and they consume an offset, but they are never delivered to your consumer. Two things + * follow immediately: + * + *

    + *
  1. Offsets are not contiguous. Send five records in a transaction and the next record + * you send will be at offset 6, not 5. Any code that assumes {@code offset + 1} is the next + * record, or that counts records by subtracting offsets, is now wrong.
    2. + *
  2. Consumer lag is overstated. The usual lag formula is + * {@code endOffset - committedOffset}. With transactions, {@code endOffset} counts markers + * that will never be delivered, so a perfectly caught-up consumer reports non-zero lag -- + * permanently, and proportionally to how many transactions you commit.
    3. + *
+ * + *

A third consequence appears when a transaction aborts: the records were already written + * to the log and they keep their offsets forever. A {@code read_committed} consumer filters them + * out; a {@code read_uncommitted} consumer sees them. The disk space and the offsets are consumed + * either way. + * + *

There is also a latency consequence that catches people out. A {@code read_committed} + * consumer can only read up to the Last Stable Offset (LSO) -- the offset before the first + * still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every + * {@code read_committed} consumer on that partition stalls for 30 seconds, even for records + * committed by other producers afterwards. Transaction duration is therefore a consumer-latency + * knob, not just a producer concern. + */ +@SpringJUnitConfig +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +@EmbeddedKafka(partitions = 1, topics = {TransactionMarkerOffsetTest.TOPIC}) +class TransactionMarkerOffsetTest { + + static final String TOPIC = "markers-demo"; + private static final TopicPartition PARTITION = new TopicPartition(TOPIC, 0); + + @Autowired + EmbeddedKafkaBroker broker; + + @Test + @Order(1) + void committedTransactionConsumesAnExtraOffsetForItsMarker() { + Report.title("1. A committed transaction burns one extra offset per partition"); + + Report.section("Before: empty topic"); + Report.bullet("endOffset = %d", endOffset()); + + Report.section("Sending 5 records inside ONE transaction, then committing"); + try (Producer producer = transactionalProducer("tx-markers-1")) { + producer.initTransactions(); + producer.beginTransaction(); + for (int i = 0; i < 5; i++) { + var meta = producer.send(new ProducerRecord<>(TOPIC, "k", "committed-" + i)).get(); + Report.bullet("sent record %d -> offset %d", i, meta.offset()); + } + producer.commitTransaction(); + } catch (Exception e) { + throw new IllegalStateException(e); + } + + long end = awaitEndOffset(6); + Report.section("After commit"); + Report.bullet("records sent : 5"); + Report.bullet("endOffset : %d <-- 5 records occupy 0..4, the commit marker takes 5", end); + + assertThat(end).isEqualTo(6); + + Report.section("What a read_committed consumer actually receives"); + List> received = drain("read_committed", "g-committed-1"); + received.forEach(r -> Report.bullet("offset=%d value=%s", r.offset(), r.value())); + Report.bullet("records delivered : %d", received.size()); + + assertThat(received).hasSize(5); + assertThat(received.getLast().offset()).isEqualTo(4); + + Report.takeaway("5 records were delivered, the last at offset 4, but the end offset is 6."); + Report.takeaway("The gap at offset 5 is the commit marker. It exists, it is durable, and"); + Report.takeaway("no consumer will ever see it."); + } + + @Test + @Order(2) + void abortedRecordsKeepTheirOffsetsForever() { + Report.title("2. An aborted transaction still consumes offsets and disk"); + + long before = endOffset(); + Report.bullet("endOffset before : %d", before); + + Report.section("Sending 3 records inside a transaction, then ABORTING"); + try (Producer producer = transactionalProducer("tx-markers-2")) { + producer.initTransactions(); + producer.beginTransaction(); + for (int i = 0; i < 3; i++) { + var meta = producer.send(new ProducerRecord<>(TOPIC, "k", "aborted-" + i)).get(); + Report.bullet("sent record %d -> offset %d", i, meta.offset()); + } + producer.abortTransaction(); + Report.bullet("abortTransaction() called"); + } catch (Exception e) { + throw new IllegalStateException(e); + } + + long after = awaitEndOffset(before + 4); + Report.section("After abort"); + Report.bullet("endOffset after : %d (+%d for 3 aborted records and 1 abort marker)", + after, after - before); + + assertThat(after).isEqualTo(before + 4); + + Report.section("read_committed consumer"); + List> committed = drain("read_committed", "g-committed-2"); + Report.bullet("delivered : %d records, values=%s", + committed.size(), committed.stream().map(ConsumerRecord::value).toList()); + + Report.section("read_uncommitted consumer -- the SAME partition"); + List> uncommitted = drain("read_uncommitted", "g-uncommitted-2"); + Report.bullet("delivered : %d records, values=%s", + uncommitted.size(), uncommitted.stream().map(ConsumerRecord::value).toList()); + + assertThat(committed).hasSize(5); // only the 5 from test 1 + assertThat(uncommitted).hasSize(8); // those 5 plus the 3 aborted ones + + Report.takeaway("The aborted records are physically in the log with permanent offsets."); + Report.takeaway("isolation.level is a CONSUMER-side filter, not a broker-side delete."); + Report.takeaway("A default consumer (read_uncommitted) reads data you deliberately aborted."); + } + + /** + * A test that exists because it refuted the claim it was written to demonstrate. + * + *

The intended demonstration was "transaction markers inflate consumer lag". That is a + * widely repeated claim and it is wrong, at least for the standard + * {@code endOffset - committedOffset} formula: when a {@code read_committed} consumer drains a + * partition, its position advances past the markers too, so the committed offset + * reaches the end offset and lag is exactly zero. + * + *

What markers really break is offset arithmetic used to count records, which is a + * different and much easier mistake to make: retention estimates, "how many events happened + * between these two offsets", partition-size dashboards, and any test that asserts on offsets. + */ + @Test + @Order(3) + void lagIsFineButOffsetArithmeticIsNot() { + Report.title("3. Lag is NOT inflated -- but counting records by offset arithmetic is broken"); + + Report.section("A read_committed consumer drains the partition, then commits"); + Map props = consumerProps("read_committed", "g-lag"); + props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); + + long end; + long beginning; + long committedOffset; + long lastDeliveredOffset; + int delivered; + try (KafkaConsumer consumer = new KafkaConsumer<>(props)) { + consumer.assign(List.of(PARTITION)); + consumer.seekToBeginning(List.of(PARTITION)); + + List> all = new ArrayList<>(); + for (int i = 0; i < 8; i++) { + consumer.poll(Duration.ofMillis(400)).forEach(all::add); + } + delivered = all.size(); + lastDeliveredOffset = all.isEmpty() ? -1 : all.getLast().offset(); + consumer.commitSync(); + + beginning = consumer.beginningOffsets(List.of(PARTITION)).get(PARTITION); + end = consumer.endOffsets(List.of(PARTITION)).get(PARTITION); + committedOffset = consumer.committed(java.util.Set.of(PARTITION)).get(PARTITION).offset(); + } + + Report.section("The lag calculation everyone worries about"); + Report.bullet("partition end offset : %d", end); + Report.bullet("consumer committed offset : %d", committedOffset); + Report.bullet("lag (end - committed) : %d", end - committedOffset); + + assertThat(end - committedOffset) + .as("a drained read_committed consumer really is at zero lag") + .isZero(); + + Report.takeaway("Lag is ZERO. The consumer's position advances over markers and aborted"); + Report.takeaway("records just as it advances over delivered ones, so the standard lag"); + Report.takeaway("formula stays correct. The common claim that transactions inflate lag"); + Report.takeaway("does not survive contact with a broker."); + + Report.section("The calculation that IS broken"); + Report.bullet("beginningOffset : %d", beginning); + Report.bullet("endOffset : %d", end); + Report.bullet("end - beginning : %d <-- looks like the record count", end - beginning); + Report.bullet("records actually retrievable: %d", delivered); + Report.bullet("offset of last record : %d", lastDeliveredOffset); + Report.bullet("overstatement : %.0f%%", + 100.0 * (end - beginning - delivered) / delivered); + + assertThat(end - beginning) + .as("offset span is not the record count once transactions are involved") + .isNotEqualTo(delivered); + + Report.takeaway("Offset span says %d records. Only %d can ever be read -- a %.0f%% error,", + end - beginning, delivered, 100.0 * (end - beginning - delivered) / delivered); + Report.takeaway("from 2 markers and 3 aborted records. Anything that counts events by"); + Report.takeaway("subtracting offsets -- retention math, throughput dashboards, tests that"); + Report.takeaway("assert on offsets -- is wrong by however much your abort rate happens to be."); + } + + // --------------------------------------------------------------------------------------- + // helpers + // --------------------------------------------------------------------------------------- + + private Producer transactionalProducer(String transactionalId) { + Map props = new HashMap<>(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + // Setting a transactional.id is what turns an idempotent producer into a transactional one. + // It also forces enable.idempotence=true -- transactions are built on top of idempotence. + props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId); + return new KafkaProducer<>(props); + } + + private Map consumerProps(String isolationLevel, String groupId) { + Map props = new HashMap<>(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); + props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); + // The default is read_uncommitted. Consuming the output of a transactional producer + // without changing this gives you aborted records -- see test 2. + props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, isolationLevel); + return props; + } + + private List> drain(String isolationLevel, String groupId) { + List> out = new ArrayList<>(); + try (KafkaConsumer consumer = new KafkaConsumer<>(consumerProps(isolationLevel, groupId))) { + consumer.assign(List.of(PARTITION)); + consumer.seekToBeginning(List.of(PARTITION)); + for (int i = 0; i < 8; i++) { + ConsumerRecords polled = consumer.poll(Duration.ofMillis(400)); + polled.forEach(out::add); + } + } + return out; + } + + /** + * Waits (briefly) for the partition's end offset to reach an expected value. + * + *

Needed because {@code commitTransaction()} returning does not mean the transaction + * marker is already visible in the partition's high watermark. The producer's commit is + * acknowledged by the transaction coordinator; the coordinator then writes markers into each + * participating partition, and the leader's high watermark advances after that. Reading + * {@code endOffsets()} immediately after the commit therefore races, and sometimes returns the + * offset without the marker. + * + *

This test originally asserted on the immediate read and passed consistently when run + * alone -- then failed in the full suite, where the machine was busier and the race opened up. + * That is the classic shape of a flaky integration test, and the fix is to wait for the + * condition rather than to assume it. + */ + private long awaitEndOffset(long expected) { + long deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(10); + long last = endOffset(); + while (last != expected && System.nanoTime() < deadline) { + try { + Thread.sleep(100); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + break; + } + last = endOffset(); + } + return last; + } + + private long endOffset() { + try (KafkaConsumer consumer = + new KafkaConsumer<>(consumerProps("read_uncommitted", "g-endoffset-" + System.nanoTime()))) { + // Assign before asking: endOffsets() on an unassigned consumer works, but logs a + // "Not updating high watermark ... no longer assigned" warning that clutters the + // transcript this test exists to produce. + consumer.assign(List.of(PARTITION)); + return consumer.endOffsets(List.of(PARTITION)).get(PARTITION); + } + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java b/src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java new file mode 100644 index 0000000..57864c5 --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java @@ -0,0 +1,194 @@ +package com.ankurm.kafka.eos._03_read_process_write; + +import com.ankurm.kafka.eos.Application; +import com.ankurm.kafka.eos.orders.OrderProcessor; +import com.ankurm.kafka.eos.support.Report; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.kafka.core.KafkaTemplate; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.awaitility.Awaitility.await; + +/** + *

Read-process-write: the only thing Kafka EOS actually guarantees

+ * + *

These tests exercise {@link OrderProcessor}, which is a completely ordinary + * {@code @KafkaListener} that sends one record. The exactly-once behaviour comes from + * configuration, not from the method body. + * + *

The point of the second test is the one that matters in production: when processing fails + * after the output record has been sent, the transaction aborts. The output record is + * physically written to the log -- it has an offset, it takes disk space -- but no + * {@code read_committed} consumer will ever see it, and the input offset is not advanced, so the + * record is redelivered. On the retry the work happens again and this time commits. The downstream + * consumer sees the result exactly once, despite the processing having run twice. + * + *

That distinction -- processed twice, committed once -- is what "exactly-once" means in + * Kafka. It has never meant that your code runs once. + */ +@SpringBootTest(classes = Application.class) +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +@EmbeddedKafka( + partitions = 1, + topics = {OrderProcessor.INPUT_TOPIC, OrderProcessor.OUTPUT_TOPIC}, + // The transaction state topic defaults to 3 replicas, which a single-broker embedded + // cluster cannot provide. Without these two overrides every transactional test fails with + // a very unhelpful COORDINATOR_NOT_AVAILABLE. + brokerProperties = { + "transaction.state.log.replication.factor=1", + "transaction.state.log.min.isr=1", + "offsets.topic.replication.factor=1" + }) +class ReadProcessWriteTest { + + @Autowired + KafkaTemplate kafkaTemplate; + + @Autowired + OrderProcessor processor; + + @Autowired + EmbeddedKafkaBroker broker; + + @BeforeEach + void reset() { + processor.reset(); + } + + @Test + @Order(1) + void happyPathProducesExactlyOneOutputRecord() { + Report.title("1. Read-process-write, happy path"); + + Report.section("Publishing 3 orders to " + OrderProcessor.INPUT_TOPIC); + kafkaTemplate.executeInTransaction(t -> { + for (String order : List.of("alpha", "beta", "gamma")) { + t.send(OrderProcessor.INPUT_TOPIC, order); + Report.bullet("sent: %s", order); + } + return null; + }); + + await().atMost(Duration.ofSeconds(30)) + .until(() -> processor.invocations().size() >= 3); + + Report.section("Listener invocations"); + processor.invocations().forEach(v -> Report.bullet("processed: %s", v)); + + Report.section("What a read_committed consumer sees on " + OrderProcessor.OUTPUT_TOPIC); + List> output = drain(OrderProcessor.OUTPUT_TOPIC, "g-happy"); + output.forEach(r -> Report.bullet("offset=%d value=%s", r.offset(), r.value())); + + assertThat(output).extracting(ConsumerRecord::value) + .containsExactlyInAnyOrder("ALPHA", "BETA", "GAMMA"); + + Report.takeaway("3 in, 3 out. The listener body contains no transaction code at all --"); + Report.takeaway("the guarantee comes entirely from spring.kafka.producer.transaction-id-prefix."); + } + + @Test + @Order(2) + void failureAfterSendRollsBackAndRedelivers() { + Report.title("2. Failure AFTER the send: processed twice, delivered once"); + + Report.section("Poisoning 'delta' so its first processing attempt throws"); + processor.poison("delta"); + + kafkaTemplate.executeInTransaction(t -> t.send(OrderProcessor.INPUT_TOPIC, "delta")); + Report.bullet("sent: delta"); + + Report.section("Waiting for the listener to be invoked more than once"); + await().atMost(Duration.ofSeconds(45)) + .until(() -> processor.invocations().stream().filter("delta"::equals).count() >= 2); + + long attempts = processor.invocations().stream().filter("delta"::equals).count(); + Report.bullet("times the listener ran for 'delta' : %d", attempts); + + Report.section("What a read_committed consumer sees"); + List> committed = + drain(OrderProcessor.OUTPUT_TOPIC, "g-rollback-committed"); + List deltas = committed.stream() + .map(ConsumerRecord::value).filter("DELTA"::equals).toList(); + Report.bullet("DELTA records visible : %d", deltas.size()); + + Report.section("What a read_uncommitted consumer sees on the SAME topic"); + List> raw = + drainUncommitted(OrderProcessor.OUTPUT_TOPIC, "g-rollback-raw"); + long rawDeltas = raw.stream().map(ConsumerRecord::value).filter("DELTA"::equals).count(); + Report.bullet("DELTA records physically in the log : %d", rawDeltas); + + assertThat(attempts).as("the failure caused a redelivery").isGreaterThanOrEqualTo(2); + assertThat(deltas).as("exactly one DELTA is committed").hasSize(1); + + Report.takeaway("The listener ran %d times and a read_committed consumer sees exactly 1.", attempts); + Report.takeaway(""); + Report.takeaway("This is the whole idea: exactly-once is a property of what is COMMITTED,"); + Report.takeaway("not of how many times your code ran. Any side effect your listener performs"); + Report.takeaway("outside Kafka -- a DB write, an HTTP call, an email -- happened %d times.", attempts); + + Report.section("A detail worth noticing in the number above"); + Report.bullet("read_uncommitted sees %d DELTA record(s), not %d", rawDeltas, attempts); + Report.takeaway("The rolled-back send never reached the broker at all. KafkaTemplate.send()"); + Report.takeaway("is asynchronous: the record sat in the producer's accumulator, and"); + Report.takeaway("abortTransaction() discarded the un-flushed batch instead of writing it."); + Report.takeaway(""); + Report.takeaway("So aborted records reach the log only if they were already flushed -- which"); + Report.takeaway("happens under load, on large batches, or when you block on the send future."); + Report.takeaway("TransactionMarkerOffsetTest forces exactly that case by calling get() on"); + Report.takeaway("each send, and there the aborted records DO occupy permanent offsets."); + Report.takeaway("Do not rely on either behaviour: rely on read_committed."); + } + + // --------------------------------------------------------------------------------------- + // helpers + // --------------------------------------------------------------------------------------- + + private List> drain(String topic, String groupId) { + return drain(topic, groupId, "read_committed"); + } + + private List> drainUncommitted(String topic, String groupId) { + return drain(topic, groupId, "read_uncommitted"); + } + + private List> drain(String topic, String groupId, String isolation) { + Map props = new HashMap<>(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); + props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); + props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); + props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, isolation); + + List> out = new ArrayList<>(); + try (KafkaConsumer consumer = new KafkaConsumer<>(props)) { + TopicPartition tp = new TopicPartition(topic, 0); + consumer.assign(Set.of(tp)); + consumer.seekToBeginning(Set.of(tp)); + for (int i = 0; i < 8; i++) { + consumer.poll(Duration.ofMillis(400)).forEach(out::add); + } + } + return out; + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java b/src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java new file mode 100644 index 0000000..a2238ea --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_04_fencing/ZombieFencingTest.java @@ -0,0 +1,193 @@ +package com.ankurm.kafka.eos._04_fencing; + +import com.ankurm.kafka.eos.support.Report; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.errors.InvalidProducerEpochException; +import org.apache.kafka.common.errors.ProducerFencedException; +import org.apache.kafka.common.serialization.StringSerializer; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; + +import java.util.HashMap; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + *

Zombie fencing: what {@code transactional.id} is actually for

+ * + *

Consider a processor that reads, transforms and writes. It stalls -- a long GC pause, a + * network partition, a container the orchestrator believes is dead. Kubernetes starts a + * replacement. The consumer group rebalances and the new instance takes over the partition. + * + *

Then the old instance wakes up. It has a partially completed transaction and it has no idea it + * was replaced. If it were allowed to commit, the same input would be processed twice and both + * results would be committed. That old instance is the zombie. + * + *

The defence is the {@code transactional.id}. It is a stable, durable identity that the + * broker's transaction coordinator remembers along with an epoch. Whenever a producer calls + * {@code initTransactions()} for an existing {@code transactional.id}, the coordinator bumps the + * epoch and refuses to serve anything holding the old one. The zombie's commit is rejected. + * + *

This is precisely why the {@code transactional.id} must be stable across restarts and + * unique per logical processor. Two common mistakes, with opposite effects: + * + *

    + *
  • Random per start-up (for example a UUID) -- fencing never happens, because the new + * instance has a different identity from the zombie. You have paid for transactions and + * bought no zombie protection at all.
    • + *
  • Shared across instances -- every instance fences every other one, continuously, and + * your consumer group thrashes with {@code ProducerFencedException}s.
    • + *
+ * + *

Spring's {@code spring.kafka.producer.transaction-id-prefix} plus a per-instance suffix is + * the answer; see {@code docs/04-fencing.md} and {@code TransactionIdSuffixStrategy} + * (Spring Kafka 3.2+). + * + *

A Kafka 4.x note

+ * KIP-890 strengthened the + * transaction protocol. Under {@code transaction.version=2} -- which this embedded broker reports + * at start-up, and which is the default on Kafka 4.x -- the producer epoch is bumped on + * every transaction, not just at {@code initTransactions()}. That closes a class of + * hanging-transaction bugs, and it also means the exception you see when fenced may be + * {@link InvalidProducerEpochException} rather than the {@link ProducerFencedException} that older + * articles describe. This test reports whichever it actually gets rather than asserting one. + */ +@SpringJUnitConfig +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +@EmbeddedKafka( + partitions = 1, + topics = {ZombieFencingTest.TOPIC}, + brokerProperties = { + "transaction.state.log.replication.factor=1", + "transaction.state.log.min.isr=1", + "offsets.topic.replication.factor=1" + }) +class ZombieFencingTest { + + static final String TOPIC = "fencing-demo"; + + @Autowired + EmbeddedKafkaBroker broker; + + @Test + @Order(1) + void secondProducerWithSameTransactionalIdFencesTheFirst() throws Exception { + Report.title("1. A second producer with the same transactional.id fences the first"); + + final String sharedId = "shared-tx-id"; + + Producer zombie = transactionalProducer(sharedId); + Producer replacement = null; + try { + Report.section("Producer A ('the zombie') starts a transaction and sends"); + zombie.initTransactions(); + zombie.beginTransaction(); + // .get() forces the record to the broker, so the transaction is genuinely in progress + // rather than sitting unsent in the accumulator. + zombie.send(new ProducerRecord<>(TOPIC, "k", "from-zombie")).get(); + Report.bullet("transactional.id = '%s'", sharedId); + Report.bullet("record sent, transaction still OPEN"); + + Report.section("Producer B starts with the SAME transactional.id"); + replacement = transactionalProducer(sharedId); + replacement.initTransactions(); + Report.bullet("initTransactions() returned -- the coordinator has bumped the epoch"); + Report.bullet("any in-flight transaction under the old epoch is now aborted"); + + Report.section("Producer A tries to commit"); + final Producer fenced = zombie; + assertThatThrownBy(fenced::commitTransaction) + .isInstanceOfAny(ProducerFencedException.class, InvalidProducerEpochException.class) + .satisfies(e -> { + Report.bullet("REJECTED with %s", e.getClass().getSimpleName()); + Report.bullet("message: %s", e.getMessage()); + }); + + Report.section("Producer B commits normally"); + replacement.beginTransaction(); + replacement.send(new ProducerRecord<>(TOPIC, "k", "from-replacement")).get(); + replacement.commitTransaction(); + Report.bullet("committed successfully"); + + Report.takeaway("The zombie could not commit. Its work was discarded by the broker,"); + Report.takeaway("not by any code you wrote. That is the entire value of transactional.id:"); + Report.takeaway("a durable identity the coordinator can fence on."); + Report.takeaway(""); + Report.takeaway("Note the exception type -- on Kafka 4.x with KIP-890 (transaction.version=2)"); + Report.takeaway("the epoch is bumped every transaction, so you may see"); + Report.takeaway("InvalidProducerEpochException where older articles promise"); + Report.takeaway("ProducerFencedException. Catch both, or catch neither and let the"); + Report.takeaway("container restart the producer, which is what Spring does."); + } finally { + // A fenced producer cannot be used again; close it without attempting a flush. + zombie.close(java.time.Duration.ZERO); + if (replacement != null) { + replacement.close(); + } + } + } + + @Test + @Order(2) + void randomTransactionalIdSilentlyDefeatsFencing() throws Exception { + Report.title("2. A random transactional.id per start-up defeats fencing entirely"); + + Report.section("Producer A with a UUID-based transactional.id"); + String idA = "processor-" + java.util.UUID.randomUUID(); + Report.bullet("transactional.id = '%s'", idA); + + try (Producer a = transactionalProducer(idA)) { + a.initTransactions(); + a.beginTransaction(); + a.send(new ProducerRecord<>(TOPIC, "k", "A-uncommitted")).get(); + + Report.section("Producer B restarts and generates a NEW UUID"); + String idB = "processor-" + java.util.UUID.randomUUID(); + Report.bullet("transactional.id = '%s'", idB); + + try (Producer b = transactionalProducer(idB)) { + b.initTransactions(); + b.beginTransaction(); + b.send(new ProducerRecord<>(TOPIC, "k", "B-committed")).get(); + b.commitTransaction(); + Report.bullet("B committed"); + } + + Report.section("Now the 'zombie' A commits -- does anything stop it?"); + a.commitTransaction(); + Report.bullet("A committed successfully. Nothing was fenced."); + + assertThat(true).isTrue(); // the point is that no exception was thrown + + Report.takeaway("Both producers committed. The coordinator saw two unrelated identities"); + Report.takeaway("and had no reason to fence either one."); + Report.takeaway(""); + Report.takeaway("This is the most damaging Kafka transactions misconfiguration, because"); + Report.takeaway("everything appears to work: transactions commit, tests pass, and you have"); + Report.takeaway("exactly the duplicate-processing problem you turned transactions on to"); + Report.takeaway("prevent. The transactional.id must be STABLE across restarts and tied to"); + Report.takeaway("the partition set an instance owns -- not to the process lifetime."); + } + } + + private Producer transactionalProducer(String transactionalId) { + Map props = new HashMap<>(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId); + props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 10_000); + return new KafkaProducer<>(props); + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java b/src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java new file mode 100644 index 0000000..554688f --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_05_database/DatabaseBoundaryTest.java @@ -0,0 +1,279 @@ +package com.ankurm.kafka.eos._05_database; + +import com.ankurm.kafka.eos.Application; +import com.ankurm.kafka.eos.db.ProcessedOrder; +import com.ankurm.kafka.eos.db.ProcessedOrderRepository; +import com.ankurm.kafka.eos.support.Report; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.TestConfiguration; +import org.springframework.context.annotation.Bean; +import org.springframework.kafka.annotation.KafkaListener; +import org.springframework.kafka.core.KafkaTemplate; +import org.springframework.kafka.test.context.EmbeddedKafka; + +import java.time.Duration; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.awaitility.Awaitility.await; + +/** + *

The boundary: Kafka's exactly-once does not extend to your database

+ * + *

This is the single most consequential misunderstanding about Kafka transactions, and it is + * worth stating as plainly as possible: + * + *

+ * A Kafka transaction is atomic across Kafka partitions and Kafka consumer offsets. + * That is all. It cannot include a database write, an HTTP call, a file, or an email. + *
+ * + *

There is no two-phase commit between Kafka and your database, and there is not going to be. + * Spring used to ship {@code ChainedKafkaTransactionManager} for this and it is deprecated, + * because chaining transaction managers is best-effort ordering, not distributed coordination: + * it commits one, then the other, and a crash in between leaves them disagreeing. + * + *

So a listener that writes to a database is at-least-once with respect to that + * database, no matter what your Kafka configuration says. The redelivery that makes Kafka's + * guarantee work is exactly the thing that duplicates your row. + * + *

What these tests show

+ *
    + *
  1. The problem. A listener that inserts unconditionally produces one row per + * attempt, not one per message.
    2. + *
  2. The fix. The same listener made idempotent with a natural dedupe key produces one + * row per message however many times it runs.
    3. + *
+ * + *

The dedupe key here is {@code topic-partition-offset}, which Kafka guarantees is unique and + * stable for a record, is available on every consumer record, and requires no coordination. + * The alternative is an event ID carried in the payload, which is better when the same logical + * event can arrive on more than one topic. Either way the principle is the same, and it is the only + * approach that survives contact with reality: make the side effect idempotent, because the + * delivery is not. + * + *

See {@code docs/05-database-boundary.md} for the outbox pattern, which inverts the problem by + * making the database the source of truth and publishing to Kafka afterwards. + */ +@SpringBootTest(classes = {Application.class, DatabaseBoundaryTest.Listeners.class}) +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +@EmbeddedKafka( + partitions = 1, + topics = {DatabaseBoundaryTest.NAIVE_TOPIC, DatabaseBoundaryTest.IDEMPOTENT_TOPIC}, + brokerProperties = { + "transaction.state.log.replication.factor=1", + "transaction.state.log.min.isr=1", + "offsets.topic.replication.factor=1" + }) +class DatabaseBoundaryTest { + + static final String NAIVE_TOPIC = "db.naive"; + static final String IDEMPOTENT_TOPIC = "db.idempotent"; + + @Autowired + KafkaTemplate kafkaTemplate; + + @Autowired + ProcessedOrderRepository repository; + + @Autowired + Listeners.NaiveDbListener naive; + + @Autowired + Listeners.IdempotentDbListener idempotent; + + @BeforeEach + void clean() { + repository.deleteAll(); + naive.reset(); + idempotent.reset(); + } + + @Test + @Order(1) + void aNaiveDatabaseWriteIsDuplicatedByRedelivery() { + Report.title("1. The problem: one row per ATTEMPT, not per message"); + + Report.section("Sending one message that will fail on its first processing attempt"); + naive.failNextAttemptFor("order-1"); + kafkaTemplate.executeInTransaction(t -> t.send(NAIVE_TOPIC, "order-1")); + + await().atMost(Duration.ofSeconds(45)) + .until(() -> naive.attempts().stream().filter("order-1"::equals).count() >= 2); + // Let any further redelivery settle before counting. + sleep(2_000); + + long attempts = naive.attempts().stream().filter("order-1"::equals).count(); + List rows = repository.findByPayload("order-1"); + + Report.bullet("listener attempts for 'order-1' : %d", attempts); + Report.bullet("rows in the database : %d", rows.size()); + rows.forEach(r -> Report.bullet(" id=%d dedupeKey=%s", r.getId(), r.getDedupeKey())); + + assertThat(attempts).isGreaterThanOrEqualTo(2); + assertThat(rows.size()).as("the naive listener duplicated the row").isGreaterThan(1); + + Report.takeaway("The Kafka transaction rolled back correctly. The database row did not,"); + Report.takeaway("because it was never in the Kafka transaction and never could have been."); + Report.takeaway(""); + Report.takeaway("Note that nothing here is misconfigured. transaction-id-prefix is set,"); + Report.takeaway("isolation.level is read_committed, the Kafka side is exactly-once. The"); + Report.takeaway("database still has %d rows for 1 message.", rows.size()); + } + + @Test + @Order(2) + void anIdempotentWriteSurvivesRedelivery() { + Report.title("2. The fix: make the side effect idempotent, not the delivery"); + + Report.section("Same failure, but the listener dedupes on topic-partition-offset"); + idempotent.failNextAttemptFor("order-2"); + kafkaTemplate.executeInTransaction(t -> t.send(IDEMPOTENT_TOPIC, "order-2")); + + await().atMost(Duration.ofSeconds(45)) + .until(() -> idempotent.attempts().stream().filter("order-2"::equals).count() >= 2); + sleep(2_000); + + long attempts = idempotent.attempts().stream().filter("order-2"::equals).count(); + List rows = repository.findByPayload("order-2"); + + Report.bullet("listener attempts for 'order-2' : %d", attempts); + Report.bullet("rows in the database : %d", rows.size()); + rows.forEach(r -> Report.bullet(" id=%d dedupeKey=%s", r.getId(), r.getDedupeKey())); + + assertThat(attempts).isGreaterThanOrEqualTo(2); + assertThat(rows).as("the idempotent listener wrote exactly one row").hasSize(1); + + Report.takeaway("The listener ran %d times and wrote 1 row.", attempts); + Report.takeaway(""); + Report.takeaway("The dedupe key is 'topic-partition-offset', which Kafka guarantees is"); + Report.takeaway("unique and stable. A UNIQUE constraint on that column turns the second"); + Report.takeaway("attempt into a no-op -- and, importantly, would still do so if the"); + Report.takeaway("duplicate came from a different process, a rebalance, or a replay."); + Report.takeaway(""); + Report.takeaway("This is the whole design rule: exactly-once processing is achieved by"); + Report.takeaway("idempotent side effects plus at-least-once delivery. Kafka transactions"); + Report.takeaway("give you the second half. You have to build the first half yourself."); + } + + private static void sleep(long millis) { + try { + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + // ------------------------------------------------------------------------------------------- + + @TestConfiguration + static class Listeners { + + @Bean + NaiveDbListener naiveDbListener(ProcessedOrderRepository repository) { + return new NaiveDbListener(repository); + } + + @Bean + IdempotentDbListener idempotentDbListener(ProcessedOrderRepository repository) { + return new IdempotentDbListener(repository); + } + + /** Base class holding the failure-injection and attempt-recording plumbing. */ + abstract static class Base { + protected final ProcessedOrderRepository repository; + private final List attempts = new CopyOnWriteArrayList<>(); + private final List failOnce = new CopyOnWriteArrayList<>(); + protected final AtomicInteger counter = new AtomicInteger(); + + protected Base(ProcessedOrderRepository repository) { + this.repository = repository; + } + + protected void record(String value) { + attempts.add(value); + } + + protected void maybeFail(String value) { + if (failOnce.remove(value)) { + throw new IllegalStateException("simulated failure after the DB write for " + value); + } + } + + public void failNextAttemptFor(String value) { + failOnce.add(value); + } + + public List attempts() { + return List.copyOf(attempts); + } + + public void reset() { + attempts.clear(); + failOnce.clear(); + } + } + + /** + * Writes unconditionally. Correct-looking, and wrong the moment anything is redelivered. + */ + static class NaiveDbListener extends Base { + private static final Logger log = LoggerFactory.getLogger(NaiveDbListener.class); + + NaiveDbListener(ProcessedOrderRepository repository) { + super(repository); + } + + @KafkaListener(topics = NAIVE_TOPIC, groupId = "db-naive") + public void onMessage(String payload) { + record(payload); + // A surrogate key means every attempt looks like new work to the database. + repository.save(new ProcessedOrder("naive-" + counter.incrementAndGet(), payload)); + log.info("naive listener wrote a row for '{}'", payload); + maybeFail(payload); + } + } + + /** + * Writes only if this exact record has not been written before, keyed on the one identifier + * Kafka guarantees to be unique and stable. + */ + static class IdempotentDbListener extends Base { + private static final Logger log = LoggerFactory.getLogger(IdempotentDbListener.class); + + IdempotentDbListener(ProcessedOrderRepository repository) { + super(repository); + } + + @KafkaListener(topics = IDEMPOTENT_TOPIC, groupId = "db-idempotent") + public void onMessage(String payload, + @org.springframework.messaging.handler.annotation.Header( + org.springframework.kafka.support.KafkaHeaders.RECEIVED_TOPIC) String topic, + @org.springframework.messaging.handler.annotation.Header( + org.springframework.kafka.support.KafkaHeaders.RECEIVED_PARTITION) int partition, + @org.springframework.messaging.handler.annotation.Header( + org.springframework.kafka.support.KafkaHeaders.OFFSET) long offset) { + record(payload); + + String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset); + if (repository.existsByDedupeKey(dedupeKey)) { + log.info("idempotent listener SKIPPED '{}' -- {} already processed", payload, dedupeKey); + } else { + repository.save(new ProcessedOrder(dedupeKey, payload)); + log.info("idempotent listener wrote a row for '{}' as {}", payload, dedupeKey); + } + maybeFail(payload); + } + } + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java b/src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java new file mode 100644 index 0000000..63e6307 --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/_06_performance/TransactionCostTest.java @@ -0,0 +1,265 @@ +package com.ankurm.kafka.eos._06_performance; + +import com.ankurm.kafka.eos.support.Report; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.StringSerializer; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.kafka.test.EmbeddedKafkaBroker; +import org.springframework.kafka.test.context.EmbeddedKafka; +import org.springframework.test.context.junit.jupiter.SpringJUnitConfig; + +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + *

What do transactions actually cost?

+ * + *

"Transactions are slow" is repeated constantly and almost never quantified. The useful version + * of the statement is much more specific: the cost is per transaction, not per record. + * + *

Committing a transaction requires the producer to talk to the transaction coordinator and the + * coordinator to write a marker into every partition the transaction touched, then wait for those + * writes to be acknowledged. That is a fixed cost of roughly one network round trip plus a few + * durable writes. It does not care whether the transaction contained one record or ten thousand. + * + *

So the question is never "should I use transactions" in isolation -- it is "how many records + * should be in each one". This benchmark sweeps exactly that, plus a non-transactional baseline. + * + *

Reading the numbers honestly

+ * These run against a single-node in-JVM broker on a laptop, so the absolute throughput is not + * meaningful and is not quoted as if it were. What is meaningful, and what does transfer to real + * clusters, is the shape: a steep penalty at one record per transaction, most of it + * recovered by a hundred, and a baseline that batching approaches but never quite reaches. + * + *

On a real cluster the per-commit cost is larger (real network, real replication), so the + * penalty at small batch sizes is worse than shown here, and batching matters more. + */ +@SpringJUnitConfig +@EmbeddedKafka( + partitions = 1, + topics = {TransactionCostTest.TOPIC}, + brokerProperties = { + "transaction.state.log.replication.factor=1", + "transaction.state.log.min.isr=1", + "offsets.topic.replication.factor=1" + }) +class TransactionCostTest { + + static final String TOPIC = "perf-demo"; + + private static final int RECORDS = 10_000; + private static final String PAYLOAD = "x".repeat(256); + + @Autowired + EmbeddedKafkaBroker broker; + + @Test + void transactionOverheadIsPerTransactionNotPerRecord() { + Report.title("The cost of a transaction is paid per COMMIT, not per record"); + Report.line("Sending %,d records of %d bytes in every configuration.", RECORDS, PAYLOAD.length()); + + // ------------------------------------------------------------------------------------ + // Measurement design, and why it is not a simple loop. + // + // Version 1 warmed up briefly, then swept batch sizes in increasing order. The largest + // batch came out FIVE TIMES faster than the non-transactional baseline -- impossible. + // Version 2 added a full-size warm-up and re-measured the baseline at the end, which + // revealed the reason: the baseline had moved +181% between the start and end of the run. + // The embedded broker, the JIT and the page cache were all still warming up, and every + // configuration was silently being credited with the warm-up that preceded it. + // + // Version 3 (this one) INTERLEAVES: each round runs every configuration once, in the same + // order, and the reported figure is the median across rounds. Warm-up drift now applies + // roughly equally to all configurations instead of accruing to whichever ran last, and the + // drift check compares the first round's baseline against the last round's. + // ------------------------------------------------------------------------------------ + final int[] batchSizes = {1, 10, 100, 1_000, 10_000}; + final int rounds = 5; + + Report.section("warm-up (discarded)"); + runNonTransactional(RECORDS); + runTransactional(RECORDS, 500, "warmup-tx-" + System.nanoTime()); + Report.bullet("done"); + + Map> samples = new LinkedHashMap<>(); + samples.put("no transaction (idempotent)", new java.util.ArrayList<>()); + for (int b : batchSizes) { + samples.put("%,d records/transaction".formatted(b), new java.util.ArrayList<>()); + } + + // Round 0 is measured and printed but NOT counted. Even after the explicit warm-up above, + // the first counted round consistently came in ~60% below the rest, and including it + // dragged every median down and made the drift check meaningless. Printing it keeps the + // decision visible rather than hidden. + Report.section("%d interleaved rounds of %,d records (round 0 discarded)".formatted(rounds, RECORDS)); + for (int round = 0; round <= rounds; round++) { + StringBuilder row = new StringBuilder("round %d%s: ".formatted(round, round == 0 ? " (discarded)" : "")); + + double base = runNonTransactional(RECORDS); + if (round > 0) samples.get("no transaction (idempotent)").add(base); + row.append("baseline=%,.0f".formatted(base)); + + for (int b : batchSizes) { + double rate = runTransactional(RECORDS, b, "perf-tx-" + b + "-" + System.nanoTime()); + if (round > 0) samples.get("%,d records/transaction".formatted(b)).add(rate); + row.append(" %d/tx=%,.0f".formatted(b, rate)); + } + Report.bullet("%s", row); + } + + Map results = new LinkedHashMap<>(); + samples.forEach((label, values) -> results.put(label, median(values))); + + java.util.List baselineSamples = samples.get("no transaction (idempotent)"); + double baselineFirstRound = baselineSamples.getFirst(); + double baselineLastRound = baselineSamples.getLast(); + double drift = 100.0 * (baselineLastRound - baselineFirstRound) / baselineFirstRound; + + // ------------------------------------------------------------------------------------ + // Stability report rather than a stability assertion. + // + // The non-transactional baseline is genuinely noisy on this setup -- it varies by roughly + // +/-50% between rounds, because sending 10,000 records with no commit is fast enough that + // the measurement is dominated by flush timing and OS scheduling. Asserting on its + // stability just produces a flaky test. + // + // The TRANSACTIONAL numbers, by contrast, are very stable, because each is dominated by a + // fixed number of commits. So the defensible claim of this benchmark is not any ratio + // against the baseline -- it is the ORDERING and SPACING among transaction sizes, which is + // what the assertions below check. + // ------------------------------------------------------------------------------------ + Report.section("measurement stability"); + samples.forEach((label, values) -> { + double min = values.stream().mapToDouble(Double::doubleValue).min().orElse(0); + double max = values.stream().mapToDouble(Double::doubleValue).max().orElse(0); + Report.bullet("%-30s median=%-12s spread=%+.0f%%", + label, "%,.0f".formatted(median(values)), 100.0 * (max - min) / median(values)); + }); + Report.bullet("baseline first counted round : %,.0f records/s", baselineFirstRound); + Report.bullet("baseline last counted round : %,.0f records/s", baselineLastRound); + Report.bullet("baseline drift : %+.1f%%", drift); + + double baseline = results.get("no transaction (idempotent)"); + + Report.section("summary (median of %d rounds)".formatted(rounds)); + Report.line(" %-32s %14s %12s", "configuration", "records/s", "vs baseline"); + results.forEach((label, rate) -> + Report.line(" %-32s %14s %11.0f%%", + label, "%,.0f".formatted(rate), 100.0 * rate / baseline)); + + double perRecord = results.get("1 records/transaction"); + double perTen = results.get("10 records/transaction"); + double perHundred = results.get("100 records/transaction"); + double perThousand = results.get("1,000 records/transaction"); + + // The real claim: throughput rises monotonically with transaction size, and does so by + // close to the batching factor while commit cost dominates. + assertThat(perTen).as("10/tx beats 1/tx").isGreaterThan(perRecord); + assertThat(perHundred).as("100/tx beats 10/tx").isGreaterThan(perTen); + assertThat(perThousand).as("1000/tx beats 100/tx").isGreaterThan(perHundred); + assertThat(perTen / perRecord) + .as("going from 1 to 10 records per transaction should be worth roughly 10x") + .isGreaterThan(4.0); + + Report.takeaway("Records per transaction, and what it buys:"); + Report.takeaway(" 1 -> %,9.0f rec/s", perRecord); + Report.takeaway(" 10 -> %,9.0f rec/s (%.1fx)", perTen, perTen / perRecord); + Report.takeaway(" 100 -> %,9.0f rec/s (%.1fx)", perHundred, perHundred / perTen); + Report.takeaway(" 1,000 -> %,9.0f rec/s (%.1fx)", perThousand, perThousand / perHundred); + Report.takeaway(""); + Report.takeaway("Each 10x increase in transaction size buys close to 10x throughput while"); + Report.takeaway("commit cost dominates, then flattens as the per-record cost takes over."); + Report.takeaway("One record per transaction is roughly %.0fx slower than a hundred.", + perHundred / perRecord); + Report.takeaway(""); + Report.takeaway("(The non-transactional baseline is quoted at %,.0f rec/s but varied by", baseline); + Report.takeaway("%.0f%% across rounds, so ratios against it are indicative only. The", + 100.0 * (baselineSamples.stream().mapToDouble(Double::doubleValue).max().orElse(0) + - baselineSamples.stream().mapToDouble(Double::doubleValue).min().orElse(0)) / baseline); + Report.takeaway("comparisons BETWEEN transaction sizes are the stable, meaningful ones.)"); + Report.takeaway(""); + Report.takeaway("The lesson is not 'transactions are slow'. It is that a commit is a fixed"); + Report.takeaway("cost, so the number of records you put inside one is the setting that"); + Report.takeaway("matters. In Spring, that number is the size of the batch returned by"); + Report.takeaway("poll(), which you control with max.poll.records -- NOT with anything"); + Report.takeaway("named 'transaction'. Raising max.poll.records is the single most"); + Report.takeaway("effective throughput fix for a transactional listener, at the price of"); + Report.takeaway("redoing more work when one record in the batch fails."); + } + + // --------------------------------------------------------------------------------------- + + private double runNonTransactional(int records) { + try (Producer producer = producer(null)) { + // Force the metadata fetch BEFORE the timer starts. The transactional path gets this + // for free inside initTransactions(), so without this line the non-transactional + // baseline pays a fixed cost its competitor does not -- which made large transactions + // appear several times FASTER than sending with no transaction at all. + producer.partitionsFor(TOPIC); + long start = System.nanoTime(); + for (int i = 0; i < records; i++) { + producer.send(new ProducerRecord<>(TOPIC, "k", PAYLOAD)); + } + producer.flush(); + return rate(records, start); + } + } + + private double runTransactional(int records, int batchSize, String transactionalId) { + try (Producer producer = producer(transactionalId)) { + producer.initTransactions(); + producer.partitionsFor(TOPIC); // symmetry with the baseline above + long start = System.nanoTime(); + int sent = 0; + while (sent < records) { + producer.beginTransaction(); + int thisBatch = Math.min(batchSize, records - sent); + for (int i = 0; i < thisBatch; i++) { + producer.send(new ProducerRecord<>(TOPIC, "k", PAYLOAD)); + } + producer.commitTransaction(); + sent += thisBatch; + } + return rate(records, start); + } + } + + private static double rate(int records, long startNanos) { + return records / ((System.nanoTime() - startNanos) / 1e9); + } + + /** + * Median of a sample list. + * + *

The median rather than the mean, because a single unlucky round -- one background process, + * one thermal event, one long GC -- moves a mean by tens of percent and a median not at all. + */ + private static double median(java.util.List values) { + java.util.List sorted = new java.util.ArrayList<>(values); + java.util.Collections.sort(sorted); + int n = sorted.size(); + return n % 2 == 1 ? sorted.get(n / 2) : (sorted.get(n / 2 - 1) + sorted.get(n / 2)) / 2.0; + } + + private Producer producer(String transactionalId) { + Map props = new HashMap<>(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.getBrokersAsString()); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + // Held constant across every configuration so the only variable is the transaction. + props.put(ProducerConfig.LINGER_MS_CONFIG, 5); + props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024); + if (transactionalId != null) { + props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionalId); + props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 60_000); + } + return new KafkaProducer<>(props); + } +} diff --git a/src/test/java/com/ankurm/kafka/eos/support/Report.java b/src/test/java/com/ankurm/kafka/eos/support/Report.java new file mode 100644 index 0000000..83756f7 --- /dev/null +++ b/src/test/java/com/ankurm/kafka/eos/support/Report.java @@ -0,0 +1,40 @@ +package com.ankurm.kafka.eos.support; + +/** + * Tiny formatting helper so every demo prints a transcript with the same shape. + * + *

This exists because the output of these tests is the deliverable: the numbers quoted + * in the blog post are copied from {@code results/}, and a reader should be able to diff their own + * run against the committed one without wading through formatting differences. + */ +public final class Report { + + private Report() { + } + + public static void title(String text) { + System.out.println(); + System.out.println("=".repeat(78)); + System.out.println(text); + System.out.println("=".repeat(78)); + } + + public static void section(String text) { + System.out.println(); + System.out.println("-- " + text + " " + "-".repeat(Math.max(0, 74 - text.length()))); + } + + public static void line(String format, Object... args) { + System.out.printf(format + "%n", args); + } + + public static void bullet(String format, Object... args) { + System.out.printf(" " + format + "%n", args); + } + + /** A conclusion the reader is meant to take away, marked so it stands out in the transcript. */ + public static void takeaway(String format, Object... args) { + System.out.println(); + System.out.printf(">> " + format + "%n", args); + } +} diff --git a/src/test/resources/logback-test.xml b/src/test/resources/logback-test.xml new file mode 100644 index 0000000..0bd8ba8 --- /dev/null +++ b/src/test/resources/logback-test.xml @@ -0,0 +1,40 @@ + + + + + + + %d{HH:mm:ss.SSS} %-5level %logger{28} - %msg%n + + + + + + + + + + + + + + + + + + + + + + + + +