Exactly-once with Spring Kafka on Spring Boot 4: demonstrations, docs and captured output

Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.

Demonstrations
  _01_idempotence   verifies idempotence from the BROKER via Admin.describeProducers rather
                    than by reading config back: a default producer registers one producer
                    state, an acks=1 producer registers zero. Also covers the ConfigException
                    you get only when idempotence is requested explicitly.
  _02_transactions  transaction markers consuming offsets, aborted records surviving in the
                    log, read_committed vs read_uncommitted, and a correction: markers do NOT
                    inflate consumer lag (a drained consumer reaches zero), they break
                    counting records by offset arithmetic (measured 100% overstatement).
  _03_read_process_write
                    the Spring loop with container-managed transactions; a listener that
                    fails after sending runs twice and is committed once.
  _04_fencing       shared transactional.id fences the zombie; a random per-startup id lets
                    both producers commit and silently defeats the guarantee entirely.
  _05_database      Kafka EOS does not extend to a database. Naive listener writes 2 rows for
                    1 message; the same listener keyed on topic-partition-offset writes 1.
  _06_performance   cost per transaction size, median of 5 interleaved rounds.

Docs
  Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
  idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
  outbox pattern, performance, and corner cases.

Measured highlights
  acks=1 producer          0 broker-tracked producer states (idempotence silently off)
  5 records in one tx      endOffset 6; offset span overstates record count by 100%
  1 record  per tx            711 records/s
  100 records per tx       78,770 records/s  (~111x)

docs/01-boot4-setup.md

@@ -0,0 +1,121 @@
+# 1. Setting up exactly-once on Spring Boot 4
+
+> Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3.
+
+## 1.1 The dependency trap that will cost you an hour
+
+On Spring Boot 3 this was enough:
+
+```xml
+<dependency>
+  <groupId>org.springframework.kafka</groupId>
+  <artifactId>spring-kafka</artifactId>
+</dependency>
+```
+
+…because `KafkaAutoConfiguration` lived in the monolithic `spring-boot-autoconfigure` jar that every
+Boot application already had on the classpath.
+
+**Boot 4 split auto-configuration into per-technology modules.** Kafka's now lives in
+`org.springframework.boot:spring-boot-kafka`. Depend on `spring-kafka` alone and you get the
+library but *no* auto-configuration: no `ProducerFactory`, no `KafkaTemplate`, no
+`KafkaTransactionManager`.
+
+The failure looks like this, and it looks like a generics problem, which it is not:
+
+```
+Error creating bean with name 'orderProcessor': Unsatisfied dependency expressed through
+constructor parameter 0: No qualifying bean of type
+'org.springframework.kafka.core.KafkaTemplate<java.lang.String, java.lang.String>' available
+```
+
+Use the starter:
+
+```xml
+<dependency>
+  <groupId>org.springframework.boot</groupId>
+  <artifactId>spring-boot-starter-kafka</artifactId>
+</dependency>
+```
+
+which pulls in both `spring-kafka` and `spring-boot-kafka`.
+
+## 1.2 The configuration
+
+The whole exactly-once switch is one property:
+
+```yaml
+spring:
+  kafka:
+    producer:
+      transaction-id-prefix: order-processor-tx-
+```
+
+Spring Boot sees it, auto-configures a `KafkaTransactionManager`, and attaches it to every listener
+container. From that point the container begins a Kafka transaction before invoking your listener,
+writes the consumer offsets *into* that transaction, and commits — or aborts everything if your
+listener throws.
+
+Two more properties matter and neither is on by default:
+
+```yaml
+spring:
+  kafka:
+    consumer:
+      isolation-level: read_committed   # default is read_uncommitted!
+      enable-auto-commit: false
+```
+
+`isolation-level` is the one people forget. Producing transactionally while consuming
+`read_uncommitted` gives you all of the cost of transactions and none of the benefit — your
+consumers will happily read records from transactions that were aborted. Proven in
+[`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java).
+
+## 1.3 Config worth stating even though it is already the default
+
+```yaml
+      acks: all
+      properties:
+        enable.idempotence: true
+        max.in.flight.requests.per.connection: 5
+```
+
+All three are defaults on Kafka 3.0+. Stating them explicitly is still worth it, because
+`enable.idempotence: true` converts a *silent* downgrade into a *startup failure* if anyone later
+sets `acks=1` for latency. See [02-idempotence.md](02-idempotence.md) — this is the single most
+common way a system that believes it is exactly-once is not.
+
+## 1.4 Testing without Docker
+
+Kafka 4.x removed ZooKeeper entirely, so `spring-kafka-test` now only ships
+`EmbeddedKafkaKraftBroker`. It runs a real broker in-JVM:
+
+```java
+@SpringBootTest(classes = Application.class)
+@EmbeddedKafka(
+        partitions = 1,
+        topics = {"orders.incoming", "orders.validated"},
+        brokerProperties = {
+                "transaction.state.log.replication.factor=1",
+                "transaction.state.log.min.isr=1",
+                "offsets.topic.replication.factor=1"
+        })
+class MyTest { }
+```
+
+**Those three `brokerProperties` are mandatory for any transactional test.** The transaction state
+log defaults to 3 replicas, which a single-broker embedded cluster cannot provide. Without them
+every transactional test fails with a `COORDINATOR_NOT_AVAILABLE` that gives no hint as to why.
+
+Two further limitations of the KRaft embedded broker:
+
+- **You cannot pin ports.** `KafkaClusterTestKit` does not support it, so the `ports` attribute of
+  `@EmbeddedKafka` is ignored. Read the address from `EmbeddedKafkaBroker.getBrokersAsString()` or
+  the `spring.embedded.kafka.brokers` system property.
+- Broker start-up costs a couple of seconds per distinct configuration. Spring's context caching
+  will reuse a broker across test classes whose configuration matches exactly, so keeping
+  `@EmbeddedKafka` annotations identical across classes is a real speed-up.
+
+---
+
+Next: [02-idempotence.md](02-idempotence.md).