Ankur
e23ca359eb
Companion suite for the ankurm.com guide. Every test runs against an in-JVM KRaft broker
(spring-kafka-test), so there is no Docker requirement. Verified on Spring Boot 4.1.0,
Spring Kafka 4.1.0, kafka-clients 4.2.1, JDK 25.0.3. results/full-run.txt is unedited
output of mvn test: 15 tests, 0 failures.
Demonstrations
_01_idempotence verifies idempotence from the BROKER via Admin.describeProducers rather
than by reading config back: a default producer registers one producer
state, an acks=1 producer registers zero. Also covers the ConfigException
you get only when idempotence is requested explicitly.
_02_transactions transaction markers consuming offsets, aborted records surviving in the
log, read_committed vs read_uncommitted, and a correction: markers do NOT
inflate consumer lag (a drained consumer reaches zero), they break
counting records by offset arithmetic (measured 100% overstatement).
_03_read_process_write
the Spring loop with container-managed transactions; a listener that
fails after sending runs twice and is committed once.
_04_fencing shared transactional.id fences the zombie; a random per-startup id lets
both producers commit and silently defeats the guarantee entirely.
_05_database Kafka EOS does not extend to a database. Naive listener writes 2 rows for
1 message; the same listener keyed on topic-partition-offset writes 1.
_06_performance cost per transaction size, median of 5 interleaved rounds.
Docs
Seven chapters: Boot 4 setup (auto-configuration moved to spring-boot-starter-kafka),
idempotence, transactions and markers, fencing and KIP-890, the database boundary and the
outbox pattern, performance, and corner cases.
Measured highlights
acks=1 producer 0 broker-tracked producer states (idempotence silently off)
5 records in one tx endOffset 6; offset span overstates record count by 100%
1 record per tx 711 records/s
100 records per tx 78,770 records/s (~111x)
116 lines
4.5 KiB
XML
116 lines
4.5 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Companion code for "Exactly-Once with Spring Kafka on Spring Boot 4" (ankurm.com).
|
|
|
|
Everything runs against an in-JVM KRaft broker started by spring-kafka-test, so there is no
|
|
Docker requirement and no external Kafka to install:
|
|
|
|
mvn test
|
|
|
|
A docker-compose.yml is included for readers who would rather point the same code at a real
|
|
broker; see docs/00-running.md.
|
|
-->
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<modelVersion>4.0.0</modelVersion>
|
|
|
|
<parent>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-parent</artifactId>
|
|
<version>4.1.0</version>
|
|
<relativePath/>
|
|
</parent>
|
|
|
|
<groupId>com.ankurm.kafka</groupId>
|
|
<artifactId>spring-kafka-exactly-once-boot4</artifactId>
|
|
<version>1.0.0</version>
|
|
<name>Exactly-once with Spring Kafka on Spring Boot 4</name>
|
|
<description>
|
|
Runnable demonstrations of idempotent producers, Kafka transactions, read-process-write,
|
|
zombie fencing, transaction markers, and the Kafka/database boundary.
|
|
</description>
|
|
|
|
<properties>
|
|
<java.version>25</java.version>
|
|
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
|
</properties>
|
|
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter</artifactId>
|
|
</dependency>
|
|
|
|
<!--
|
|
IMPORTANT, and a real Spring Boot 3 -> 4 migration trap.
|
|
|
|
On Boot 3 you could depend on org.springframework.kafka:spring-kafka directly and still
|
|
get auto-configuration, because KafkaAutoConfiguration lived in the monolithic
|
|
spring-boot-autoconfigure jar that every application already had.
|
|
|
|
Boot 4 split auto-configuration into per-technology modules. Kafka's now lives in
|
|
org.springframework.boot:spring-boot-kafka. Depend on spring-kafka alone and you get the
|
|
library but NO auto-configuration: no ProducerFactory, no KafkaTemplate, no
|
|
KafkaTransactionManager. The failure is a confusing NoSuchBeanDefinitionException for
|
|
KafkaTemplate at startup, which looks like a generics problem and is not.
|
|
|
|
spring-boot-starter-kafka pulls in both spring-kafka and spring-boot-kafka.
|
|
See docs/01-boot4-setup.md.
|
|
-->
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-kafka</artifactId>
|
|
</dependency>
|
|
|
|
<!-- Used only by the module that shows why "Kafka + DB atomically" is not a thing. -->
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-data-jpa</artifactId>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>com.h2database</groupId>
|
|
<artifactId>h2</artifactId>
|
|
<scope>runtime</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-test</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
<!--
|
|
spring-kafka-test pulls in the Kafka broker itself. On Spring Kafka 4.x this is
|
|
EmbeddedKafkaKraftBroker: Kafka 4.x removed ZooKeeper entirely, so the old
|
|
ZooKeeper-based embedded broker no longer exists.
|
|
-->
|
|
<dependency>
|
|
<groupId>org.springframework.kafka</groupId>
|
|
<artifactId>spring-kafka-test</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>org.awaitility</groupId>
|
|
<artifactId>awaitility</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
|
|
<build>
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-surefire-plugin</artifactId>
|
|
<configuration>
|
|
<!--
|
|
The demos print narrated output that is meant to be read. Without this,
|
|
Surefire swallows stdout unless a test fails.
|
|
-->
|
|
<redirectTestOutputToFile>false</redirectTestOutputToFile>
|
|
<trimStackTrace>false</trimStackTrace>
|
|
</configuration>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
</project>
|