Exactly-Once with Spring Kafka on Boot 4: Idempotent Producers and Transactions

“Exactly-once” is the most oversold phrase in distributed systems, and Kafka is where most Java developers meet it. The feature is real, it works, and it is genuinely useful. It also guarantees something much narrower than its name suggests, and the gap between the two is where production incidents live. This post is a careful tour of what Spring Kafka actually gives you on Spring Boot 4.1: what idempotent producers protect, what transactions add, and — the part usually skipped — where the guarantee stops. Every claim below was verified against a live broker rather than taken from documentation, and several of them came out differently than expected. One widely repeated claim about consumer lag turned out to be false. Another about aborted records turned out to be conditional. The complete runnable suite is in the companion repository — fifteen tests against an in-JVM broker, no Docker required.

Versions (July 2026): Verified against Spring Boot 4.1.0, Spring Kafka 4.1.0, kafka-clients 4.2.1 and JDK 25.0.3. Kafka 4.x is KRaft-only — ZooKeeper is gone — and runs the strengthened transaction protocol from KIP-890 at transaction.version=2, which changes some behaviour older articles describe.

The one-sentence version

Before any code, the sentence that the rest of this post elaborates:

Kafka's exactly-once is atomic across Kafka partitions and Kafka consumer offsets. That is the entire scope. It does not mean your listener runs once, and it cannot include your database.

Everything that follows is either a mechanism that delivers that guarantee, or a consequence of its boundaries.

Setting it up on Boot 4 (and the dependency that will cost you an hour)

Start with the trap, because it is new in Boot 4 and the error message points the wrong way. On Boot 3, depending on org.springframework.kafka:spring-kafka was enough, because KafkaAutoConfiguration lived in the monolithic spring-boot-autoconfigure jar every application already had. Boot 4 split auto-configuration into per-technology modules. Kafka's now lives in org.springframework.boot:spring-boot-kafka. Depend on spring-kafka alone and you get the library but no auto-configuration — no ProducerFactory, no KafkaTemplate, no KafkaTransactionManager. The failure looks like this:

Error creating bean with name 'orderProcessor': Unsatisfied dependency expressed
through constructor parameter 0: No qualifying bean of type
'org.springframework.kafka.core.KafkaTemplate<java.lang.String, java.lang.String>' available

That reads like a generics problem. It is not. Use the starter:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>

With that in place, the entire exactly-once switch is one property:

spring:
  kafka:
    producer:
      transaction-id-prefix: order-processor-tx-   # this line is the whole feature
    consumer:
      isolation-level: read_committed              # default is read_uncommitted!
      enable-auto-commit: false

Spring Boot sees the prefix, auto-configures a KafkaTransactionManager, and attaches it to every listener container.

Half of it you already have: idempotent producers

An idempotent producer solves one specific problem. You send a record, the broker writes it, and the acknowledgement is lost coming back. The producer retries. Without idempotence the broker now holds the record twice, and no amount of consumer-side care can undo that — the duplicate is a real record with its own offset. The fix is a producer ID plus a per-partition sequence number; the broker rejects a repeat. It has been on by default since Kafka 3.0. Most articles assert that and move on. It is more interesting to ask the broker:

// Ask the broker what idempotence state it is tracking for this partition
List<ProducerState> states = admin.describeProducers(List.of(partition))
        .partitionResult(partition).get().activeProducers();

A producer with idempotence enabled appears there with a real producer ID. A producer without it creates no such state, because there is nothing to de-duplicate against. Measured:

-- default producer, 3 records sent --
  producerId=0 producerEpoch=0 lastSequence=2
  producer states tracked : 1

-- acks=1 producer, 3 records sent --
  producer states tracked : 0

The silent downgrade. acks and enable.idempotence interact differently depending on whether you set idempotence explicitly. Set only acks=1 and idempotence is silently disabled — no warning, no exception, nothing in the logs. Set acks=1 and enable.idempotence=true and you get a ConfigException at construction: “Must set acks to all in order to use the idempotent producer.” The first case is the dangerous one and it is everywhere — somebody set acks=1 for latency, in a properties file, years ago. So set enable.idempotence=true explicitly even though it is already the default, not because it changes behaviour but because it converts a silent downgrade into a startup failure. That is the only reason, and it is a good one.

Idempotence still is not exactly-once processing: it covers one producer session writing to one partition. It says nothing about a consumer processing a record twice after a rebalance, or about offsets and outputs committing together. That is what transactions add.

The other half: transactions and read-process-write

A transaction makes writes atomic across partitions — and lets a consumer offset commit be one of those writes. That last part is the entire trick.

In Spring you write none of that:

@KafkaListener(topics = "orders.incoming", groupId = "order-processor")
public void process(String order) {
    // No beginTransaction. No sendOffsetsToTransaction. No commit.
    // The container did all of it; this send simply enlists.
    kafkaTemplate.send("orders.validated", order.toUpperCase());
}

The listener body is deliberately ordinary. The guarantee is entirely in the configuration.

What happens when processing fails

The interesting case is failure after the output has been sent. Measured on a listener poisoned to throw on its first attempt:

times the listener ran for 'delta'      : 2
DELTA records visible (read_committed)  : 1

The work happened twice. One result was committed. That is what exactly-once means in Kafka, and it has never meant that your code runs once.

The markers nobody mentions

When a transaction commits or aborts, Kafka writes a control record — a transaction marker — into every partition it touched. Control records are real log entries that consume an offset but are never delivered to your consumer.

Three consequences, all measured:

• Offsets are not contiguous. Five records in a transaction, and the next record is at offset 6. In the Spring read-process-write test, three messages processed in three transactions landed at offsets 0, 2 and 4.
• Counting records by offset arithmetic is broken. After 5 committed records, 3 aborted and 2 markers: offset span says 10, retrievable records are 5 — a 100% overstatement. Retention math, throughput dashboards and tests that assert on offsets are all affected.
• isolation.level is a consumer-side filter, not a broker-side delete. The aborted records are physically in the log with permanent offsets. A default (read_uncommitted) consumer reads data you deliberately aborted.

A correction, since this post is meant to be checked rather than believed. It is widely repeated that transaction markers inflate consumer lag, because endOffset counts records that are never delivered. That is false, and the test written to demonstrate it disproved it instead. A drained read_committed consumer's position advances over markers and aborted records exactly as it advances over delivered ones, so endOffset - committedOffset reaches zero. Lag monitoring is fine. Offset arithmetic is what breaks. The distinction matters because the two mistakes have completely different fixes.

What transactional.id is actually for

A processor stalls — long GC pause, network partition, a container the orchestrator gave up on. A replacement starts and takes over the partition. Then the old instance wakes up holding a half-finished transaction. That is the zombie, and fencing it is what transactional.id exists for: the broker remembers that identity along with an epoch, and every initTransactions() bumps the epoch and invalidates anything holding the old one.

Producer A: initTransactions, beginTransaction, send   (transaction OPEN)
Producer B: initTransactions with the SAME id          (epoch bumped)
Producer A: commitTransaction
  -> REJECTED with ProducerFencedException
Producer B: commits normally

Now the version that should worry you. Give each start-up a fresh random id — a UUID, which looks like sensible uniqueness hygiene:

// Catastrophic, and everything appears to work
props.put(TRANSACTIONAL_ID_CONFIG, "processor-" + UUID.randomUUID());

Producer A (uuid-1): transaction open
Producer B (uuid-2): commits
Producer A (uuid-1): commits successfully. Nothing was fenced.

Both commit. The coordinator saw two unrelated identities and had no reason to fence either one. This is the most damaging Kafka transactions misconfiguration precisely because it is invisible: transactions commit, tests pass, and you have exactly the duplicate processing you enabled transactions to prevent. The id must be stable across restarts and unique per instance — an application name plus a stable ordinal, never a random value and never one shared by every replica. One Kafka 4.x note: under KIP-890 the producer epoch is bumped on every transaction, so the exception may be InvalidProducerEpochException rather than the ProducerFencedException older articles promise. Catch both.

Where the guarantee stops: your database

This is the most consequential misunderstanding, so it is worth being blunt. A Kafka transaction cannot include a database write, an HTTP call, a file, or an email. There is no two-phase commit between Kafka and your database and there is not going to be. Spring's ChainedKafkaTransactionManager existed for this and is deprecated, because chaining transaction managers is best-effort ordering, not distributed coordination — it narrows the window, it does not close it. So a listener that writes to a database is at-least-once with respect to that database, whatever your Kafka configuration says. Measured, on a listener that fails once after writing:

-- naive listener (unconditional insert) --
listener attempts : 2
rows in database  : 2      <-- one message, two rows

-- idempotent listener (dedupe on topic-partition-offset) --
listener attempts : 2
rows in database  : 1

Nothing is misconfigured in the first case. transaction-id-prefix is set, isolation.level is read_committed, the Kafka side is genuinely exactly-once. The database still has two rows for one message. The fix is not a cleverer transaction manager. It is to make the side effect idempotent:

// topic-partition-offset is unique and stable, needs no coordination,
// and is available on every consumer record
String dedupeKey = "%s-%d-%d".formatted(topic, partition, offset);
if (!repository.existsByDedupeKey(dedupeKey)) {
    repository.save(new ProcessedOrder(dedupeKey, payload));
}

with a UNIQUE constraint on that column doing the real enforcement, since two concurrent attempts can both pass the exists check. Which gives the rule the whole topic reduces to:

Exactly-once processing = at-least-once delivery + idempotent side effects. Kafka transactions give you the delivery half, and give it to you very well. You build the other half yourself. Anyone describing end-to-end exactly-once across your whole system is describing Kafka-to-Kafka and has not thought about the database.

The alternative, when the work genuinely cannot be made idempotent, is the transactional outbox: write the business change and an outbox row in one local database transaction, and let a relay publish to Kafka afterwards. That is covered with a worked example in docs/05-database-boundary.md.

What it costs

“Transactions are slow” is repeated constantly and almost never quantified. The useful version is far more specific: the cost is per transaction, not per record. A commit is a round trip to the coordinator plus a durable marker write per partition. That fixed cost does not care whether the transaction held one record or ten thousand. 10,000 records of 256 bytes, median of five interleaved rounds:

Records per transaction	Throughput	Gain over previous row
no transaction (idempotent only)	801,880 rec/s	—
1	711 rec/s	—
10	8,452 rec/s	11.9x
100	78,770 rec/s	9.3x
1,000	330,969 rec/s	4.2x
10,000	878,071 rec/s	2.7x

One record per transaction is roughly 111x slower than one hundred. Each 10x increase buys close to 10x while commit cost dominates, then flattens. And the setting that controls this in Spring is max.poll.records — not anything named “transaction”. With a KafkaTransactionManager, the transaction boundary is the batch returned by poll(). Raising it is the single most effective throughput fix for a transactional listener; the price is that one poisoned record rolls back and redoes the whole batch. (These are laptop numbers against a single-node in-JVM broker. The non-transactional baseline was noisy at 26% spread; the transactional measurements were stable. Comparisons between transaction sizes are the trustworthy part, and the benchmark took three revisions to stop measuring its own warm-up — the reasons are documented in the source.)

A checklist

1. Use spring-boot-starter-kafka, not bare spring-kafka, on Boot 4.
2. Set transaction-id-prefix — stable across restarts, unique per instance, never random.
3. Set isolation-level: read_committed on every consumer of a transactional topic.
4. Set enable.idempotence: true explicitly, as an assertion against a future acks=1.
5. Make every non-Kafka side effect idempotent. Assume redelivery, because you will get it.
6. Never count records by subtracting offsets.
7. Tune max.poll.records, and keep transaction.timeout.ms comfortably above worst-case batch processing time.
8. If your topology is Kafka-only, consider Kafka Streams with processing.guarantee=exactly_once_v2 instead — it does all of this for you.

Everything else

The repository goes considerably deeper than this post. A few of the things it covers that did not fit here, each with runnable proof:

• Aborted records only reach the log if they were already flushed — abortTransaction() discards un-flushed batches, so whether a rolled-back record occupies a permanent offset depends on timing. docs/03 §3.6
• The Last Stable Offset makes transaction duration a consumer-latency setting — one slow producer stalls every read_committed consumer on the partition. docs/03 §3.5
• transaction.timeout.ms versus max.poll.interval.ms — exceed either and you get both symptoms at once, in a confusing log. docs/07 §7.1
• Dead letter topics are published inside the transaction, so “moved to DLT and offset advanced” is atomic — which is why a DLT is the correct terminal state for a poison record under EOS. docs/07 §7.5
• Manual acknowledgment does nothing when a transaction manager is present, and executeInTransaction must not be called inside a listener. docs/07 §7.3–7.4
• Hanging transactions block log compaction indefinitely — if a compacted topic stops shrinking, check for open transactions before checking compaction settings. docs/07 §7.8
• EOSMode V2 and TransactionIdSuffixStrategy — why older guidance about one transactional.id per topic-partition no longer applies. docs/04 §4.4
• Testing without Docker — the three brokerProperties that every transactional @EmbeddedKafka test needs, and why omitting them gives an unhelpful COORDINATOR_NOT_AVAILABLE. docs/01 §1.4

Reproducing this

git clone https://ankurm.com/git.app/asmhatre/spring-kafka-exactly-once-boot4.git
cd spring-kafka-exactly-once-boot4
mvn test

Fifteen tests, about three minutes, no Docker — every test starts its own in-JVM KRaft broker. Unedited output of the full run is committed as results/full-run.txt.

Further reading

• spring-kafka-exactly-once-boot4 — the companion repository: seven documentation chapters and fifteen runnable demonstrations with captured output
• Spring for Apache Kafka: Transactions — reference documentation
• Spring for Apache Kafka: Exactly Once Semantics
• KIP-890: Transactions Server-Side Defense — the Kafka 4.x protocol change
• Apache Kafka: Transaction Protocol
• Spring Data JPA 3 to 4 Migration Guide — the other half of a Boot 4 upgrade
• Micrometer to OpenTelemetry: The Spring Boot 4 Observability Guide — how to see any of this in production