spring-kafka-exactly-once-boot4/docs/02-idempotence.md

2. Idempotent producers

Runnable companion: IdempotentProducerTest

2.1 The problem it solves

A producer sends a record. The broker writes it. The acknowledgement is lost on the way back — a network blip, a leader election, a GC pause. The producer retries. The broker now holds the record twice, and no amount of consumer-side care can undo that: the duplicate is a real, distinct record with its own offset and its own content.

This is not exotic. It is the normal consequence of retries on an unreliable network, and retries are on by default because without them you drop records instead.

2.2 How it works

The producer obtains a producer ID (PID) from the broker and stamps every record batch with a per-partition sequence number. The broker tracks the last accepted sequence per PID per partition and rejects a repeat. A retry of a batch the broker already has is silently dropped instead of appended.

The guarantee has a precise scope: exactly-once delivery per partition, for the lifetime of one producer session. It does not survive a producer restart (a new PID is issued), and it says nothing about ordering across partitions.

2.3 It has been the default since Kafka 3.0

ProducerConfig.configDef().defaultValues().get("enable.idempotence")   // true
ProducerConfig.configDef().defaultValues().get("acks")                 // all

You can verify it from the broker's side rather than trusting the config, which is what the test in this repository does:

admin.describeProducers(List.of(partition))
     .partitionResult(partition).get().activeProducers();

A producer with idempotence on appears there with a real producer ID, epoch and last sequence number. A non-idempotent producer creates no such state, because there is nothing to de-duplicate against. Measured output:

-- default producer --
  producerId=0 producerEpoch=0 lastSequence=2
  producer states tracked : 1

-- acks=1 producer --
  producer states tracked : 0

2.4 The silent downgrade

acks and enable.idempotence interact, and — this is the important part — they interact differently depending on whether you set idempotence explicitly:

Configuration	Result
acks=1 only	Idempotence silently disabled. No warning, no exception.
acks=1 + enable.idempotence=true	ConfigException at construction.

ConfigException: Must set acks to all in order to use the idempotent producer.
Otherwise we cannot guarantee idempotence.

The first row is the dangerous one and it is extremely common: someone set acks=1 for latency, in a properties file, years ago, and the system has been quietly at-least-once ever since. Nothing in the logs says so.

Therefore: set enable.idempotence=true explicitly even though it is the default. Not because it changes behaviour, but because it converts a silent downgrade into a startup failure. That is the only reason, and it is a good one.

2.5 The other constraints

Setting	Requirement	Fails how
acks	must be all	loudly if idempotence explicit, silently otherwise
retries	must be > 0	ConfigException
max.in.flight.requests.per.connection	must be <= 5	ConfigException

Five in-flight requests is safe with idempotence because sequence numbers let the broker reorder within that window. Without idempotence, max.in.flight > 1 silently breaks ordering on retry: a failed batch that is retried can land after a batch that was sent later. That is a second, independent reason not to disable idempotence.

2.6 What idempotence is not

It is not exactly-once processing. It protects one producer session writing to one partition. It does nothing about:

• a producer restarting and re-sending from an application-level buffer,
• a consumer processing a record twice after a rebalance,
• offsets and output records committing atomically,
• anything your listener writes to a database.

For those you need transactions (03), fencing (04) and idempotent side effects (05).

---

Next: 03-transactions-and-markers.md.