spring-kafka-exactly-once-boot4/docs/03-transactions-and-markers.md

3. Transactions, isolation, and the markers nobody mentions

Runnable companion: TransactionMarkerOffsetTest, ReadProcessWriteTest

3.1 What a transaction adds over idempotence

Idempotence protects one producer session writing to one partition. A transaction makes a set of writes atomic across partitions — and, crucially, lets a consumer offset commit be one of those writes.

That last part is the whole trick behind read-process-write. The offset for the input record is written into the same transaction as the output record, via sendOffsetsToTransaction. There is no window between "produced the output" and "committed the offset" in which a crash produces a duplicate, because there is no between.

Setting transactional.id on a producer also forces enable.idempotence=true — transactions are built on top of idempotence, not as an alternative to it.

3.2 In Spring, you write none of that

spring.kafka.producer.transaction-id-prefix: order-processor-tx-
spring.kafka.consumer.isolation-level: read_committed

@KafkaListener(topics = "orders.incoming", groupId = "order-processor")
public void process(String order) {
    kafkaTemplate.send("orders.validated", order.toUpperCase());
}

The container begins the transaction before the listener runs, enlists the KafkaTemplate send, sends the offsets into the transaction, and commits — or aborts everything if the method throws. The listener body is deliberately ordinary. The guarantee is entirely in the configuration.

3.3 isolation.level defaults to the wrong thing

read_uncommitted is the default. A consumer reading a transactional topic with the default configuration will happily deliver records from transactions that were aborted.

Measured, on one partition, after committing 5 records and aborting 3:

read_committed   : 5 records  [committed-0 .. committed-4]
read_uncommitted : 8 records  [committed-0 .. committed-4, aborted-0, aborted-1, aborted-2]

isolation.level is a consumer-side filter, not a broker-side delete. The aborted records are physically in the log with permanent offsets and they consume disk. Producing transactionally without setting read_committed on every consumer gives you the cost of transactions and none of the benefit.

3.4 Transaction markers consume offsets

When a transaction commits or aborts, Kafka writes a control record — a transaction marker — into every partition it touched. Control records are real log entries that consume an offset, but are never delivered to your consumer.

records sent : 5
endOffset    : 6      <- records occupy 0..4; the commit marker takes 5
last delivered offset : 4

Three consequences:

Offsets are not contiguous. Send five records in a transaction and the next record is at offset 6, not 5. Any code assuming offset + 1 is the next record is wrong. In the Spring read-process-write test, three messages processed in three transactions landed at offsets 0, 2 and 4 — a marker between each.

Counting records by offset arithmetic is broken. After 5 committed records, 3 aborted records and 2 markers:

beginningOffset             : 0
endOffset                   : 10
end - beginning             : 10   <- looks like the record count
records actually retrievable: 5
overstatement               : 100%

Retention estimates, "how many events between these offsets", partition-size dashboards and tests that assert on offsets are all wrong by however much your marker and abort rate happens to be.

Consumer lag, however, is fine. This is worth stating because the opposite is widely repeated. A drained read_committed consumer's position advances over markers and aborted records exactly as it advances over delivered ones, so endOffset - committedOffset reaches zero:

partition end offset      : 10
consumer committed offset : 10
lag                       : 0

The claim that transactions inflate consumer lag does not survive contact with a broker. The claim that they break offset arithmetic does.

3.5 The Last Stable Offset, and why transaction duration is a consumer-latency setting

A read_committed consumer can only read up to the Last Stable Offset — the offset before the first still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every read_committed consumer on that partition stalls for 30 seconds, including for records committed by other producers in the meantime.

So transaction.timeout.ms and how long you keep a transaction open are consumer-latency knobs, not just producer concerns. Long-running transactions are the most common cause of "our consumers randomly freeze for exactly N seconds".

3.6 Aborted records only reach the log if they were flushed

A detail that surprised this repository's own test. KafkaTemplate.send() is asynchronous. If the transaction aborts while the record is still sitting in the producer's accumulator, abortTransaction() discards the un-flushed batch and the record never reaches the broker at all.

So aborted records occupy permanent offsets only when they were already flushed — which happens under load, on large batches, or when you block on the send future. Both behaviours are correct and neither is something to rely on. Rely on read_committed.

3.7 Exactly-once means committed once, not executed once

The clearest demonstration in this repository: a listener that fails after sending its output.

times the listener ran for 'delta' : 2
DELTA records visible (read_committed) : 1

The work happened twice. One result was committed. That is what exactly-once means in Kafka, and it has never meant that your code runs once. Every side effect outside Kafka happened twice — see 05-database-boundary.md.

---

Next: 04-fencing.md.