# 3. Transactions, isolation, and the markers nobody mentions

> Runnable companion: [`TransactionMarkerOffsetTest`](../src/test/java/com/ankurm/kafka/eos/_02_transactions/TransactionMarkerOffsetTest.java),
> [`ReadProcessWriteTest`](../src/test/java/com/ankurm/kafka/eos/_03_read_process_write/ReadProcessWriteTest.java)

## 3.1 What a transaction adds over idempotence

Idempotence protects one producer session writing to one partition. A **transaction** makes a set of
writes atomic across partitions — *and*, crucially, lets a consumer offset commit be one of those
writes.

That last part is the whole trick behind read-process-write. The offset for the input record is
written **into the same transaction** as the output record, via `sendOffsetsToTransaction`. There is
no window between "produced the output" and "committed the offset" in which a crash produces a
duplicate, because there is no between.

Setting `transactional.id` on a producer also forces `enable.idempotence=true` — transactions are
built on top of idempotence, not as an alternative to it.

## 3.2 In Spring, you write none of that

```yaml
spring.kafka.producer.transaction-id-prefix: order-processor-tx-
spring.kafka.consumer.isolation-level: read_committed
```

```java
@KafkaListener(topics = "orders.incoming", groupId = "order-processor")
public void process(String order) {
    kafkaTemplate.send("orders.validated", order.toUpperCase());
}
```

The container begins the transaction before the listener runs, enlists the `KafkaTemplate` send,
sends the offsets into the transaction, and commits — or aborts everything if the method throws.
The listener body is deliberately ordinary. **The guarantee is entirely in the configuration.**

## 3.3 `isolation.level` defaults to the wrong thing

`read_uncommitted` is the default. A consumer reading a transactional topic with the default
configuration will happily deliver records from transactions that were **aborted**.

Measured, on one partition, after committing 5 records and aborting 3:

```
read_committed   : 5 records  [committed-0 .. committed-4]
read_uncommitted : 8 records  [committed-0 .. committed-4, aborted-0, aborted-1, aborted-2]
```

`isolation.level` is a **consumer-side filter**, not a broker-side delete. The aborted records are
physically in the log with permanent offsets and they consume disk. Producing transactionally
without setting `read_committed` on every consumer gives you the cost of transactions and none of
the benefit.

## 3.4 Transaction markers consume offsets

When a transaction commits or aborts, Kafka writes a **control record** — a transaction marker —
into every partition it touched. Control records are real log entries that **consume an offset**,
but are never delivered to your consumer.

```
records sent : 5
endOffset    : 6      <- records occupy 0..4; the commit marker takes 5
last delivered offset : 4
```

Three consequences:

**Offsets are not contiguous.** Send five records in a transaction and the next record is at offset
6, not 5. Any code assuming `offset + 1` is the next record is wrong. In the Spring
read-process-write test, three messages processed in three transactions landed at offsets **0, 2 and
4** — a marker between each.

**Counting records by offset arithmetic is broken.** After 5 committed records, 3 aborted records
and 2 markers:

```
beginningOffset             : 0
endOffset                   : 10
end - beginning             : 10   <- looks like the record count
records actually retrievable: 5
overstatement               : 100%
```

Retention estimates, "how many events between these offsets", partition-size dashboards and tests
that assert on offsets are all wrong by however much your marker and abort rate happens to be.

**Consumer lag, however, is fine.** This is worth stating because the opposite is widely repeated.
A drained `read_committed` consumer's *position* advances over markers and aborted records exactly
as it advances over delivered ones, so `endOffset - committedOffset` reaches zero:

```
partition end offset      : 10
consumer committed offset : 10
lag                       : 0
```

The claim that transactions inflate consumer lag does not survive contact with a broker. The claim
that they break offset arithmetic does.

## 3.5 The Last Stable Offset, and why transaction duration is a consumer-latency setting

A `read_committed` consumer can only read up to the **Last Stable Offset** — the offset before the
first still-open transaction. If a producer opens a transaction and holds it for 30 seconds, every
`read_committed` consumer on that partition stalls for 30 seconds, *including for records committed
by other producers in the meantime*.

So `transaction.timeout.ms` and how long you keep a transaction open are consumer-latency knobs, not
just producer concerns. Long-running transactions are the most common cause of "our consumers
randomly freeze for exactly N seconds".

## 3.6 Aborted records only reach the log if they were flushed

A detail that surprised this repository's own test. `KafkaTemplate.send()` is asynchronous. If the
transaction aborts while the record is still sitting in the producer's accumulator,
`abortTransaction()` **discards the un-flushed batch** and the record never reaches the broker at
all.

So aborted records occupy permanent offsets only when they were already flushed — which happens
under load, on large batches, or when you block on the send future. Both behaviours are correct and
neither is something to rely on. Rely on `read_committed`.

## 3.7 Exactly-once means committed once, not executed once

The clearest demonstration in this repository: a listener that fails *after* sending its output.

```
times the listener ran for 'delta' : 2
DELTA records visible (read_committed) : 1
```

The work happened twice. One result was committed. **That is what exactly-once means in Kafka, and
it has never meant that your code runs once.** Every side effect outside Kafka happened twice —
see [05-database-boundary.md](05-database-boundary.md).

---

Next: [04-fencing.md](04-fencing.md).