Add the kafka-error-handling module

kafka-error-handling/docs/01-two-kinds-of-failure.md

@@ -0,0 +1,53 @@
+[Module README](../README.md) · [DefaultErrorHandler →](02-default-error-handler.md)
+
+# 1. Two kinds of failure, and why they need different machinery
+
+Kafka delivery is at-least-once. The container commits offsets after your listener returns
+([the basics article](https://ankurm.com/spring-boot-4-1-kafka-producer-consumer-serialisation/)
+covers why), so if the listener throws, the offset does not move and the record comes back.
+Everything in this module is about what happens next.
+
+There are two failures, and conflating them is why a bad record can take a partition down for
+hours.
+
+## Failure inside the listener
+
+Your code threw. The record deserialized fine; the work failed. The container catches it, hands
+it to a `CommonErrorHandler`, and that decides whether to retry, how long to wait, and what to do
+when the attempts run out.
+
+This splits again, and the split matters more than any back-off setting:
+
+| | example | retrying it |
+|---|---|---|
+| **transient** | timeout, 503, deadlock, connection reset | may succeed |
+| **permanent** | validation failure, missing entity, malformed field | will fail identically |
+
+Retrying a permanent failure ten times buys nothing and costs ten times the latency plus nine
+misleading log lines. Spring Kafka lets you say so:
+
+```java
+handler.addNotRetryableExceptions(PermanentFailure.class);
+```
+
+That single line is worth more than tuning the back-off, and it is the one most people skip.
+
+## Failure before the listener
+
+The bytes on the topic are not what the deserializer expects. Someone changed a schema, or
+published with a different serializer, or your `__TypeId__` header names a class you do not
+trust.
+
+This one is nastier, because it happens **inside `poll()`**, before any listener exists to throw
+from. There is no error handler in the path. The consumer cannot advance past the offset,
+retries the same record on the next poll, fails again, and does that forever — at whatever rate
+the poll loop runs. The partition is stopped and the only symptom is a growing lag with a
+consumer that looks healthy.
+
+That is a **poison pill**, and the cure is a different mechanism from the one above:
+`ErrorHandlingDeserializer`, covered in [chapter 3](03-poison-pills.md).
+
+Keeping these two apart is the whole point of this module. One needs a retry policy; the other
+needs a wrapper around the deserializer. Neither fixes the other.
+
+[DefaultErrorHandler →](02-default-error-handler.md)

kafka-error-handling/docs/02-default-error-handler.md

@@ -0,0 +1,71 @@
+[← Two kinds of failure](01-two-kinds-of-failure.md) · [Module README](../README.md) · [Poison pills →](03-poison-pills.md)
+
+# 2. What the default actually does
+
+If you configure nothing, the container factory installs a `DefaultErrorHandler` with
+`SeekUtils.DEFAULT_BACK_OFF` and a recoverer that logs. Run the back-off and read it off
+([`docs/output/default-backoff.txt`](output/default-backoff.txt)):
+
+```
+=== DefaultErrorHandler default back-off ===
+  interval          0 ms
+  max attempts      9 retries
+  SeekUtils.DEFAULT_MAX_FAILURES = 10
+  retry intervals   [0, 0, 0, 0, 0, 0, 0, 0, 0]
+  total deliveries  10
+```
+
+**Ten deliveries, zero milliseconds apart, and then the record is dropped.**
+
+Both halves of that surprise people. It is not "retry with backoff" — it is ten immediate
+attempts as fast as the consumer thread can run them, which against a downstream that is
+overloaded is ten times the load at the worst moment. And "then dropped" means exactly that: the
+default recoverer logs the failure and the offset moves on. There is no dead-letter topic unless
+you make one.
+
+## Giving it a back-off and a destination
+
+```java
+@Bean
+DefaultErrorHandler errorHandler(KafkaOperations<String, Object> template) {
+    DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
+    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
+    handler.addNotRetryableExceptions(PermanentFailure.class);
+    return handler;
+}
+```
+
+`FixedBackOff(1000L, 2)` is one delivery plus two retries. Measured
+([`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt)):
+
+```
+=== transient failure ===
+  deliveries        3
+  gap between 1&2   1007 ms (FixedBackOff interval 1000)
+```
+
+and the classified permanent failure gets exactly one delivery before going to the DLT.
+
+**`ExponentialBackOffWithMaxRetries` is usually the better choice** than `FixedBackOff` for a
+transient downstream, because a fixed interval synchronises every consumer in the group into
+retrying at the same instant.
+
+## The cost of blocking retries
+
+`DefaultErrorHandler` retries **on the consumer thread**. For the whole back-off, that partition
+processes nothing else. `FixedBackOff(1000L, 2)` is three seconds of a stalled partition per
+failing record — fine. A one-minute exponential back-off over five attempts is five minutes, and
+if failures are correlated you have a stalled consumer group, not a retry policy.
+
+Two consequences worth planning for:
+
+- **`max.poll.interval.ms` is your ceiling.** Default five minutes. Block longer than that
+  between polls and the broker evicts the consumer from the group, triggering a rebalance —
+  which usually makes things worse. A back-off schedule that can exceed it is a bug.
+- **Ordering is preserved**, which is the one thing blocking retries give you that
+  [retry topics](05-retryable-topic.md) do not.
+
+That trade — ordering versus throughput under failure — is the real decision, and it is covered
+in [chapter 5](05-retryable-topic.md).
+
+[Poison pills &rarr;](03-poison-pills.md)

kafka-error-handling/docs/03-poison-pills.md

@@ -0,0 +1,95 @@
+[← DefaultErrorHandler](02-default-error-handler.md) · [Module README](../README.md) · [The DLT →](04-the-dlt.md)
+
+# 3. Poison pills
+
+A record whose bytes cannot be deserialized fails inside `poll()`, before any listener exists.
+No error handler is in the path. The offset cannot advance, so the next poll fetches the same
+record and fails identically. Forever.
+
+The consumer is up, the group is stable, no exception reaches your code, and lag grows. It is
+one of the few Kafka failures with no good symptom.
+
+## `ErrorHandlingDeserializer`
+
+```yaml
+spring:
+  kafka:
+    consumer:
+      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
+      properties:
+        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
+```
+
+It wraps the real deserializer, catches the failure, and returns a **null value with the
+exception in a header**. The record then flows normally into the container, the listener is
+skipped, and the error handler gets a record it can recover — which is to say, the poison pill
+becomes an ordinary failure.
+
+Use `spring.deserializer.key.delegate.class` for keys. A malformed key is rarer and just as
+fatal.
+
+## What arrives on the DLT
+
+From [`docs/output/poison-pill.txt`](output/poison-pill.txt):
+
+```
+kafka_dlt-exception-fqcn           org.springframework.kafka.support.serializer.DeserializationException
+kafka_dlt-exception-cause-fqcn     org.springframework.kafka.support.serializer.DeserializationException
+kafka_dlt-exception-message        failed to deserialize
+kafka_dlt-original-topic           payments
+kafka_dlt-original-partition       0x00000000
+kafka_dlt-original-offset          0x0000000000000000
+kafka_dlt-original-consumer-group  payments
+__TypeId__                         [B
+DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
+```
+
+The listener was never invoked — asserted in the test — and the record is off the partition,
+which is the whole win.
+
+Two details in that block are worth stopping on.
+
+**`original-partition` and `original-offset` are binary.** They are big-endian `int` and `long`,
+not text. Printing them as a string gives you mojibake, which is why the transcript renders them
+as hex. A DLT tool that treats every header as UTF-8 will show garbage for exactly the three
+fields you need in order to find the original record.
+
+**The payload is base64.** `"eyB0aGlzIGlzIG5vdCBqc29u"` decodes to `{ this is not json`. The
+recoverer publishes with the **application's** producer, whose value serializer is
+`JacksonJsonSerializer`; the failed value is a `byte[]`; Jackson writes a `byte[]` as a base64
+JSON string. So the DLT does not hold what arrived — it holds base64 of it, wrapped in quotes.
+
+Replaying that topic naively republishes a quoted base64 string, which fails to deserialize, and
+now you have a poison pill in your poison-pill queue.
+
+## The fix
+
+Give the recoverer a template per value type:
+
+```java
+Map<Class<?>, KafkaOperations<?, ?>> templates = new LinkedHashMap<>();
+templates.put(byte[].class, byteTemplate);     // ByteArraySerializer
+templates.put(Object.class, jsonTemplate);
+DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);
+```
+
+```
+=== byte-aware DLT ===
+  DLT payload -> { this is not json
+```
+
+Byte for byte what was published. Replay is now a copy from one topic to another.
+
+Three things had to be right to get there, and each failed first — they are commented in
+[`ErrorHandlerConfiguration`](../src/main/java/com/ankurm/kafkaerrors/ErrorHandlerConfiguration.java):
+
+1. **`KafkaAutoConfiguration`'s template is `@ConditionalOnMissingBean(KafkaTemplate.class)`.**
+   Declaring `byteTemplate` removed the auto-configured `KafkaTemplate` from the context
+   entirely. Once you declare one template, you own all of them.
+2. With two templates present, `KafkaOperations<String, Object>` stops resolving, and
+   `@Qualifier` alone does **not** rescue it — the generic check runs first. Use
+   `KafkaOperations<?, ?>`, which is what the recoverer's constructor wants anyway.
+3. The map is `Map<Class<?>, KafkaOperations<?, ?>>`, matched by value type, with
+   `Object.class` as the fallback.
+
+[The DLT &rarr;](04-the-dlt.md)

kafka-error-handling/docs/04-the-dlt.md

@@ -0,0 +1,74 @@
+[← Poison pills](03-poison-pills.md) · [Module README](../README.md) · [Retry topics →](05-retryable-topic.md)
+
+# 4. The dead-letter topic
+
+## The suffix is `-dlt`, not `.DLT`
+
+```java
+public static final String RetryTopicConstants.DEFAULT_RETRY_SUFFIX = "-retry";
+public static final String RetryTopicConstants.DEFAULT_DLT_SUFFIX   = "-dlt";
+```
+
+Older Spring Kafka used `.DLT`, and most of the material online still says so. Getting it wrong
+is not an exception — it is this, at WARN, once per record:
+
+```
+o.s.k.l.DeadLetterPublishingRecoverer : Destination resolver returned non-existent partition
+payments-dlt-0, KafkaProducer will determine partition to use for this topic
+[Producer] ... {payments-dlt=UNKNOWN_TOPIC_OR_PARTITION}
+```
+
+and then, on a cluster with auto-topic-creation disabled, the record is **gone**. Your safety net
+dropped it and logged a warning. This module's tests were written against `payments.DLT` first
+and failed exactly this way.
+
+Two things follow: pre-create your DLT topics as part of provisioning, and alert on that WARN.
+
+## Same partition by default
+
+`DeadLetterPublishingRecoverer` publishes to the **same partition number** as the original. If
+your DLT has fewer partitions than the source topic, records from the high-numbered partitions
+have nowhere to go. Either give the DLT the same partition count, or set
+
+```java
+recoverer.setPartitionResolver((record, ex) -> null);   // let the producer choose
+```
+
+## The headers, and the one that will mislead you
+
+From [`docs/output/retry-and-dlt.txt`](output/retry-and-dlt.txt):
+
+```
+kafka_dlt-exception-fqcn           org.springframework.kafka.listener.ListenerExecutionFailedException
+kafka_dlt-exception-cause-fqcn     com.ankurm.kafkaerrors.Failures$TransientFailure
+kafka_dlt-original-topic           payments
+kafka_dlt-original-consumer-group  payments
+```
+
+**`kafka_dlt-exception-fqcn` is always the wrapper** for a listener failure. Build a DLT triage
+dashboard grouped by that header and every failure in the estate lands in one bucket called
+`ListenerExecutionFailedException`. The field you want is `kafka_dlt-exception-cause-fqcn`.
+
+(For a deserialization failure there is no wrapper, so the two headers agree. That inconsistency
+is worth knowing if you are writing a tool over them.)
+
+`kafka_dlt-original-consumer-group` is the one that saves you when several groups consume the
+same topic and share a DLT.
+
+## Replay
+
+A DLT is only useful if you can put records back. The mechanics are a copy:
+
+1. read from `<topic>-dlt` with a **byte-array** deserializer — the payload may be the thing that
+   could not be deserialized
+2. read `kafka_dlt-original-topic` and `kafka_dlt-original-consumer-group` to decide where it
+   belongs and whether it is yours
+3. republish to the original topic, **stripping the `kafka_dlt-*` headers** so a second failure
+   is not confused with the first
+4. do it deliberately, in bounded batches, after the cause is fixed
+
+Automatic replay is almost always wrong: the records are on the DLT precisely because something
+was not transient, and a loop that moves them back on a timer is a slow-motion outage. A replay
+you run by hand, having read the failure, is the tool worth building.
+
+[Retry topics &rarr;](05-retryable-topic.md)

kafka-error-handling/docs/05-retryable-topic.md

@@ -0,0 +1,89 @@
+[← The DLT](04-the-dlt.md) · [Module README](../README.md)
+
+# 5. Non-blocking retries with `@RetryableTopic`
+
+Blocking retries stall the partition. `@RetryableTopic` republishes the failed record to a
+separate topic and lets the main partition carry on.
+
+```java
+@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
+        sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
+        exclude = Failures.PermanentFailure.class)
+@KafkaListener(topics = "invoices", groupId = "invoices")
+public void onInvoice(ConsumerRecord<String, Payment> record, ...) { ... }
+```
+
+**Two API changes in Spring Kafka 4.x will stop older examples compiling:**
+
+- the attribute is **`backOff`**, not `backoff`
+- the annotation is **`org.springframework.kafka.annotation.BackOff`**, not
+  `org.springframework.retry.annotation.Backoff`. Spring Kafka 4 dropped the spring-retry
+  dependency and brought its own.
+
+The failure is `package org.springframework.retry.annotation does not exist`, which reads like a
+missing dependency and is not.
+
+Also new in 4.1: `sameIntervalTopicReuseStrategy` defaults to `SINGLE_TOPIC` in
+`RetryTopicConfigurationBuilder`, aligning it with the annotation's default.
+
+## What it actually does
+
+From [`docs/output/retry-topics.txt`](output/retry-topics.txt) — a failing record and a good one
+published back to back on the same partition:
+
+```
+=== @RetryableTopic delivery trace ===
+  +0      ms  invoices                     transient-1
+  +531    ms  invoices-retry-500           transient-1
+  +550    ms  invoices                     ok-1
+  +1554   ms  invoices-retry-1000          transient-1
+  +3560   ms  invoices-retry-2000          transient-1
+  DLT: [transient-1 on invoices-dlt]
+```
+
+Read the third line. `ok-1` was processed at +550 ms, while `transient-1` was still two retries
+from giving up. With a blocking handler it would have waited for the whole schedule.
+
+**Retry topics are named by the delay, not the attempt number.** `invoices-retry-500`,
+`invoices-retry-1000`, `invoices-retry-2000` — that is
+`TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE`, the default. So provisioning topics ahead of
+time means knowing your whole back-off schedule in advance, and **changing the multiplier changes
+the topic names**, orphaning whatever is still sitting in the old ones. Deploy that change the
+way you would a rename.
+
+## The cost
+
+**Per-key ordering is gone for any record that fails.** That is not a side effect; it is the
+mechanism. If `invoice-7` fails and `invoice-7`'s next event succeeds, they are processed out of
+order, and no configuration prevents it.
+
+So the decision is not "blocking or non-blocking", it is:
+
+| | blocking (`DefaultErrorHandler`) | non-blocking (`@RetryableTopic`) |
+|---|---|---|
+| ordering under failure | preserved | lost for the failing key |
+| partition throughput under failure | stalled | unaffected |
+| topics to provision | 1 + DLT | 1 + one per distinct delay + DLT |
+| long back-offs | limited by `max.poll.interval.ms` | unlimited |
+
+If your consumer is idempotent and order-insensitive — most notification, indexing and cache-warm
+consumers are — retry topics are strictly better. If it applies state transitions per key,
+blocking retries with a short schedule and a fast DLT are usually the safer answer.
+
+Use `exclude` (or `include`) rather than retrying everything: a `PermanentFailure` here skips the
+retry topics entirely and goes straight to `invoices-dlt`.
+
+## `@DltHandler`
+
+```java
+@DltHandler
+public void onDlt(ConsumerRecord<String, Payment> record,
+        @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { ... }
+```
+
+Without one, the framework still creates and populates the DLT — it just logs and moves on, and
+nothing in your application has looked at the record. A `@DltHandler` that increments a counter
+and writes a structured log line is the minimum worth having, because a DLT nobody watches is a
+queue that grows until someone notices the disk.
+
+[Module README](../README.md)

kafka-error-handling/docs/output/default-backoff.txt

@@ -0,0 +1,6 @@
+=== DefaultErrorHandler default back-off ===
+  interval          0 ms
+  max attempts      9 retries
+  SeekUtils.DEFAULT_MAX_FAILURES = 10
+  retry intervals   [0, 0, 0, 0, 0, 0, 0, 0, 0]
+  total deliveries  10

kafka-error-handling/docs/output/poison-pill.txt

@@ -0,0 +1,15 @@
+=== poison pill on the DLT ===
+  kafka_dlt-exception-fqcn           org.springframework.kafka.support.serializer.DeserializationException
+  kafka_dlt-exception-cause-fqcn     org.springframework.kafka.support.serializer.DeserializationException
+  kafka_dlt-exception-message        failed to deserialize
+  kafka_dlt-exception-stacktrace     org.springframework.kafka.support.serializer.DeserializationException: failed to deseriali...
+  kafka_dlt-original-topic           payments
+  kafka_dlt-original-partition       0x00000000
+  kafka_dlt-original-offset          0x0000000000000000
+  kafka_dlt-original-timestamp       0x000001a04bd35409
+  kafka_dlt-original-timestamp-type  CreateTime
+  kafka_dlt-original-consumer-group  payments
+  __TypeId__                         [B
+  DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"
+=== byte-aware DLT ===
+  DLT payload -> { this is not json

kafka-error-handling/docs/output/retry-and-dlt.txt

@@ -0,0 +1,15 @@
+=== transient failure ===
+  deliveries        3
+  gap between 1&2   1007 ms (FixedBackOff interval 1000)
+=== transient failure on the DLT ===
+  kafka_dlt-exception-fqcn           org.springframework.kafka.listener.ListenerExecutionFailedException
+  kafka_dlt-exception-cause-fqcn     com.ankurm.kafkaerrors.Failures$TransientFailure
+  kafka_dlt-exception-message        Listener method 'public void com.ankurm.kafkaerrors.PaymentListener.onPayment(org.apache.k...
+  kafka_dlt-exception-stacktrace     org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'publ...
+  kafka_dlt-original-topic           payments
+  kafka_dlt-original-partition       0x00000000
+  kafka_dlt-original-offset          0x0000000000000002
+  kafka_dlt-original-timestamp       0x000001a04bd35f33
+  kafka_dlt-original-timestamp-type  CreateTime
+  kafka_dlt-original-consumer-group  payments
+  __TypeId__                         com.ankurm.kafkaerrors.Payment

kafka-error-handling/docs/output/retry-topics.txt

@@ -0,0 +1,8 @@
+=== @RetryableTopic delivery trace ===
+  +0      ms  invoices                     transient-1
+  +522    ms  invoices-retry-500           transient-1
+  +539    ms  invoices                     ok-1
+  +1548   ms  invoices-retry-1000          transient-1
+  +3554   ms  invoices-retry-2000          transient-1
+  DLT: [transient-1 on invoices-dlt]
+  topics touched: [invoices, invoices-retry-1000, invoices-retry-2000, invoices-retry-500]

kafka-error-handling/docs/output/tests.txt

@@ -0,0 +1,4 @@
+Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.009 s -- in com.ankurm.kafkaerrors.RetryableTopicTest
+Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.809 s -- in com.ankurm.kafkaerrors.DeadLetterTopicTest
+Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.ankurm.kafkaerrors.DefaultBackOffTest
+Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.935 s -- in com.ankurm.kafkaerrors.ByteAwareDltTest