2.4 KiB
Module README · DefaultErrorHandler →
1. Two kinds of failure, and why they need different machinery
Kafka delivery is at-least-once. The container commits offsets after your listener returns (the basics article covers why), so if the listener throws, the offset does not move and the record comes back. Everything in this module is about what happens next.
There are two failures, and conflating them is why a bad record can take a partition down for hours.
Failure inside the listener
Your code threw. The record deserialized fine; the work failed. The container catches it, hands
it to a CommonErrorHandler, and that decides whether to retry, how long to wait, and what to do
when the attempts run out.
This splits again, and the split matters more than any back-off setting:
| example | retrying it | |
|---|---|---|
| transient | timeout, 503, deadlock, connection reset | may succeed |
| permanent | validation failure, missing entity, malformed field | will fail identically |
Retrying a permanent failure ten times buys nothing and costs ten times the latency plus nine misleading log lines. Spring Kafka lets you say so:
handler.addNotRetryableExceptions(PermanentFailure.class);
That single line is worth more than tuning the back-off, and it is the one most people skip.
Failure before the listener
The bytes on the topic are not what the deserializer expects. Someone changed a schema, or
published with a different serializer, or your __TypeId__ header names a class you do not
trust.
This one is nastier, because it happens inside poll(), before any listener exists to throw
from. There is no error handler in the path. The consumer cannot advance past the offset,
retries the same record on the next poll, fails again, and does that forever — at whatever rate
the poll loop runs. The partition is stopped and the only symptom is a growing lag with a
consumer that looks healthy.
That is a poison pill, and the cure is a different mechanism from the one above:
ErrorHandlingDeserializer, covered in chapter 3.
Keeping these two apart is the whole point of this module. One needs a retry policy; the other needs a wrapper around the deserializer. Neither fixes the other.