Kafka Error Handling with Spring Kafka 4.1: DLT, Retry Topics and Poison Pills

Kafka delivery is at-least-once. The container commits offsets after your listener returns, so if the listener throws, the offset does not move and the record comes back. Everything here is about what happens next — and the stock answer is not the one most people assume. Configure nothing and Spring Kafka gives you ten deliveries, zero milliseconds apart, and then the record is dropped. Not “retry with backoff”. Not “dead-letter it”. Ten immediate attempts against whatever was already failing, and then silence. There is also a second, nastier failure that none of that machinery touches, because it happens before your listener exists.

Verified against. JDK 25 (Temurin 25.0.4.1+1) · Spring Boot 4.1.1 · Spring Kafka 4.1.1 · kafka-clients 4.2.1. Versions from maven-metadata.xml and Boot’s own spring-boot-dependencies POM. Every transcript below came from one of the six tests in the companion project, running against a real Kafka broker started in-process in KRaft mode.

This assumes the ground covered in the producer and consumer article — in particular that the container’s default AckMode is BATCH, which is the sentence this whole article exists because of.

If you are here because…	Start at
a record failed and you cannot find it anywhere	What the default actually does
a consumer is healthy but lag keeps growing	Poison pills
your DLT is empty and there is a WARN about a partition	The dead-letter topic
every DLT record shows the same exception class	The dead-letter topic
replaying the DLT fails to deserialize	Poison pills
retries are stalling a whole partition	Non-blocking retries
package org.springframework.retry.annotation does not exist	Non-blocking retries

Two kinds of failure, and they need different machinery

The first failure splits again, and this split matters more than any back-off setting:

	example	retrying it
transient	timeout, 503, deadlock, connection reset	may succeed
permanent	validation failure, missing entity, malformed field	will fail identically

Retrying a permanent failure ten times buys nothing and costs ten times the latency plus nine log lines that look like an outage. Spring Kafka lets you say so, and this one line is worth more than any amount of back-off tuning:

handler.addNotRetryableExceptions(PermanentFailure.class);

What the default actually does

The stock handler is a DefaultErrorHandler with SeekUtils.DEFAULT_BACK_OFF. A BackOff is a state machine, so you can just run it:

=== DefaultErrorHandler default back-off ===
  interval          0 ms
  max attempts      9 retries
  SeekUtils.DEFAULT_MAX_FAILURES = 10
  retry intervals   [0, 0, 0, 0, 0, 0, 0, 0, 0]
  total deliveries  10

Both halves of that surprise people. It is not backoff — it is ten attempts as fast as the consumer thread can run them, which against an overloaded downstream is ten times the load at the worst possible moment. And “then dropped” means exactly that: the default recoverer logs and the offset moves on. There is no dead-letter topic unless you make one.

@Bean
DefaultErrorHandler errorHandler(KafkaOperations<String, Object> template) {
    DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2));
    handler.addNotRetryableExceptions(PermanentFailure.class);
    return handler;
}

Measured, that is one delivery plus two retries a second apart, and the classified permanent failure gets exactly one delivery before the DLT:

=== transient failure ===
  deliveries        3
  gap between 1&2   1007 ms (FixedBackOff interval 1000)

=== permanent failure ===
  deliveries        1

Blocking retries stall the partition, and max.poll.interval.ms is your ceiling. DefaultErrorHandler retries on the consumer thread, so for the whole back-off that partition processes nothing else. Three seconds per failing record is fine. A one-minute exponential back-off over five attempts is five minutes — and the default max.poll.interval.ms is five minutes, after which the broker evicts the consumer and triggers a rebalance, which makes everything worse. A back-off schedule that can exceed it is a bug, not a tuning choice.

Prefer ExponentialBackOffWithMaxRetries to FixedBackOff for a transient downstream: a fixed interval synchronises every consumer in the group into retrying at the same instant.

Poison pills

A record whose bytes cannot be deserialized fails inside poll(). No error handler is in the path, the offset cannot advance, and the same record fails on every subsequent poll. The fix is a different mechanism entirely:

spring:
  kafka:
    consumer:
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JacksonJsonDeserializer

It wraps the real deserializer, catches the failure, and returns a null value with the exception in a header. The record then flows normally into the container, the listener is skipped, and the error handler gets something it can recover — the poison pill becomes an ordinary failure. Use spring.deserializer.key.delegate.class for keys too; a malformed key is rarer and just as fatal. What lands on the DLT:

kafka_dlt-exception-fqcn           org.springframework.kafka.support.serializer.DeserializationException
kafka_dlt-exception-message        failed to deserialize
kafka_dlt-original-topic           payments
kafka_dlt-original-partition       0x00000000
kafka_dlt-original-offset          0x0000000000000000
kafka_dlt-original-consumer-group  payments
__TypeId__                         [B
DLT payload -> "eyB0aGlzIGlzIG5vdCBqc29u"

The listener was never invoked and the record is off the partition, which is the win. Two details in that block deserve stopping on. original-partition and original-offset are binary. They are big-endian int and long, not text — the transcript renders them as hex because printing them as UTF-8 gives mojibake. A DLT tool that treats every header as a string will show garbage for exactly the three fields you need to find the original record. The payload is base64. "eyB0aGlzIGlzIG5vdCBqc29u" decodes to { this is not json. The recoverer publishes with the application’s producer, whose value serializer is a JSON one; the failed value is a byte[]; Jackson writes a byte[] as a base64 string. So the DLT does not hold what arrived — it holds base64 of it, wrapped in quotes. Replay that naively and you republish a quoted base64 string, which fails to deserialize, and now there is a poison pill in your poison-pill queue. The fix is a template per value type:

Map<Class<?>, KafkaOperations<?, ?>> templates = new LinkedHashMap<>();
templates.put(byte[].class, byteTemplate);     // ByteArraySerializer
templates.put(Object.class, jsonTemplate);
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(templates);

=== byte-aware DLT ===
  DLT payload -> { this is not json

Byte for byte what was published. Replay is now a copy from one topic to another.

Three things had to be right to get there, and each one failed first. (1) KafkaAutoConfiguration‘s template is @ConditionalOnMissingBean(KafkaTemplate.class), so declaring byteTemplate removed the auto-configured KafkaTemplate from the context entirely — once you declare one template you own all of them. (2) With two templates present, KafkaOperations<String, Object> stops resolving, and @Qualifier alone does not rescue it because the generic check runs first; use KafkaOperations<?, ?>, which is what the recoverer’s constructor wants anyway. (3) The map is matched by value type with Object.class as the fallback.

The dead-letter topic

The suffix is -dlt, not .DLT

RetryTopicConstants.DEFAULT_RETRY_SUFFIX = "-retry";
RetryTopicConstants.DEFAULT_DLT_SUFFIX   = "-dlt";

Older Spring Kafka used .DLT, and most of the material online still says so. Getting it wrong is not an exception. It is this, at WARN, once per record:

o.s.k.l.DeadLetterPublishingRecoverer : Destination resolver returned non-existent partition
payments-dlt-0, KafkaProducer will determine partition to use for this topic
[Producer] ... {payments-dlt=UNKNOWN_TOPIC_OR_PARTITION}

and then, on a cluster with auto-topic-creation disabled, the record is gone. Your safety net dropped it and logged a warning. The tests in the companion project were written against payments.DLT first and failed in exactly this way. Two things follow: pre-create DLT topics as part of provisioning, and alert on that WARN. It is the only signal. Note also that DeadLetterPublishingRecoverer targets the same partition number as the original by default — but verifyPartition is on by default too, so when that partition does not exist on the DLT it falls back to letting the producer choose rather than losing the record. Which means the WARN above appears for two quite different situations: a DLT with fewer partitions than its source (handled, harmless) and a DLT topic that does not exist at all (fatal). The second half of the line is what distinguishes them — UNKNOWN_TOPIC_OR_PARTITION only appears for the second. To choose the partition yourself, pass a destination resolver to the constructor: a BiFunction<ConsumerRecord<?, ?>, Exception, TopicPartition>, returning a partition of -1 to defer to the producer. There is no setter for it.

The header that will mislead your triage dashboard

kafka_dlt-exception-fqcn           org.springframework.kafka.listener.ListenerExecutionFailedException
kafka_dlt-exception-cause-fqcn     com.ankurm.kafkaerrors.Failures$TransientFailure
kafka_dlt-original-topic           payments
kafka_dlt-original-consumer-group  payments

kafka_dlt-exception-fqcn is always the wrapper for a listener failure. Group a DLT dashboard by it and every failure in the estate lands in one bucket called ListenerExecutionFailedException. The field you want is kafka_dlt-exception-cause-fqcn. There is an inconsistency to code around, too: for a deserialization failure there is no wrapper, so the two headers agree. Any tool reading these must handle both shapes. kafka_dlt-original-consumer-group is the one that saves you when several groups consume the same topic and share a DLT.

Replay

A DLT is only useful if you can put records back, and the mechanics are a deliberate copy:

1. read from <topic>-dlt with a byte-array deserializer — the payload may be the thing that could not be deserialized
2. read kafka_dlt-original-topic and kafka_dlt-original-consumer-group to decide where it belongs and whether it is yours
3. republish to the original topic, stripping the kafka_dlt-* headers so a second failure is not confused with the first
4. do it in bounded batches, after the cause is fixed

Automatic replay is almost always wrong: records are on the DLT precisely because something was not transient, and a loop that moves them back on a timer is a slow-motion outage. A replay you run by hand, having read the failure, is the tool worth building.

Non-blocking retries with @RetryableTopic

@RetryableTopic(attempts = "4", backOff = @BackOff(delay = 500, multiplier = 2.0),
        sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC,
        exclude = Failures.PermanentFailure.class)
@KafkaListener(topics = "invoices", groupId = "invoices")
public void onInvoice(ConsumerRecord<String, Payment> record, ...) { ... }

Two API changes in Spring Kafka 4.x stop older examples compiling. The attribute is backOff, not backoff; and the annotation is org.springframework.kafka.annotation.BackOff, not org.springframework.retry.annotation.Backoff — Spring Kafka 4 dropped the spring-retry dependency and brought its own. The compiler says package org.springframework.retry.annotation does not exist, which reads like a missing dependency and is not. Adding spring-retry back makes it compile against the wrong annotation.

Also new in 4.1: sameIntervalTopicReuseStrategy now defaults to SINGLE_TOPIC in RetryTopicConfigurationBuilder, aligning it with the annotation’s default.

A failing record and a good one, published back to back on the same partition:

=== @RetryableTopic delivery trace ===
  +0      ms  invoices                     transient-1
  +531    ms  invoices-retry-500           transient-1
  +550    ms  invoices                     ok-1
  +1554   ms  invoices-retry-1000          transient-1
  +3560   ms  invoices-retry-2000          transient-1
  DLT: [transient-1 on invoices-dlt]

Read the third line. ok-1 was processed at +550 ms, while transient-1 was still two retries from giving up. With a blocking handler it would have waited for the whole schedule. Note the topic names. Retry topics are named by the delay, not the attempt number — invoices-retry-500, -1000, -2000. That is TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE, the default. Provisioning topics ahead of time therefore means knowing your whole back-off schedule in advance, and changing the multiplier renames them, orphaning whatever is still sitting in the old ones. Deploy that change the way you would deploy a rename.

The cost

Per-key ordering is gone for any record that fails. That is not a side effect; it is the mechanism. If invoice-7 fails and its next event succeeds, they are processed out of order, and no configuration prevents it. So the decision is not blocking versus non-blocking, it is:

	blocking (DefaultErrorHandler)	non-blocking (@RetryableTopic)
ordering under failure	preserved	lost for the failing key
partition throughput under failure	stalled	unaffected
topics to provision	1 + DLT	1 + one per distinct delay + DLT
long back-offs	limited by max.poll.interval.ms	unlimited

If your consumer is idempotent and order-insensitive — most notification, indexing and cache-warming consumers are — retry topics are strictly better. If it applies state transitions per key, blocking retries with a short schedule and a fast DLT are usually safer. Use exclude or include rather than retrying everything. A PermanentFailure here skips the retry topics entirely and goes straight to invoices-dlt. Finally, add a @DltHandler:

@DltHandler
public void onDlt(ConsumerRecord<String, Payment> record,
        @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { ... }

Without one the framework still creates and populates the DLT — it just logs and moves on, and nothing in your application has looked at the record. A handler that increments a counter and writes a structured log line is the minimum worth having, because a DLT nobody watches is a queue that grows until someone notices the disk.

What to actually configure

The honest shortlist, in order of value. (1) ErrorHandlingDeserializer — nothing else prevents a partition stopping outright, and it costs two properties. (2) A recoverer, so failures land somewhere instead of being dropped; pre-create the -dlt topics. (3) Exception classification, which saves more than any back-off. (4) Alerts on the DLT depth and on the recoverer’s WARN. Only then think about retry topics, and only if losing per-key ordering is genuinely acceptable.

And the thing not to do: leave the defaults, ship, and discover in an incident that ten instant retries and a silent drop is what “Spring Kafka handles errors” meant.

Further reading

• The companion project — six tests against a real broker, and every transcript above
• Chapter 3: poison pills — the base64 problem and the fix, in full
• Chapter 5: non-blocking retries
• Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch — why delivery is at-least-once in the first place
• Exactly-Once with Spring Kafka on Boot 4 — when at-least-once is not enough
• Spring Boot and RabbitMQ: Exchanges, Queues, Bindings and a Working Dead-Letter Queue — the same problem where the broker helps you
• Spring for Apache Kafka reference: non-blocking retries