Spring Boot 4.1 and Apache Kafka: Producer, Consumer and Serialisation from Scratch

Most Kafka tutorials start with a broker that is already running, a String payload, and a producer that never fails. That is the fifteen-minute version, and it leaves out everything you will spend the following fortnight on. This is the version underneath. A producer, a consumer, serialisation of an actual domain object, and the question of where a record lands — with a real broker running in-process, so every claim is a transcript rather than a paragraph. It is the on-ramp beneath the exactly-once article, which assumes all of it. Under Spring Boot 4 there are three things that did not used to be true, and each one fails in a way that does not name its cause: the dependency you remember no longer brings the auto-configuration, the JSON serializer everyone configures now belongs to the previous major version of Jackson, and the durability settings you were taught to set are already on.

Verified against. JDK 25 (Temurin 25.0.4.1+1) · Spring Boot 4.1.1 · Spring Kafka 4.1.1 · kafka-clients 4.2.1 · Jackson 3.1.5 (tools.jackson) · Testcontainers 2.0.5. Note kafka-clients: Maven Central has 4.3.1, but 4.2.1 is what the Boot BOM manages, and that is the one to run. Versions came from maven-metadata.xml and from Boot’s own spring-boot-dependencies POM. All eight tests in the companion project run against a real broker.

If you are here because…	Start at
there is no KafkaTemplate bean	The dependency you remember
your listener never fires	Consuming
Can't serialize data on a payload with a date	Two Jackson families
not in the trusted packages	Two Jackson families
messages are being processed out of order	Keys and partitions
you are wondering whether you need acks=all	The defaults are not where you think
you want to test without installing anything	Testing without a broker

The dependency you remember is the wrong one

Under Boot 3 you added org.springframework.kafka:spring-kafka and got a KafkaTemplate. Under Boot 4 you get this, at context refresh, from an application that compiled perfectly:

No qualifying bean of type 'org.springframework.kafka.core.KafkaTemplate<java.lang.String,
com.ankurm.kafkabasics.OrderEvent>' available

Boot 4 moved every auto-configuration out of spring-boot-autoconfigure into per-technology modules. Kafka’s now lives in spring-boot-kafka, package org.springframework.boot.kafka.autoconfigure, and a bare spring-kafka dependency does not bring it. Nothing warns you, because from Spring’s point of view you simply did not ask for the auto-configuration.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>

The same shape applies across Boot 4: spring-boot-starter-amqp for RabbitMQ, spring-boot-starter-restclient for RestClient.Builder. The rule of thumb is that if you are depending on a library directly rather than through a Boot starter, you are probably missing its auto-configuration. The migration guide has the general case. What the starter then gives you:

Bean	What it is for
KafkaTemplate<?, ?>	producing; typed by your injection point
ProducerFactory / ConsumerFactory	built from spring.kafka.*
KafkaListenerContainerFactory	what @KafkaListener binds to
KafkaAdmin	creates NewTopic beans at startup

KafkaAdmin deserves an early caveat: it creates topics from NewTopic beans, but it will not change an existing topic’s partition count. A NewTopic bean that disagrees with the cluster is silently ignored, not applied, which is a very quiet way to believe you have twelve partitions when you have three.

Producing, and the return value everybody throws away

public void sendAndForget(OrderEvent event) {
    this.template.send(TOPIC, event.orderId(), event);   // returns a CompletableFuture. Ignored.
}

KafkaTemplate.send is asynchronous and returns a CompletableFuture<SendResult<K, V>>. Discarding it discards the only notification you will get. The method returns normally, the record may never reach the broker, and nothing in your logs says so — the producer’s own retry and expiry messages sit under org.apache.kafka at WARN, which most applications turn down to keep startup quiet. This is the most common way to lose messages in a Spring Kafka application, and it looks exactly like correct code.

this.template.send(TOPIC, event.orderId(), event)
        .whenComplete((result, ex) -> {
            if (ex != null) {
                log.error("send failed for orderId={}", event.orderId(), ex);
            }
        });

Or block, when the caller genuinely must not proceed without a durable write:

SendResult<String, OrderEvent> result = this.template.send(record).get();
result.getRecordMetadata().partition();   // where it landed
result.getRecordMetadata().offset();      // and at what offset

That costs a network round trip plus the replication acknowledgement, so it belongs at the edges of a system rather than inside a loop.

Serialisation: two Jackson families, and your IDE will suggest the wrong one

Kafka moves byte[]. Everything else is a Serializer and a Deserializer. Under Boot 4 there is a fork in the road here that no existing example mentions, because Spring Kafka 4.1 ships two complete JSON families:

Class	Jackson	Mapper type
JsonSerializer / JsonDeserializer / JsonSerde	2.x	com.fasterxml.jackson.databind.ObjectMapper
JacksonJsonSerializer / JacksonJsonDeserializer / JacksonJsonSerde	3.x	tools.jackson.databind.json.JsonMapper

Spring Boot 4 is a Jackson 3 application. But the class named JsonSerializer — the one every existing example configures, the one autocomplete offers first, the one with the obvious name — is the Jackson 2 one. It does not fail at startup. It fails on the first payload containing a java.time value:

org.apache.kafka.common.errors.SerializationException: Can't serialize data
  [OrderEvent[orderId=o-1, ..., placedAt=2026-08-29T10:15:30Z]] for topic [orders]
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
  Java 8 date/time type `java.time.Instant` not supported by default:
  add Module "com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling

The message tells you to add a Jackson 2 module. Doing so works, and leaves you running two Jackson stacks — one for your HTTP layer, one for your messaging layer, with independent configuration and independent surprises. The better fix is one word longer:

spring:
  kafka:
    producer:
      value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
    consumer:
      value-deserializer: org.springframework.kafka.support.serializer.JacksonJsonDeserializer

Jackson 3 handles java.time with no module and no configuration:

{"orderId":"o-1","customerId":"c-1","amount":10.00,"placedAt":"2026-08-29T10:15:30Z"}

Note amount: 10.00, not 10.0 and not "10.00". BigDecimal keeps its scale through a round trip. A double would not, and money in a double is a different article.

The rule across both messaging stacks. Spring AMQP 4.1 does the same thing: Jackson2JsonMessageConverter is the Jackson 2 one, JacksonJsonMessageConverter is the Jackson 3 one. Under Boot 4, if the class name contains a 2, it belongs to the previous major version of Jackson — and the one without a number is the current one, which is the opposite of the convention you would guess.

The type header, and why it refuses your class

The three-argument serialize(topic, headers, data) overload writes a header naming the class:

__TypeId__  ->  com.ankurm.kafkabasics.OrderEvent

The consumer reads it and builds that class. Which is convenient, and is also a remote class-selection primitive, so the deserializer refuses anything outside its trusted packages:

... is not in the trusted packages: [java.util, java.lang]

Three ways out, in descending order of preference:

1. spring.json.trusted.packages: com.example.orders — an allow-list of your own packages.
2. spring.json.value.default.type — ignore the header, always build this class.
3. spring.json.trusted.packages: "*" — do not.

Option 2 is better than its position suggests when producer and consumer belong to different teams, because it makes the consumer’s contract the consumer’s own decision. The __TypeId__ header couples two services by fully qualified class name: rename or move a class in the producer and every consumer that trusts the header breaks at runtime, with no compile-time warning anywhere. spring.json.type.mapping — a logical name to class map, configured on both sides — is the version of this that survives a refactor.

Keys and partitions: the ordering guarantee in disguise

Kafka orders records within a partition. Not within a topic. So the key is not a label; it decides which records are ordered relative to each other, and choosing it is the most consequential line in a producer. Six keys, three partitions, against a real broker:

key    partition   murmur2 & 0x7fffffff % 3   Math.abs(murmur2) % 3
o-1    0           0                          0
o-2    0           0                          2
o-3    1           1                          1
o-4    1           1                          1
o-5    2           2                          0
o-6    1           1                          1

The default partitioner is murmur2 of the serialized key bytes, masked positive, modulo the partition count. Look at the last two columns: & 0x7fffffff and Math.abs disagree on two of six keys, because clearing the sign bit is not the same number as negating it. If you ever reimplement the partitioner to predict placement — for a test, a migration, a routing table — Math.abs gives you the right answer about half the time, which is the worst available failure mode.

Three consequences worth internalising. A null key is not a key. Records without one are spread by the sticky partitioner, so nothing about their relative order is guaranteed. Two events describing the same entity, unkeyed, can be processed out of order by different consumers. Adding partitions repartitions every key. The modulus changes, so keys move. Anything relying on per-key ordering loses it across the resize for records still in flight. Choose the partition count with room to grow; changing it later is a data-ordering event. Key cardinality is your parallelism ceiling. Keying by customerId when one customer is 40% of traffic gives you a hot partition that no amount of consumer scaling fixes, because one partition is consumed by exactly one member of a group. Choosing the key is choosing what must stay ordered. By orderId if operations on one order must not overtake each other; by customerId if that must hold across a customer’s orders. Those are different systems with different throughput ceilings.

Consuming

@KafkaListener(topics = "orders", groupId = "orders-basic")
public void onOrder(@Payload OrderEvent event,
        @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
        @Header(KafkaHeaders.OFFSET) long offset) { ... }

No poll loop, no offset commit, no rebalance listener — the container owns all three. That is the real value Spring Kafka adds over the plain client.

“My listener never fires.” auto.offset.reset decides where a consumer group with no committed offset starts, and the kafka-clients default is latest — skip everything already in the topic. So a brand-new group attached to a topic full of records reads nothing. Boot does not change this; you have to: spring.kafka.consumer.auto-offset-reset: earliest. It only applies while there is no committed offset, which is why the symptom vanishes the first time it works and never comes back.

The other reliable cause is a groupId mismatch between the annotation and spring.kafka.consumer.group-id. The annotation wins, and a typo there creates a brand-new group, which then hits the paragraph above.

spring.kafka.listener.concurrency creates that many consumers in the group, each on its own thread, and the ceiling is the partition count. With 3 partitions, concurrency: 10 gives you three working consumers and seven idle ones. There is no setting that makes two threads consume one partition, because that would break the ordering guarantee above. Declare the key header as required = false unless you are certain every record has one — a null key against a required header parameter fails conversion rather than arriving as null.

The defaults are not where you think they are

Here is the effective producer configuration of a stock Boot 4.1 application, printed from the running ProducerFactory against the ConfigDef defaults of the kafka-clients on the classpath:

property                                       kafka-clients default        set by Spring Boot
acks                                           all                          -
enable.idempotence                             true                         -
retries                                        2147483647                   -
max.in.flight.requests.per.connection          5                            -
linger.ms                                      5                            -
delivery.timeout.ms                            120000                       -

Spring Boot sets nothing on the producer beyond bootstrap servers and serializers. Since Kafka 3.0 the client defaults are acks=all and enable.idempotence=true, so a stock Boot application already has a durable, deduplicating producer. Two consequences that cut against habit:

• You do not need to set acks=all. It is already on, and adding it changes nothing.
• An old runbook that sets acks=1 or retries=0 is now a downgrade. Those lines were written when the defaults were weaker. Deleting them makes the system safer — the opposite of how configuration usually ages.

delivery.timeout.ms at two minutes is the one worth revisiting: it is the total budget for a send including retries, and it is the clock your future is waiting on when it never completes.

And a property that is not where you would look for it

The container’s default AckMode is BATCH: commit the whole poll() batch after the listener has returned for every record in it. That is at-least-once delivery, which is why your listener must be idempotent and why error handling is a necessary sequel rather than an optional one. Now trace enable.auto.commit, because every debugging session about commits starts in the wrong place:

• The kafka-clients default is true.
• Spring Boot does not set it. It is absent from ConsumerFactory.getConfigurationProperties(), before the containers start and after.
• ConsumerFactory.isAutoCommit() returns true on a stock Boot 4.1 application, because it reads that same absent key and falls back to the client default.
• And yet no consumer auto-commits, because ListenerConsumer.determineAutoCommit checks whether the factory config contains the key and, when it does not, calls setProperty("enable.auto.commit", "false") on the per-container Properties handed to createConsumer.

So the shared factory never learns, its public accessor answers the opposite of the truth, and the real value lives in an override map you cannot reach from application code. All four statements are asserted in the companion project, including the counter-intuitive one:

assertThat(this.consumerFactory.isAutoCommit()).isTrue();

If you need to know whether a running consumer auto-commits, read the consumer’s own metrics or its startup log line. Not the factory.

Manual acknowledgement

@KafkaListener(topics = "orders")
void onOrder(OrderEvent event, Acknowledgment acknowledgment) {
    process(event);
    acknowledgment.acknowledge();
}

Nothing commits until acknowledge() runs, which is what you want when the work must be durable before the offset moves. The failure mode is worth knowing in advance: an Acknowledgment you forget to call stalls the partition — not immediately, but once max.poll.records of un-acknowledged records accumulate. The symptom is a consumer that works for a while and then stops, which reads like a broker problem and is not.

Testing without installing anything

@SpringBootTest
@EmbeddedKafka(topics = "orders", partitions = 3)
class KeysAndPartitionsTest { ... }

spring-kafka-test starts EmbeddedKafkaKraftBroker — the actual Apache Kafka broker classes, in KRaft mode, in-process. No ZooKeeper, no container, no daemon. It binds a random port and exposes it as ${spring.embedded.kafka.brokers}, and it is ready in about three seconds. Every transcript in this article came from it, on a machine with no Docker. For the handful of tests where the difference between “the broker classes” and “the broker you deploy” matters — TLS, SASL, quotas, real network behaviour — use Testcontainers:

@TestConfiguration(proxyBeanMethods = false)
public class TestcontainersConfiguration {

    @Bean
    @ServiceConnection
    KafkaContainer kafkaContainer() {
        return new KafkaContainer(DockerImageName.parse("apache/kafka:4.1.0"));
    }

}

@ServiceConnection registers the container’s bootstrap servers as the application’s, which removes the @DynamicPropertySource block older examples all carry.

Two Testcontainers coordinates changed, and both will bite. The Maven artifact is org.testcontainers:testcontainers-kafka, not org.testcontainers:kafka — Testcontainers 2.x prefixed every module, the old coordinate stopped at 1.21.4, and Boot 4.1.1 imports testcontainers-bom 2.0.5 which manages only the new names. The failure is 'dependencies.dependency.version' for org.testcontainers:kafka:jar is missing, which does not mention the rename.

The class moved too: org.testcontainers.kafka.KafkaContainer (Apache Kafka, KRaft), not org.testcontainers.containers.KafkaContainer (Confluent images, ZooKeeper).

Where to go next

Do you need Kafka at all? Worth asking before the second service. Kafka’s value is a durable, replayable, partition-ordered log that many independent consumers read at their own pace. If you have one producer, one consumer, and no need to replay, a database table with a status column is less operational surface and easier to reason about — and if what you want is per-message routing, retries and a dead-letter queue with no ordering requirement, RabbitMQ does that with a queue argument rather than a retry topic. Kafka earns its keep when the log itself is the asset.

• The companion project — eight tests against a real broker, and the transcripts above
• Chapter 3: serialisation — the two Jackson families in full
• Chapter 6: acknowledgement — where enable.auto.commit actually lives
• Kafka Error Handling with Spring Kafka 4.1 — what happens when the listener throws, and what a poison pill does to a partition
• Exactly-Once with Spring Kafka on Boot 4 — transactions, once at-least-once is not enough
• Spring Boot 3 to 4 Migration Guide — the module split behind the missing bean
• Spring for Apache Kafka reference documentation