Top 40 Kafka Interview Questions and Answers (Scenario-Based)
40 scenario-based Kafka interview questions with short answers: ordering, acks and replication, rebalancing, lag, exactly-once, dead-letter topics and share groups. Claims are verified against a real Kafka 4.4.0 broker with downloadable Java files and their actual output.
Apache Kafka is a distributed, append-only commit log. Producers append records to topics; each topic is split into partitions, and every record in a partition gets a sequential offset. Consumers read by offset and remember where they are, so a record is not deleted when it is read — it is kept for a retention period and can be read again. Almost every Kafka interview scenario is about one of four things: ordering, delivery guarantees, rebalancing, or what to do with a record that will not process.
This post has 40 scenario-based questions, each phrased the way an interviewer actually asks it (“consumer lag keeps growing — what do you check?”) with a short answer and the trap behind it. The claims that are easy to get wrong are not argued from memory: they run against a real Kafka 4.4.0 broker in three small Java files you can download, and the console output below is their actual output.
Scenario
What the interviewer is testing
Question #
Events for one order arrive out of order
Keys, partitions, ordering scope
3–5
Messages lost when a broker dies
acks, replication, min.insync.replicas
6–8
Duplicate records after a retry
Idempotent producer, idempotent consumer
9, 31
Consumer group keeps rebalancing
Poll interval, session timeout, assignors
18–21
Lag is growing
Throughput math, partitions, slow handlers
23, 39
Exactly-once
Transactions, read_committed, side effects
27–30
One bad record blocks the partition
Dead-letter topic, retry topics
32
Queue semantics on Kafka
Share groups (KIP-932)
33
Versions used here. Apache Kafka 4.4.0 (KRaft mode, single broker on localhost), kafka-clients 4.4.0, JDK 25.0.4.1. The defaults table further down is printed from the real 4.4.0 client classes, not copied from documentation. Checked on 22 October 2026.
How keys, partitions and consumers fit together
Key-to-partition mapping taken from the lab run below (order-1 on partition 1, order-2 and order-3 on partition 0).
The diagram is drawn from the first lab run below: the six keyed sends really did land on partitions 0 and 1, never on 2 in this sample. Two ideas answer a large share of interview questions: ordering exists only inside a partition, and a partition belongs to one consumer of a group at a time. Everything about scaling, rebalancing and lag follows from those two rules.
Six experiments against a real broker
Download the files, start a Kafka 4.4.0 broker on localhost:9092 and run them with the Kafka client jars on the classpath: KafkaInterviewLab.java, KafkaShareLab.java and KafkaDefaults.java. The snippets below are abbreviated excerpts; the console blocks are the actual output of a single run (each run creates fresh topics with a random suffix).
order-1 always goes to partition 1 with offsets 0, 1, 2; order-2 and order-3 share partition 0. Within partition 0 the offsets are 0 (order-2), 1 (order-3), 2 (order-2): order is kept per partition, not across partitions. This is the evidence behind questions 3 to 5.
2. Offsets and lag
c.commitSync();
long committed = c.committed(Set.of(tp)).get(tp).offset();
long end = c.endOffsets(Set.of(tp)).get(tp);
System.out.println("[2] polled=" + recs.count() + " committed=" + committed
+ " logEnd=" + end + " lag=" + (end - committed));
[2] polled=4 committed=4 logEnd=10 lag=6
Ten records are in the log; the consumer polled four (max.poll.records=4) and committed. Lag is the gap: 10 − 4 = 6. The committed offset is the position of the next record to read, which is why committing after processing gives at-least-once.
3. Rebalancing
var c1 = consumer(group, Map.of()); c1.subscribe(List.of(topic)); // topic has 4 partitions
var c2 = consumer(group, Map.of()); c2.subscribe(List.of(topic)); // second member of the same group
...
c2.close(); // c1 gets everything back
A second member joins and the four partitions are split two and two; when it leaves, the first consumer takes all four. While that reassignment happens the affected partitions are not being consumed, which is the cost question 18 is about.
4. Transactions: read_committed vs read_uncommitted
try (var p = producer(Map.of("transactional.id", "tx-" + run, "enable.idempotence", true))) {
p.initTransactions();
p.beginTransaction(); p.send(new ProducerRecord<>(topic, "k", "committed-1")).get(); p.commitTransaction();
p.beginTransaction(); p.send(new ProducerRecord<>(topic, "k", "aborted-2")).get(); p.abortTransaction();
}
// then read the topic twice: isolation.level = read_uncommitted, then read_committed
[4] read_uncommitted sees [committed-1, aborted-2]
[4] read_committed sees [committed-1]
Same topic, two isolation levels. read_uncommitted (the default) sees the aborted record; read_committed does not. A consumer in a pipeline that relies on transactions must set isolation.level=read_committed, or the transaction protects nothing.
5. Poison pill and dead-letter topic
try {
sum += Integer.parseInt(r.value());
} catch (NumberFormatException e) {
ProducerRecord<String, String> d = new ProducerRecord<>(dlt, r.key(), r.value());
d.headers().add("x-original-offset", Long.toString(r.offset()).getBytes());
d.headers().add("x-exception", e.getClass().getSimpleName().getBytes());
p.send(d).get();
}
[5] processed sum=70, sent to DLT=1
DLT record value=not-a-number original-offset=2 exception=NumberFormatException
The input was 10, 20, not-a-number, 40. The three good values sum to 70 (10 + 20 + 40) and the bad one went to the DLT with its original offset (2) and exception class as headers, so someone can inspect it later without the partition ever stalling.
6. Share groups: two workers on one partition
cp.put("share.acknowledgement.mode", "explicit");
try (var c = new KafkaShareConsumer<String, String>(cp)) {
c.subscribe(List.of(topic));
for (var r : c.poll(Duration.ofMillis(500))) {
handle(r);
c.acknowledge(r, AcknowledgeType.ACCEPT);
}
c.commitSync();
}
[6] 1 partition, 2 share consumers, 20 records delivered
both workers got work: true
no record handled twice: true
With a classic consumer group, two consumers on a single partition leaves one idle. Here both share-group workers received records from the same partition and no record was handled twice. The lab starts both workers first and produces afterwards with processing slowed down (150 ms per record), so the work is visibly shared; with fast handlers one worker can take most records.
Real defaults in Kafka 4.4
Interviewers love “what is the default for X”. KafkaDefaults.java reads them from the client config definitions rather than from memory:
Notable: the producer is already safe by default (acks=all plus idempotence), while the consumer still defaults to auto.offset.reset=latest, enable.auto.commit=true and the classic group protocol. The two-assignor default list is why cooperative rebalancing is available without configuration.
Open the accordions you want to practise. Groups: basics, ordering and replication (1–8), producer (9–15), consumers and groups (16–26), delivery semantics (27–32), operations and design (33–40).
1. What is Kafka, and why would you pick it over a plain message queue?
A replicated, partitioned commit log. Records are retained after being read, so many independent consumer groups can read the same stream at their own pace and any group can replay history. A classic queue deletes a message once it is acknowledged. Pick Kafka for event streaming, fan-out and replay; pick a queue for task distribution with per-message routing. Comparison: Kafka vs RabbitMQ vs Pulsar.
2. Explain topic, partition and offset in two sentences.
A topic is a named stream split into partitions; a partition is an ordered, immutable log, and an offset is a record’s position inside one partition. Offsets are not unique across partitions, so a record is identified by topic + partition + offset.
3. Scenario: events for the same order are processed out of order. What went wrong?
Either the records have no key (so they are spread over partitions) or they use different keys for the same order. Kafka orders records only within a partition, and the default partitioner sends the same key to the same partition. In the lab, order-1 lands on partition 1 every time (offsets 0, 1, 2). Fix: key by the entity whose order matters (order id, account id).
4. Scenario: the business wants strict global ordering. Can Kafka do that?
Only with a single partition, which caps throughput and parallelism to one consumer per group. Usually the requirement is really ordering per entity, which keys already give. If global order truly is required, say what it costs.
5. Scenario: traffic doubled, so you add partitions to a keyed topic. What breaks?
The key-to-partition mapping is hash(key) % partitionCount, so changing the count remaps keys: new records for order-1 may go to a different partition than its older records, and per-key order is lost across the change. You also cannot reduce a topic’s partitions. Choose the count with headroom up front, and for existing keyed topics plan a migration to a new topic.
6. Scenario: a broker dies. How do replication and the ISR protect data?
Each partition has a leader and followers; followers that are caught up form the in-sync replica set (ISR). If the leader dies, a new leader is elected from the ISR, so no acknowledged record is lost provided it was acknowledged by all in-sync replicas (acks=all) and the ISR was not shrunk to one. Replication factor 3 with min.insync.replicas=2 is the standard production setting.
7. Scenario: records vanished after a leader crash although the producer saw success. Why?
The producer used acks=1 (or 0): the leader acknowledged, then died before followers copied the record. Since Kafka 3.0 the producer default is acks=all with idempotence on (see the defaults table below), but older or hand-tuned configs may still use acks=1. acks=all only helps if min.insync.replicas is at least 2; with 1, all can mean “only the leader”.
8. What is unclean leader election and when would you enable it?
Electing a replica that is not in the ISR as leader, which restores availability but can lose records the old leader had. It is off by default. Enable it only for topics where availability beats durability (metrics, clickstream), never for payments.
9. Scenario: a network blip made the producer retry. Could that create duplicates?
Without idempotence, yes: the first send may have succeeded and only the acknowledgement was lost. With enable.idempotence=true (the default in Kafka 4.x) the broker tracks a producer id and per-partition sequence numbers and discards the retry, so the partition gets the record once. This protects against producer retries only, not against your application sending the same business event twice.
10. What do the default producer settings in Kafka 4.4 look like?
Printed from the real client: acks=all, enable.idempotence=true, retries=2147483647 bounded by delivery.timeout.ms=120000, linger.ms=5, batch.size=16384, max.in.flight.requests.per.connection=5 and no compression. The full table is in the lab section below.
11. How do linger.ms and batch.size trade latency for throughput?
The producer groups records per partition into batches and sends a batch when it is full (batch.size) or has waited linger.ms. A little linger gives bigger batches, better compression and fewer requests at the cost of a few milliseconds of latency. For high volume, raise linger to 10–50 ms and batch size to 64–128 KB and measure.
12. When is compression worth enabling and where does it happen?
Almost always for text-heavy payloads (JSON, Avro): the producer compresses whole batches (lz4 or zstd are common picks), the broker stores them compressed, and consumers decompress. It reduces network and disk, and costs some CPU. Bigger batches compress better, which is another reason for a small linger.
13. What does delivery.timeout.ms control?
The upper bound on the whole send, from send() returning to success or failure, including retries and time in the batch. Retries are effectively unlimited by count; this timeout is what ends them. Set request.timeout.ms smaller than it.
14. Scenario: send() reported failure after the retries. What do you do?
Do not drop it silently. Use the callback or Future to detect the failure, then retry later, persist the event for a retry job, or alert. If the event came from a database change, the transactional outbox keeps it safely until it is published.
15. Scenario: one partition is far busier than the others. How do you diagnose and fix it?
A hot key: one key (a big customer, a null tenant id) produces most traffic and the hash puts it on one partition. Check per-partition offsets or byte rates. Fixes: pick a higher-cardinality key, add a salt for that key (and accept loss of per-key order across salts), or a custom partitioner for known heavy keys.
16. How does a consumer group split the work?
Within a group each partition is assigned to exactly one consumer, so consumers share the partitions; different groups each receive all records. The lab starts with one consumer on 4 partitions ([0, 1, 2, 3]), adds a second (c1=[0, 1] c2=[2, 3]), then removes it and c1 gets all four back.
17. Scenario: you have 10 consumers and 6 partitions. What happens?
Six consumers get one partition each and four sit idle. Parallelism in a classic group is capped by the partition count. To scale further you need more partitions, or share groups (question 33) if you want record-level work distribution.
18. What triggers a rebalance?
A consumer joins or leaves (including crashing or being slow), the subscription changes, or partitions are added. During an eager rebalance the group stops processing while partitions are reassigned. The lab’s section 3 triggers two of those on purpose.
19. Scenario: the group rebalances over and over. What do you check?
First max.poll.interval.ms (default 300000): if one batch of up to max.poll.records (500) takes longer than that to process, the consumer is kicked out and the group rebalances; lower max.poll.records or speed up the handler. Then session.timeout.ms (45000) and heartbeat.interval.ms (3000), GC pauses, and pods being restarted by failing probes. Rolling deployments also cause a rebalance per pod.
20. How do cooperative rebalancing and static membership reduce disruption?
The default assignors in 4.4 are RangeAssignor then CooperativeStickyAssignor; the cooperative one moves only the partitions that must move instead of revoking everything. Static membership (group.instance.id) lets a restarted consumer come back with its old partitions inside the session timeout, so a rolling restart does not trigger a rebalance per pod.
21. What is the new consumer rebalance protocol (KIP-848)?
A server-side protocol where the broker computes assignments and consumers reconcile incrementally, removing the group-wide “stop the world”. It is opt-in on the client: the default in 4.4 is still group.protocol=classic, and you set group.protocol=consumer to use it. The broker lab reports the supported protocols as classic, consumer and streams.
22. Auto-commit or manual commit — which and why?
Auto-commit (the client default is enable.auto.commit=true) commits offsets on a timer, which can commit records you have polled but not yet processed, so a crash loses them (at-most-once for that window). Manual commit after processing gives at-least-once. Commit per batch, not per record.
23. Scenario: what exactly is consumer lag, and how do you compute it?
Lag is the log end offset minus the group’s committed offset, per partition. The lab polls 4 records of 10, commits, and reports committed=4 logEnd=10 lag=6. Lag in records can mislead: also watch the time since the oldest unprocessed record.
24. What does auto.offset.reset do, and what is its trap?
It applies only when the group has no committed offset (new group, or the offset expired): latest (default) starts at the end and silently skips existing data; earliest starts at the beginning. A new group on a busy topic with the default will look like it receives nothing.
25. Scenario: a bug corrupted three hours of processing. How do you replay?
Stop the consumers, then reset the group’s offsets to a timestamp before the bug (kafka-consumer-groups.sh --reset-offsets --to-datetime) or use seek in code, fix the bug, and restart. Replay is only possible within retention, and your processing must be idempotent because it will see records again.
26. What is a compacted topic and when do you use it?
With cleanup.policy=compact Kafka keeps at least the latest record per key, deleting older ones in the background; a record with a null value (tombstone) deletes the key. Use it for changelog-style state (latest address per customer, config), not for event history. Compaction is eventual, so readers can still see older values for a while.
27. Explain at-most-once, at-least-once and exactly-once in Kafka terms.
At-most-once: commit the offset before processing (a crash loses records). At-least-once: process, then commit (a crash repeats records) — the practical default. Exactly-once: Kafka transactions make a consume-process-produce loop atomic within Kafka, but it does not extend to external side effects. See exactly-once with Spring Kafka.
28. How do Kafka transactions work, and what is read_committed for?
A transactional producer (transactional.id) writes records and commit or abort markers atomically across partitions. Consumers with isolation.level=read_committed skip aborted records. The default read_uncommitted shows them. In the lab one record is committed and one aborted; compare what each level sees in the output below.
29. Scenario: a consume-transform-produce service must not duplicate output after a crash. How?
Use a transactional producer and send the consumed offsets to the transaction (sendOffsetsToTransaction) so the output records and the input offset commit together, and downstream consumers read with read_committed. If the instance crashes mid-way, the transaction is aborted and the input is reprocessed.
30. Scenario: you enabled exactly-once but the customer was emailed twice. Why?
Kafka’s guarantee covers records inside Kafka. An email, an HTTP call or a database write outside the transaction is a side effect that Kafka cannot roll back. Make the side effect idempotent (a dedupe key, a unique constraint) or record it in the same database transaction as the offset or an outbox.
31. Scenario: how do you make a consumer idempotent?
Give each event a stable id and record processed ids with the effect in one local transaction (or rely on a natural key such as an upsert on the business key). A duplicate then becomes a no-op. This is the single most useful pattern, because at-least-once delivery means duplicates will arrive.
32. Scenario: one malformed record keeps failing and blocks the partition. What do you do?
A poison pill. Retrying forever stalls everything behind it. Send it to a dead-letter topic with context headers (original offset, exception) and continue; the lab does exactly this: processed sum=70, sent to DLT=1. For transient failures add retry topics with increasing delays. Full Spring treatment: Spring Kafka 4.1 error handling, DLT and retry topics.
33. What are Kafka share groups and when would you use them?
Share groups (KIP-932) give queue semantics on Kafka: several consumers can read the same partition concurrently, and each record is delivered to one of them with per-record acknowledgement (accept, release, reject). Use them for job processing where you want more workers than partitions and do not need per-key order. The lab runs 2 workers on 1 partition. Deeper: share groups with Spring Kafka 4.
34. Kafka or RabbitMQ for task processing?
RabbitMQ (or a share group) fits per-message routing, priorities and work queues with acknowledgements. Kafka fits high-throughput streams, replay and many independent consumers. Say what the workload needs (ordering, replay, throughput, routing) rather than naming a favourite.
35. How do you evolve message schemas without breaking consumers?
Use a schema format with compatibility rules (Avro or Protobuf with a schema registry) and pick a mode: backward compatible means new readers can read old data, so add fields with defaults and never remove or retype. For JSON, version the event and tolerate unknown fields. Deploy consumers before producers for backward-compatible changes.
36. Retention versus compaction: how long does Kafka keep data?
By time (retention.ms, default 7 days) or size (retention.bytes) for delete-policy topics; by key for compacted ones. Retention is per topic and independent of whether anyone has read the data. A consumer that is offline longer than retention loses its position.
37. What changed in Kafka 4.x that an interviewer might ask about?
ZooKeeper is gone: Kafka runs only in KRaft mode, where a controller quorum stores metadata in Kafka itself. The new consumer group protocol is available, share groups arrived, and the producer defaults of acks=all plus idempotence (since 3.0) are the baseline. Migration notes for Spring: Spring Kafka 3 to 4 migration guide.
38. Scenario: how would you reprocess everything from the start into a new database?
Create a new consumer group with auto.offset.reset=earliest and let it read the retained history while the old group keeps serving. Throttle if the target cannot keep up, and make writes idempotent. If retention is shorter than your history, you need a snapshot plus the log.
39. Scenario: producers outrun consumers and lag keeps growing. What are your options?
Measure first: is the handler slow (profile it, batch database writes), is the group capped by partitions (add partitions and consumers), or is a rebalance loop stalling work (question 19)? Only then scale. Also check whether a downstream dependency is the bottleneck; adding consumers there makes it worse.
40. Which metrics tell you a Kafka system is healthy?
Broker: under-replicated and offline partitions, ISR shrink rate, request latency, disk and network. Producer: error and retry rate, batch size, request latency. Consumer: lag per partition, time since last poll, rebalance rate, commit failures. Alert on trends, not single spikes. Micrometer exposes the client metrics; see Spring Boot 4.1 Kafka producers and consumers.
Common interview questions
Follow-ups that come after the answers above:
“How would you guarantee no message is lost end to end?” (replication factor 3, acks=all, min.insync.replicas=2, commit after processing, idempotent handler)
“Exactly-once — really?” (inside Kafka yes; with the outside world only via idempotence)
“What happens to a consumer that is too slow?” (poll interval, rebalance, lag; question 19)
“How many partitions should I create?” (target throughput divided by per-partition throughput, plus headroom; remember you cannot shrink)
“Can two consumers in one group read the same partition?” (not in a classic group; yes in a share group)
Assuming global order. Kafka orders per partition only; unkeyed records are spread across partitions.
Auto-commit with slow processing. Offsets are committed for records that have not been handled yet, so a crash loses them.
Long handlers and max.poll.interval.ms. A batch that takes longer than the interval gets the consumer removed and starts a rebalance loop.
Forgetting read_committed. Transactions on the producer side do nothing for consumers that still use the default isolation level.
Treating exactly-once as end-to-end. Emails, REST calls and external databases need their own idempotence.
Retrying a poison pill forever. One bad record blocks the partition; use a dead-letter topic.
Adding partitions to a keyed topic without realising keys are remapped.
Relying on auto.offset.reset=latest for a new group and wondering why history is missing.
Should you even use Kafka? If you need a task queue for a few thousand jobs a day, a database table or RabbitMQ is simpler to run and reason about. Kafka earns its operational cost when you need high throughput, replay, or many independent consumers of the same events. Be ready to say which of those you actually have.
No Comments yet!