spring-messaging-demo/kafka-basics/docs/04-keys-and-partitions.md

← Serialisation · Module README · Consuming →

4. Keys and partitions: the ordering guarantee in disguise

Kafka orders records within a partition. Not within a topic. So the key is not a label — it decides which records are ordered with respect to each other, and it is the most consequential line in a producer.

Same key, same partition, forever:

key    partition   murmur2 & 0x7fffffff % 3   Math.abs(murmur2) % 3
o-1    0           0                          0
o-2    0           0                          2
o-3    1           1                          1
o-4    1           1                          1
o-5    2           2                          0
o-6    1           1                          1

From docs/output/key-to-partition.txt, produced against a real broker by KeysAndPartitionsTest.

The default partitioner is murmur2 of the serialized key bytes, masked positive, modulo the partition count. Note the third and fourth columns: & 0x7fffffff and Math.abs disagree on two of six keys, because clearing the sign bit is not the same number as negating it. If you reimplement the partitioner to predict placement — for a test, for a migration, for a routing table — Math.abs gives you the right answer about half the time, which is the worst available failure mode.

Three consequences

A null key is not a key. Records without one are spread across partitions by the sticky partitioner, so nothing about their relative order is guaranteed. If two events describe the same entity and you did not key them, they can be processed out of order by different consumers.

Adding partitions repartitions every key. The modulus changes, so o-2 moves. Anything relying on per-key ordering loses it across the resize for records still in flight. Pick the partition count with room to grow; changing it later is a data-ordering event, not a capacity knob.

Key cardinality is your parallelism ceiling. Keying by customerId when one customer is 40% of traffic gives you a hot partition that no amount of consumer scaling fixes, because one partition is consumed by exactly one member of a group.

Choosing the key is choosing what must stay ordered. Order events by orderId if operations on one order must not overtake each other; by customerId if that is true across a customer's orders. Those are different systems.

Consuming →