2.5 KiB
← Serialisation · Module README · Consuming →
4. Keys and partitions: the ordering guarantee in disguise
Kafka orders records within a partition. Not within a topic. So the key is not a label — it decides which records are ordered with respect to each other, and it is the most consequential line in a producer.
Same key, same partition, forever:
key partition murmur2 & 0x7fffffff % 3 Math.abs(murmur2) % 3
o-1 0 0 0
o-2 0 0 2
o-3 1 1 1
o-4 1 1 1
o-5 2 2 0
o-6 1 1 1
From docs/output/key-to-partition.txt, produced against a real
broker by
KeysAndPartitionsTest.
The default partitioner is murmur2 of the serialized key bytes, masked positive, modulo
the partition count. Note the third and fourth columns: & 0x7fffffff and Math.abs disagree
on two of six keys, because clearing the sign bit is not the same number as negating it. If you
reimplement the partitioner to predict placement — for a test, for a migration, for a routing
table — Math.abs gives you the right answer about half the time, which is the worst available
failure mode.
Three consequences
A null key is not a key. Records without one are spread across partitions by the sticky partitioner, so nothing about their relative order is guaranteed. If two events describe the same entity and you did not key them, they can be processed out of order by different consumers.
Adding partitions repartitions every key. The modulus changes, so o-2 moves. Anything
relying on per-key ordering loses it across the resize for records still in flight. Pick the
partition count with room to grow; changing it later is a data-ordering event, not a capacity
knob.
Key cardinality is your parallelism ceiling. Keying by customerId when one customer is 40%
of traffic gives you a hot partition that no amount of consumer scaling fixes, because one
partition is consumed by exactly one member of a group.
Choosing the key is choosing what must stay ordered. Order events by orderId if operations on
one order must not overtake each other; by customerId if that is true across a customer's
orders. Those are different systems.