52 lines
2.5 KiB
Markdown
52 lines
2.5 KiB
Markdown
[← Serialisation](03-serialisation.md) · [Module README](../README.md) · [Consuming →](05-consuming.md)
|
|
|
|
# 4. Keys and partitions: the ordering guarantee in disguise
|
|
|
|
Kafka orders records **within a partition**. Not within a topic. So the key is not a label — it
|
|
decides which records are ordered with respect to each other, and it is the most consequential
|
|
line in a producer.
|
|
|
|
Same key, same partition, forever:
|
|
|
|
```
|
|
key partition murmur2 & 0x7fffffff % 3 Math.abs(murmur2) % 3
|
|
o-1 0 0 0
|
|
o-2 0 0 2
|
|
o-3 1 1 1
|
|
o-4 1 1 1
|
|
o-5 2 2 0
|
|
o-6 1 1 1
|
|
```
|
|
|
|
From [`docs/output/key-to-partition.txt`](output/key-to-partition.txt), produced against a real
|
|
broker by
|
|
[`KeysAndPartitionsTest`](../src/test/java/com/ankurm/kafkabasics/KeysAndPartitionsTest.java).
|
|
|
|
The default partitioner is `murmur2` of the **serialized key bytes**, masked positive, modulo
|
|
the partition count. Note the third and fourth columns: `& 0x7fffffff` and `Math.abs` disagree
|
|
on two of six keys, because clearing the sign bit is not the same number as negating it. If you
|
|
reimplement the partitioner to predict placement — for a test, for a migration, for a routing
|
|
table — `Math.abs` gives you the right answer about half the time, which is the worst available
|
|
failure mode.
|
|
|
|
## Three consequences
|
|
|
|
**A null key is not a key.** Records without one are spread across partitions by the sticky
|
|
partitioner, so nothing about their relative order is guaranteed. If two events describe the same
|
|
entity and you did not key them, they can be processed out of order by different consumers.
|
|
|
|
**Adding partitions repartitions every key.** The modulus changes, so `o-2` moves. Anything
|
|
relying on per-key ordering loses it across the resize for records still in flight. Pick the
|
|
partition count with room to grow; changing it later is a data-ordering event, not a capacity
|
|
knob.
|
|
|
|
**Key cardinality is your parallelism ceiling.** Keying by `customerId` when one customer is 40%
|
|
of traffic gives you a hot partition that no amount of consumer scaling fixes, because one
|
|
partition is consumed by exactly one member of a group.
|
|
|
|
Choosing the key is choosing what must stay ordered. Order events by `orderId` if operations on
|
|
one order must not overtake each other; by `customerId` if that is true across a customer's
|
|
orders. Those are different systems.
|
|
|
|
[Consuming →](05-consuming.md)
|