41 lines
2.7 KiB
Plaintext
41 lines
2.7 KiB
Plaintext
# 10,000,000 rows: manager-step wall time by partition count (2 vCPUs)
|
|
|
|
Same job, same code path, same data (deterministic seed 100) -- only partition.grid-size and
|
|
the shard directory (which controls how many resources MultiResourcePartitioner sees) change.
|
|
risk.iterations=150 (the default) for all four runs. Each number is the 'Step: [ordersManagerStep]
|
|
executed in ...' line from that run's own log -- wall time for the whole partitioned step,
|
|
including every worker partition and the manager step's own bookkeeping.
|
|
|
|
grid-size manager-step rows/sec speedup-vs-1
|
|
1 70.485s 141,874 1.00x
|
|
2 49.647s 201,422 1.42x
|
|
4 53.102s 188,317 1.33x
|
|
8 55.521s 180,112 1.27x
|
|
|
|
Best result at grid-size 2 -- matching this sandbox's 2 vCPUs exactly. Beyond that, wall time gets
|
|
WORSE with every doubling: grid-size 8 is slower than grid-size 4, which is slower than grid-size
|
|
2. More partitions past the physical core count does not sit still, it actively costs time --
|
|
context-switch and scheduling overhead with no additional CPU to absorb it. The speedup at
|
|
grid-size 2 (1.42x) is also well short of the 2x a naive "twice the cores" mental model predicts;
|
|
docs/06-why-cpu-bound-not-io-bound.md and docs/10-scaling-sensitivity-to-data-size.md discuss the
|
|
two candidate reasons this article checked (H2's single-writer MVStore, and fixed per-partition
|
|
startup cost) and what evidence separates them.
|
|
|
|
--- The same sweep at a smaller scale (300,000 rows) tells a different story ---
|
|
grid-size 1: 5.127s grid-size 2: 4.677s (1.10x) grid-size 4: 5.158s (0.99x) grid-size 8: 6.812s (0.75x)
|
|
|
|
At 300K rows, grid-size 8 is not just worse than grid-size 2 -- it is worse than NOT partitioning
|
|
at all. The fixed cost of standing up a partition (opening the shard file, acquiring a JDBC
|
|
connection, thread handoff) is the same few milliseconds whether a job processes 10,000,000 rows
|
|
or 300,000; at the smaller scale there is less real work to amortize it against, so
|
|
over-partitioning is a strictly worse mistake on a smaller job than on a larger one. Whether
|
|
partitioning helps at all is a function of BOTH core count and data volume, not core count alone.
|
|
|
|
--- A CPU-heavier variant (risk.iterations=5000, 300,000 rows) narrows the gap towards 2x ---
|
|
grid-size 1: 9.184s grid-size 2: 7.277s (1.26x)
|
|
|
|
Raising the per-item CPU cost pushed the grid-size-2 speedup from 1.10x to 1.26x at the same data
|
|
volume -- evidence, not proof, that at least part of the shortfall from a clean 2x is the shared
|
|
H2 writer, not thread overhead alone: more CPU-bound work per item dilutes the fixed write-lock
|
|
cost relative to total time, and the measured speedup moved in exactly that direction.
|