5.0 KiB
10. Scaling sensitivity to data size, and the honest ceiling
← Previous | README | Next: Production checklist →
The full-scale numbers
Same job, same code path (chapter 2), same deterministically
seeded 10,000,000-row dataset, only partition.grid-size and the shard directory changing, on this
module's 2-vCPU sandbox:
| grid-size | manager-step wall time | rows/sec | speedup vs. grid-size 1 |
|---|---|---|---|
| 1 | 70.485s | 141,874 | 1.00x |
| 2 | 49.647s | 201,422 | 1.42x (best) |
| 4 | 53.102s | 188,317 | 1.33x |
| 8 | 55.521s | 180,112 | 1.27x |
Full transcript: docs/output/09-full-scale-throughput.txt.
Best result at grid-size 2 — exactly the physical core count. Past that, wall time gets
worse with every doubling: more partitions than cores does not sit still, it costs real time in
scheduling and context-switch overhead with no additional compute to absorb it.
The same sweep at 300,000 rows tells a different story
| grid-size | wall time | speedup |
|---|---|---|
| 1 | 5.127s | 1.00x |
| 2 | 4.677s | 1.10x |
| 4 | 5.158s | 0.99x |
| 8 | 6.812s | 0.75x (worse than not partitioning at all) |
At the smaller volume, grid-size 8 does not just lose to grid-size 2 — it loses to grid-size
- The fixed cost of standing up one partition (opening its shard file, acquiring a JDBC connection from the pool, a thread handoff) is roughly the same fixed number of milliseconds whichever scale the job runs at. At 10,000,000 rows there is enough real work per partition to amortize that fixed cost into irrelevance; at 300,000 rows split eight ways (37,500 rows each) there is not. Over-partitioning is a strictly worse mistake on a smaller job than on a larger one — the "right" gridSize is a function of data volume as well as core count, not core count alone, and a gridSize tuned against a large nightly batch can be actively harmful applied unchanged to a smaller one.
The two candidate explanations for the sub-2x ceiling at the best setting
Even at the best-measured setting (grid-size 2 on 2 cores), speedup tops out at 1.42x, not 2x. Chapter 6 found one piece of evidence pointing at the shared H2 file database as a contributing serialization point: raising per-item CPU cost (5000 iterations instead of 150, at the 300K scale) moved the grid-size-2 speedup from 1.10x to 1.26x, consistent with diluting a fixed write-lock cost across more total work — but this module did not isolate the writer completely (per-partition tables, or an in-memory sink, compared directly), so treat that as evidence pointing in a direction, not a closed investigation. A production job choosing a real target database (one built for concurrent writers, not a single embedded MVStore file) would need to re-measure this, not assume the same ceiling applies.
Going deeper
- H2's MVStore concurrency model (single writer per store):
H2 MVStore documentation (
rel="nofollow"). - The CPU-vs-I/O-bound design choice this chapter's numbers depend on: chapter 6.