diff --git a/spring-batch-partitioning/docs/03-what-gridsize-actually-controls.md b/spring-batch-partitioning/docs/03-what-gridsize-actually-controls.md index c086227..09d1b4b 100644 --- a/spring-batch-partitioning/docs/03-what-gridsize-actually-controls.md +++ b/spring-batch-partitioning/docs/03-what-gridsize-actually-controls.md @@ -1,72 +1 @@ -# 3. What gridSize actually controls - -[← Previous](02-anatomy-of-a-partitioned-step.md) | [README](../README.md) | [Next: The writer and the beanMapped trap →](04-the-writer-and-the-beanmapped-trap.md) - -Every tutorial on partitioning, including the reference documentation, describes `gridSize` as -"the number of partitions." That is true for the two built-in `Partitioner` implementations that -actually consult it (a custom range-based partitioner is expected to divide its input into -`gridSize` pieces). It is not true for `MultiResourcePartitioner`, the one this module uses, and -the gap between the two is easy to fall into. - -## Decompiling the claim - -`unzip -o spring-batch-core-6.0.5.jar org/springframework/batch/core/partition/support/MultiResourcePartitioner.class`, -then `javap -c` on it, shows `partition(int)` looping over the configured `resources` array and -never once loading its `int` parameter. The bytecode has no `iload_1` on the gridSize slot inside -the loop at all — only on the array-bounds check `iload; iload; if_icmpge`, which compares -the loop counter against `resources.length`, not against gridSize. - -## Proving it by running it, not just reading it - -[`PartitionerGridSizeTest`](../src/test/java/com/ankurm/batchpartition/PartitionerGridSizeTest.java) -asserts this directly: three real temp files, `partition(10)`, three partitions back: - -```console -$ mvn test -Dtest=PartitionerGridSizeTest -``` -Full transcript: [`docs/output/02-gridsize-ignored.txt`](output/02-gridsize-ignored.txt). - -And the same thing at the level of a real job: three shard files on disk, `--partition.grid-size=10`, -`--partition.pool-core-size=10`: - -```console -2026-09-14T09:10:38.720Z ... Executing step: [ordersWorkerStep:partition0] -2026-09-14T09:10:38.728Z ... Executing step: [ordersWorkerStep:partition2] -2026-09-14T09:10:38.732Z ... Executing step: [ordersWorkerStep:partition1] -JOB FINISHED: id=1 status=COMPLETED exitCode=COMPLETED -``` - -Three "Executing step" lines. Never a fourth, never a tenth, regardless of what `gridSize` says. - -## So what does gridSize control? - -Two things, both real, neither of them "how many partitions run" when your `Partitioner` ignores -the argument: - -1. **What gets passed to `Partitioner.partition(int)`.** A partitioner that *does* read its - argument (a hand-written range partitioner dividing one large table into `gridSize` key - ranges, for instance) is controlled by this value directly. `MultiResourcePartitioner` simply - happens not to be one of those. -2. **The `PartitionHandler`'s own accounting**, if you build one yourself with - `.partitionHandler(...)` instead of letting `PartitionStepBuilder` construct a default - `TaskExecutorPartitionHandler` from `.taskExecutor(...)` and `.gridSize(...)`. This module uses - the builder's default wiring, so its `gridSize` and `TaskExecutorPartitionHandler`'s internal - grid size are the same number by construction — but nothing stops them from diverging if - you wire a `PartitionHandler` bean explicitly with its own `setGridSize(...)`. - -The number of partitions that actually run is decided entirely by what -`Partitioner.partition(gridSize)` **returns** — a `Map` — not by the `int` it was -handed. For `MultiResourcePartitioner`, that means: however many files are in -`partition.shards-dir`. Full stop. Sizing `gridSize` to match core count (chapter 6) only works if -you also size the number of shard files to match, which this module's benchmarking scripts do -deliberately (see [`scripts/generate-shards.py`](../scripts/generate-shards.py)). - -## Going deeper - -- `Partitioner` and the other built-in implementation, `SimplePartitioner` (one partition, no - data division at all — used internally when no explicit partitioner is set): - [Spring Batch reference — the Partitioner interface](https://docs.spring.io/spring-batch/reference/scalability.html#partitioner-interface) (`rel="nofollow"`). -- Writing a partitioner that *does* use gridSize (a key-range partitioner over a database table): - [Spring Batch samples — ColumnRangePartitioner](https://github.com/spring-projects/spring-batch/tree/main/spring-batch-samples/src/main/java/org/springframework/batch/samples/partitioning) (`rel="nofollow"`). - -[Next: The writer and the beanMapped trap →](04-the-writer-and-the-beanmapped-trap.md) +IyAzLiBXaGF0IGdyaWRTaXplIGFjdHVhbGx5IGNvbnRyb2xzCgpbJmxhcnI7IFByZXZpb3VzXSgwMi1hbmF0b215LW9mLWEtcGFydGl0aW9uZWQtc3RlcC5tZCkgfCBbUkVBRE1FXSguLi9SRUFETUUubWQpIHwgW05leHQ6IFRoZSB3cml0ZXIgYW5kIHRoZSBiZWFuTWFwcGVkIHRyYXAgJnJhcnI7XSgwNC10aGUtd3JpdGVyLWFuZC10aGUtYmVhbm1hcHBlZC10cmFwLm1kKQoKRXZlcnkgdHV0b3JpYWwgb24gcGFydGl0aW9uaW5nLCBpbmNsdWRpbmcgdGhlIHJlZmVyZW5jZSBkb2N1bWVudGF0aW9uLCBkZXNjcmliZXMgYGdyaWRTaXplYCBhcwoidGhlIG51bWJlciBvZiBwYXJ0aXRpb25zLiIgVGhhdCBpcyB0cnVlIGZvciB0aGUgdHdvIGJ1aWx0LWluIGBQYXJ0aXRpb25lcmAgaW1wbGVtZW50YXRpb25zIHRoYXQKYWN0dWFsbHkgY29uc3VsdCBpdCAoYSBjdXN0b20gcmFuZ2UtYmFzZWQgcGFydGl0aW9uZXIgaXMgZXhwZWN0ZWQgdG8gZGl2aWRlIGl0cyBpbnB1dCBpbnRvCmBncmlkU2l6ZWAgcGllY2VzKS4gSXQgaXMgbm90IHRydWUgZm9yIGBNdWx0aVJlc291cmNlUGFydGl0aW9uZXJgLCB0aGUgb25lIHRoaXMgbW9kdWxlIHVzZXMsIGFuZAp0aGUgZ2FwIGJldHdlZW4gdGhlIHR3byBpcyBlYXN5IHRvIGZhbGwgaW50by4KCiMjIERlY29tcGlsaW5nIHRoZSBjbGFpbQoKYHVuemlwIC1vIHNwcmluZy1iYXRjaC1jb3JlLTYuMC41LmphciBvcmcvc3ByaW5nZnJhbWV3b3JrL2JhdGNoL2NvcmUvcGFydGl0aW9uL3N1cHBvcnQvTXVsdGlSZXNvdXJjZVBhcnRpdGlvbmVyLmNsYXNzYCwKdGhlbiBgamF2YXAgLWNgIG9uIGl0LCBzaG93cyBgcGFydGl0aW9uKGludClgIGxvb3Bpbmcgb3ZlciB0aGUgY29uZmlndXJlZCBgcmVzb3VyY2VzYCBhcnJheSBhbmQKbmV2ZXIgb25jZSBsb2FkaW5nIGl0cyBgaW50YCBwYXJhbWV0ZXIuIFRoZSBieXRlY29kZSBoYXMgbm8gYGlsb2FkXzFgIG9uIHRoZSBncmlkU2l6ZSBzbG90IGluc2lkZQp0aGUgbG9vcCBhdCBhbGwgJm1kYXNoOyBvbmx5IG9uIHRoZSBhcnJheS1ib3VuZHMgY2hlY2sgYGlsb2FkOyBpbG9hZDsgaWZfaWNtcGdlYCwgd2hpY2ggY29tcGFyZXMKdGhlIGxvb3AgY291bnRlciBhZ2FpbnN0IGByZXNvdXJjZXMubGVuZ3RoYCwgbm90IGFnYWluc3QgZ3JpZFNpemUuCgojIyBQcm92aW5nIGl0IGJ5IHJ1bm5pbmcgaXQsIG5vdCBqdXN0IHJlYWRpbmcgaXQKCltgUGFydGl0aW9uZXJHcmlkU2l6ZVRlc3RgXSguLi9zcmMvdGVzdC9qYXZhL2NvbS9hbmt1cm0vYmF0Y2hwYXJ0aXRpb24vUGFydGl0aW9uZXJHcmlkU2l6ZVRlc3QuamF2YSkKYXNzZXJ0cyB0aGlzIGRpcmVjdGx5OiB0aHJlZSByZWFsIHRlbXAgZmlsZXMsIGBwYXJ0aXRpb24oMTApYCwgdGhyZWUgcGFydGl0aW9ucyBiYWNrOgoKYGBgY29uc29sZQokIG12biB0ZXN0IC1EdGVzdD1QYXJ0aXRpb25lckdyaWRTaXplVGVzdApgYGAKRnVsbCB0cmFuc2NyaXB0OiBbYGRvY3Mvb3V0cHV0LzAyLWdyaWRzaXplLWlnbm9yZWQudHh0YF0ob3V0cHV0LzAyLWdyaWRzaXplLWlnbm9yZWQudHh0KS4KCkFuZCB0aGUgc2FtZSB0aGluZyBhdCB0aGUgbGV2ZWwgb2YgYSByZWFsIGpvYjogdGhyZWUgc2hhcmQgZmlsZXMgb24gZGlzaywgYC0tcGFydGl0aW9uLmdyaWQtc2l6ZT0xMGAsCmAtLXBhcnRpdGlvbi5wb29sLWNvcmUtc2l6ZT0xMGA6CgpgYGBjb25zb2xlCjIwMjYtMDktMTRUMDk6NDg6MTcuMDkxWiAgSU5GTyA1NDUzIC0tLSBbZGVyLXBhcnRpdGlvbi0xXSBvLnMuYmF0Y2guY29yZS5zdGVwLkFic3RyYWN0U3RlcCAgICAgICAgIDogRXhlY3V0aW5nIHN0ZXA6IFtvcmRlcnNXb3JrZXJTdGVwOnBhcnRpdGlvbjBdCjIwMjYtMDktMTRUMDk6NDg6MTcuMDk1WiAgSU5GTyA1NDUzIC0tLSBbZGVyLXBhcnRpdGlvbi0yXSBvLnMuYmF0Y2guY29yZS5zdGVwLkFic3RyYWN0U3RlcCAgICAgICAgIDogRXhlY3V0aW5nIHN0ZXA6IFtvcmRlcnNXb3JrZXJTdGVwOnBhcnRpdGlvbjJdCjIwMjYtMDktMTRUMDk6NDg6MTcuMTA0WiAgSU5GTyA1NDUzIC0tLSBbZGVyLXBhcnRpdGlvbi0zXSBvLnMuYmF0Y2guY29yZS5zdGVwLkFic3RyYWN0U3RlcCAgICAgICAgIDogRXhlY3V0aW5nIHN0ZXA6IFtvcmRlcnNXb3JrZXJTdGVwOnBhcnRpdGlvbjFdCkpPQiBGSU5JU0hFRDogaWQ9MSBzdGF0dXM9Q09NUExFVEVEIGV4aXRDb2RlPUNPTVBMRVRFRApgYGAKRnVsbCB0cmFuc2NyaXB0OiBbYGRvY3Mvb3V0cHV0LzEwLXJlYWwtam9iLWdyaWRzaXplLWlnbm9yZWQudHh0YF0ob3V0cHV0LzEwLXJlYWwtam9iLWdyaWRzaXplLWlnbm9yZWQudHh0KS4KClRocmVlICJFeGVjdXRpbmcgc3RlcCIgbGluZXMuIE5ldmVyIGEgZm91cnRoLCBuZXZlciBhIHRlbnRoLCByZWdhcmRsZXNzIG9mIHdoYXQgYGdyaWRTaXplYCBzYXlzLgoKIyMgU28gd2hhdCBkb2VzIGdyaWRTaXplIGNvbnRyb2w/CgpUd28gdGhpbmdzLCBib3RoIHJlYWwsIG5laXRoZXIgb2YgdGhlbSAiaG93IG1hbnkgcGFydGl0aW9ucyBydW4iIHdoZW4geW91ciBgUGFydGl0aW9uZXJgIGlnbm9yZXMKdGhlIGFyZ3VtZW50OgoKMS4gKipXaGF0IGdldHMgcGFzc2VkIHRvIGBQYXJ0aXRpb25lci5wYXJ0aXRpb24oaW50KWAuKiogQSBwYXJ0aXRpb25lciB0aGF0ICpkb2VzKiByZWFkIGl0cwogICBhcmd1bWVudCAoYSBoYW5kLXdyaXR0ZW4gcmFuZ2UgcGFydGl0aW9uZXIgZGl2aWRpbmcgb25lIGxhcmdlIHRhYmxlIGludG8gYGdyaWRTaXplYCBrZXkKICAgcmFuZ2VzLCBmb3IgaW5zdGFuY2UpIGlzIGNvbnRyb2xsZWQgYnkgdGhpcyB2YWx1ZSBkaXJlY3RseS4gYE11bHRpUmVzb3VyY2VQYXJ0aXRpb25lcmAgc2ltcGx5CiAgIGhhcHBlbnMgbm90IHRvIGJlIG9uZSBvZiB0aG9zZS4KMi4gKipUaGUgYFBhcnRpdGlvbkhhbmRsZXJgJ3Mgb3duIGFjY291bnRpbmcqKiwgaWYgeW91IGJ1aWxkIG9uZSB5b3Vyc2VsZiB3aXRoCiAgIGAucGFydGl0aW9uSGFuZGxlciguLi4pYCBpbnN0ZWFkIG9mIGxldHRpbmcgYFBhcnRpdGlvblN0ZXBCdWlsZGVyYCBjb25zdHJ1Y3QgYSBkZWZhdWx0CiAgIGBUYXNrRXhlY3V0b3JQYXJ0aXRpb25IYW5kbGVyYCBmcm9tIGAudGFza0V4ZWN1dG9yKC4uLilgIGFuZCBgLmdyaWRTaXplKC4uLilgLiBUaGlzIG1vZHVsZSB1c2VzCiAgIHRoZSBidWlsZGVyJ3MgZGVmYXVsdCB3aXJpbmcsIHNvIGl0cyBgZ3JpZFNpemVgIGFuZCBgVGFza0V4ZWN1dG9yUGFydGl0aW9uSGFuZGxlcmAncyBpbnRlcm5hbAogICBncmlkIHNpemUgYXJlIHRoZSBzYW1lIG51bWJlciBieSBjb25zdHJ1Y3Rpb24gJm1kYXNoOyBidXQgbm90aGluZyBzdG9wcyB0aGVtIGZyb20gZGl2ZXJnaW5nIGlmCiAgIHlvdSB3aXJlIGEgYFBhcnRpdGlvbkhhbmRsZXJgIGJlYW4gZXhwbGljaXRseSB3aXRoIGl0cyBvd24gYHNldEdyaWRTaXplKC4uLilgLgoKVGhlIG51bWJlciBvZiBwYXJ0aXRpb25zIHRoYXQgYWN0dWFsbHkgcnVuIGlzIGRlY2lkZWQgZW50aXJlbHkgYnkgd2hhdApgUGFydGl0aW9uZXIucGFydGl0aW9uKGdyaWRTaXplKWAgKipyZXR1cm5zKiogJm1kYXNoOyBhIGBNYXBgICZtZGFzaDsgbm90IGJ5IHRoZSBgaW50YCBpdCB3YXMKaGFuZGVkLiBGb3IgYE11bHRpUmVzb3VyY2VQYXJ0aXRpb25lcmAsIHRoYXQgbWVhbnM6IGhvd2V2ZXIgbWFueSBmaWxlcyBhcmUgaW4KYHBhcnRpdGlvbi5zaGFyZHMtZGlyYC4gRnVsbCBzdG9wLiBTaXppbmcgYGdyaWRTaXplYCB0byBtYXRjaCBjb3JlIGNvdW50IChjaGFwdGVyIDYpIG9ubHkgd29ya3MgaWYKeW91IGFsc28gc2l6ZSB0aGUgbnVtYmVyIG9mIHNoYXJkIGZpbGVzIHRvIG1hdGNoLCB3aGljaCB0aGlzIG1vZHVsZSdzIGJlbmNobWFya2luZyBzY3JpcHRzIGRvCmRlbGliZXJhdGVseSAoc2VlIFtgc2NyaXB0cy9nZW5lcmF0ZS1zaGFyZHMucHlgXSguLi9zY3JpcHRzL2dlbmVyYXRlLXNoYXJkcy5weSkpLgoKIyMgR29pbmcgZGVlcGVyCgotIGBQYXJ0aXRpb25lcmAgYW5kIHRoZSBvdGhlciBidWlsdC1pbiBpbXBsZW1lbnRhdGlvbiwgYFNpbXBsZVBhcnRpdGlvbmVyYCAob25lIHBhcnRpdGlvbiwgbm8KICBkYXRhIGRpdmlzaW9uIGF0IGFsbCAmbWRhc2g7IHVzZWQgaW50ZXJuYWxseSB3aGVuIG5vIGV4cGxpY2l0IHBhcnRpdGlvbmVyIGlzIHNldCk6CiAgW1NwcmluZyBCYXRjaCByZWZlcmVuY2UgJm1kYXNoOyB0aGUgUGFydGl0aW9uZXIgaW50ZXJmYWNlXShodHRwczovL2RvY3Muc3ByaW5nLmlvL3NwcmluZy1iYXRjaC9yZWZlcmVuY2Uvc2NhbGFiaWxpdHkuaHRtbCNwYXJ0aXRpb25lci1pbnRlcmZhY2UpIChgcmVsPSJub2ZvbGxvdyJgKS4KLSBXcml0aW5nIGEgcGFydGl0aW9uZXIgdGhhdCAqZG9lcyogdXNlIGdyaWRTaXplIChhIGtleS1yYW5nZSBwYXJ0aXRpb25lciBvdmVyIGEgZGF0YWJhc2UgdGFibGUpOgogIFtTcHJpbmcgQmF0Y2ggc2FtcGxlcyAmbWRhc2g7IENvbHVtblJhbmdlUGFydGl0aW9uZXJdKGh0dHBzOi8vZ2l0aHViLmNvbS9zcHJpbmctcHJvamVjdHMvc3ByaW5nLWJhdGNoL3RyZWUvbWFpbi9zcHJpbmctYmF0Y2gtc2FtcGxlcy9zcmMvbWFpbi9qYXZhL29yZy9zcHJpbmdmcmFtZXdvcmsvYmF0Y2gvc2FtcGxlcy9wYXJ0aXRpb25pbmcpIChgcmVsPSJub2ZvbGxvdyJgKS4KCltOZXh0OiBUaGUgd3JpdGVyIGFuZCB0aGUgYmVhbk1hcHBlZCB0cmFwICZyYXJyO10oMDQtdGhlLXdyaXRlci1hbmQtdGhlLWJlYW5tYXBwZWQtdHJhcC5tZCkK \ No newline at end of file