73 lines
4.3 KiB
Markdown
73 lines
4.3 KiB
Markdown
# 3. What gridSize actually controls
|
|
|
|
[← Previous](02-anatomy-of-a-partitioned-step.md) | [README](../README.md) | [Next: The writer and the beanMapped trap →](04-the-writer-and-the-beanmapped-trap.md)
|
|
|
|
Every tutorial on partitioning, including the reference documentation, describes `gridSize` as
|
|
"the number of partitions." That is true for the two built-in `Partitioner` implementations that
|
|
actually consult it (a custom range-based partitioner is expected to divide its input into
|
|
`gridSize` pieces). It is not true for `MultiResourcePartitioner`, the one this module uses, and
|
|
the gap between the two is easy to fall into.
|
|
|
|
## Decompiling the claim
|
|
|
|
`unzip -o spring-batch-core-6.0.5.jar org/springframework/batch/core/partition/support/MultiResourcePartitioner.class`,
|
|
then `javap -c` on it, shows `partition(int)` looping over the configured `resources` array and
|
|
never once loading its `int` parameter. The bytecode has no `iload_1` on the gridSize slot inside
|
|
the loop at all — only on the array-bounds check `iload; iload; if_icmpge`, which compares
|
|
the loop counter against `resources.length`, not against gridSize.
|
|
|
|
## Proving it by running it, not just reading it
|
|
|
|
[`PartitionerGridSizeTest`](../src/test/java/com/ankurm/batchpartition/PartitionerGridSizeTest.java)
|
|
asserts this directly: three real temp files, `partition(10)`, three partitions back:
|
|
|
|
```console
|
|
$ mvn test -Dtest=PartitionerGridSizeTest
|
|
```
|
|
Full transcript: [`docs/output/02-gridsize-ignored.txt`](output/02-gridsize-ignored.txt).
|
|
|
|
And the same thing at the level of a real job: three shard files on disk, `--partition.grid-size=10`,
|
|
`--partition.pool-core-size=10`:
|
|
|
|
```console
|
|
2026-09-14T09:10:38.720Z ... Executing step: [ordersWorkerStep:partition0]
|
|
2026-09-14T09:10:38.728Z ... Executing step: [ordersWorkerStep:partition2]
|
|
2026-09-14T09:10:38.732Z ... Executing step: [ordersWorkerStep:partition1]
|
|
JOB FINISHED: id=1 status=COMPLETED exitCode=COMPLETED
|
|
```
|
|
|
|
Three "Executing step" lines. Never a fourth, never a tenth, regardless of what `gridSize` says.
|
|
|
|
## So what does gridSize control?
|
|
|
|
Two things, both real, neither of them "how many partitions run" when your `Partitioner` ignores
|
|
the argument:
|
|
|
|
1. **What gets passed to `Partitioner.partition(int)`.** A partitioner that *does* read its
|
|
argument (a hand-written range partitioner dividing one large table into `gridSize` key
|
|
ranges, for instance) is controlled by this value directly. `MultiResourcePartitioner` simply
|
|
happens not to be one of those.
|
|
2. **The `PartitionHandler`'s own accounting**, if you build one yourself with
|
|
`.partitionHandler(...)` instead of letting `PartitionStepBuilder` construct a default
|
|
`TaskExecutorPartitionHandler` from `.taskExecutor(...)` and `.gridSize(...)`. This module uses
|
|
the builder's default wiring, so its `gridSize` and `TaskExecutorPartitionHandler`'s internal
|
|
grid size are the same number by construction — but nothing stops them from diverging if
|
|
you wire a `PartitionHandler` bean explicitly with its own `setGridSize(...)`.
|
|
|
|
The number of partitions that actually run is decided entirely by what
|
|
`Partitioner.partition(gridSize)` **returns** — a `Map` — not by the `int` it was
|
|
handed. For `MultiResourcePartitioner`, that means: however many files are in
|
|
`partition.shards-dir`. Full stop. Sizing `gridSize` to match core count (chapter 6) only works if
|
|
you also size the number of shard files to match, which this module's benchmarking scripts do
|
|
deliberately (see [`scripts/generate-shards.py`](../scripts/generate-shards.py)).
|
|
|
|
## Going deeper
|
|
|
|
- `Partitioner` and the other built-in implementation, `SimplePartitioner` (one partition, no
|
|
data division at all — used internally when no explicit partitioner is set):
|
|
[Spring Batch reference — the Partitioner interface](https://docs.spring.io/spring-batch/reference/scalability.html#partitioner-interface) (`rel="nofollow"`).
|
|
- Writing a partitioner that *does* use gridSize (a key-range partitioner over a database table):
|
|
[Spring Batch samples — ColumnRangePartitioner](https://github.com/spring-projects/spring-batch/tree/main/spring-batch-samples/src/main/java/org/springframework/batch/samples/partitioning) (`rel="nofollow"`).
|
|
|
|
[Next: The writer and the beanMapped trap →](04-the-writer-and-the-beanmapped-trap.md)
|