Add spring-batch-partitioning: manager/worker partitioning, gridSize semantics, rejected-partition recovery, and a real 10M-row scaling sweep

2026-09-14 09:36:35 +00:00
parent b81af72bc3
commit 34e9f5b243
40 changed files with 2071 additions and 0 deletions
@@ -0,0 +1,105 @@
+# 2. The anatomy of a partitioned step
+
+[&larr; Previous](01-the-problem-and-mental-model.md) | [README](../README.md) | [Next: What gridSize actually controls &rarr;](03-what-gridsize-actually-controls.md)
+
+The whole manager/worker wiring is four beans in
+[`BatchConfig`](../src/main/java/com/ankurm/batchpartition/config/BatchConfig.java):
+
+```java
+@Bean
+public Partitioner partitioner() throws IOException {
+    var resolver = new PathMatchingResourcePatternResolver();
+    Resource[] resources = resolver.getResources("file:" + shardsDir + "/*.csv");
+    // ... sort, then:
+    var partitioner = new MultiResourcePartitioner();
+    partitioner.setResources(resources);
+    partitioner.setKeyName("fileName");
+    return partitioner;
+}
+
+@Bean
+public Step managerStep(JobRepository jobRepository, Partitioner partitioner, Step workerStep,
+                         TaskExecutor partitionTaskExecutor) {
+    return new StepBuilder("ordersManagerStep", jobRepository)
+            .partitioner("ordersWorkerStep", partitioner)
+            .step(workerStep)
+            .gridSize(gridSize)
+            .taskExecutor(partitionTaskExecutor)
+            .build();
+}
+```
+
+Full source: [`BatchConfig.java`](../src/main/java/com/ankurm/batchpartition/config/BatchConfig.java).
+`StepBuilder.partitioner(String, Partitioner)` returns a `PartitionStepBuilder`
+(`org.springframework.batch.core.step.builder.PartitionStepBuilder`, confirmed by decompiling
+`spring-batch-core-6.0.5.jar` &mdash; see [chapter 3](03-what-gridsize-actually-controls.md)),
+which is where `.step(...)`, `.taskExecutor(...)` and `.gridSize(...)` live. The `"ordersWorkerStep"`
+string is the base name every partition's `StepExecution` is built from &mdash; it is why the logs
+in [`docs/output/05-happy-path-4-partitions.txt`](output/05-happy-path-4-partitions.txt) show step
+names like `ordersWorkerStep:partition0`.
+
+## Why there is no separately-coded "single-threaded baseline"
+
+The single most load-bearing design decision in this module: `--partition.grid-size=1` against a
+one-shard directory drives the *exact same* `partitioner()` / `workerStep()` / `managerStep()`
+code as `--partition.grid-size=8` against an eight-shard directory. There is no `if` branch, no
+separate `@Profile`, no hand-written "just call the reader directly" fallback. The only two things
+that change between every run in
+[`docs/output/09-full-scale-throughput.txt`](output/09-full-scale-throughput.txt) are the number of
+shard files on disk and the `gridSize`/pool-size properties passed on the command line.
+
+That matters because a partitioning benchmark that compares a hand-written single-threaded step
+against a partitioned one is comparing two variables at once: thread count, AND whatever
+incidental differences exist between the two code paths (a different reader class, a different
+transaction boundary, different connection pool behavior). Isolating gridSize as the only variable
+is what makes the numbers in [chapter 6](06-why-cpu-bound-not-io-bound.md) and
+[chapter 10](10-scaling-sensitivity-to-data-size.md) mean what they claim to mean.
+
+## The worker step is nothing special
+
+```java
+@Bean
+public Step workerStep(JobRepository jobRepository, PlatformTransactionManager transactionManager,
+                        FlatFileItemReader<Order> shardReader, ItemProcessor<Order, RiskScoredOrder> riskProcessor,
+                        ItemWriter<RiskScoredOrder> riskWriter, PartitionStatsListener partitionStatsListener) {
+    return new StepBuilder("ordersWorkerStep", jobRepository)
+            .<Order, RiskScoredOrder>chunk(1000)
+            .transactionManager(transactionManager)
+            .reader(shardReader)
+            .processor(riskProcessor)
+            .writer(riskWriter)
+            .listener(partitionStatsListener)
+            .build();
+}
+```
+
+Chunk size 1000, one processor ([chapter 6](06-why-cpu-bound-not-io-bound.md)), one writer
+([chapter 4](04-the-writer-and-the-beanmapped-trap.md)). The only partition-aware piece is the
+`@StepScope` reader, which late-binds to whichever file the manager step assigned it:
+
+```java
+@Bean
+@StepScope
+public FlatFileItemReader<Order> shardReader(@Value("#{stepExecutionContext['fileName']}") Resource shardFile) {
+    return new FlatFileItemReaderBuilder<Order>()
+            .name("shardReader")
+            .resource(shardFile)
+            // ...
+            .build();
+}
+```
+
+`stepExecutionContext['fileName']` reads the exact key `MultiResourcePartitioner.setKeyName("fileName")`
+wrote into each partition's `ExecutionContext` back in chapter 1's diagram. `@StepScope` is what
+makes the late binding possible: this reader bean is not constructed until a specific
+`StepExecution` (a specific partition) starts, at which point Spring resolves `#{...}` against
+*that* execution's context, not some shared default.
+
+## Going deeper
+
+- `@StepScope` and late binding in general:
+  [Spring Batch reference &mdash; Late Binding](https://docs.spring.io/spring-batch/reference/step/late-binding.html) (`rel="nofollow"`).
+- `PartitionStepBuilder`'s full decompiled surface:
+  [`docs/output/03-package-repackaging-javap.txt`](output/03-package-repackaging-javap.txt).
+
+[Next: What gridSize actually controls &rarr;](03-what-gridsize-actually-controls.md)