106 lines
5.1 KiB
Markdown
106 lines
5.1 KiB
Markdown
# 2. The anatomy of a partitioned step
|
|
|
|
[← Previous](01-the-problem-and-mental-model.md) | [README](../README.md) | [Next: What gridSize actually controls →](03-what-gridsize-actually-controls.md)
|
|
|
|
The whole manager/worker wiring is four beans in
|
|
[`BatchConfig`](../src/main/java/com/ankurm/batchpartition/config/BatchConfig.java):
|
|
|
|
```java
|
|
@Bean
|
|
public Partitioner partitioner() throws IOException {
|
|
var resolver = new PathMatchingResourcePatternResolver();
|
|
Resource[] resources = resolver.getResources("file:" + shardsDir + "/*.csv");
|
|
// ... sort, then:
|
|
var partitioner = new MultiResourcePartitioner();
|
|
partitioner.setResources(resources);
|
|
partitioner.setKeyName("fileName");
|
|
return partitioner;
|
|
}
|
|
|
|
@Bean
|
|
public Step managerStep(JobRepository jobRepository, Partitioner partitioner, Step workerStep,
|
|
TaskExecutor partitionTaskExecutor) {
|
|
return new StepBuilder("ordersManagerStep", jobRepository)
|
|
.partitioner("ordersWorkerStep", partitioner)
|
|
.step(workerStep)
|
|
.gridSize(gridSize)
|
|
.taskExecutor(partitionTaskExecutor)
|
|
.build();
|
|
}
|
|
```
|
|
|
|
Full source: [`BatchConfig.java`](../src/main/java/com/ankurm/batchpartition/config/BatchConfig.java).
|
|
`StepBuilder.partitioner(String, Partitioner)` returns a `PartitionStepBuilder`
|
|
(`org.springframework.batch.core.step.builder.PartitionStepBuilder`, confirmed by decompiling
|
|
`spring-batch-core-6.0.5.jar` — see [chapter 3](03-what-gridsize-actually-controls.md)),
|
|
which is where `.step(...)`, `.taskExecutor(...)` and `.gridSize(...)` live. The `"ordersWorkerStep"`
|
|
string is the base name every partition's `StepExecution` is built from — it is why the logs
|
|
in [`docs/output/05-happy-path-4-partitions.txt`](output/05-happy-path-4-partitions.txt) show step
|
|
names like `ordersWorkerStep:partition0`.
|
|
|
|
## Why there is no separately-coded "single-threaded baseline"
|
|
|
|
The single most load-bearing design decision in this module: `--partition.grid-size=1` against a
|
|
one-shard directory drives the *exact same* `partitioner()` / `workerStep()` / `managerStep()`
|
|
code as `--partition.grid-size=8` against an eight-shard directory. There is no `if` branch, no
|
|
separate `@Profile`, no hand-written "just call the reader directly" fallback. The only two things
|
|
that change between every run in
|
|
[`docs/output/09-full-scale-throughput.txt`](output/09-full-scale-throughput.txt) are the number of
|
|
shard files on disk and the `gridSize`/pool-size properties passed on the command line.
|
|
|
|
That matters because a partitioning benchmark that compares a hand-written single-threaded step
|
|
against a partitioned one is comparing two variables at once: thread count, AND whatever
|
|
incidental differences exist between the two code paths (a different reader class, a different
|
|
transaction boundary, different connection pool behavior). Isolating gridSize as the only variable
|
|
is what makes the numbers in [chapter 6](06-why-cpu-bound-not-io-bound.md) and
|
|
[chapter 10](10-scaling-sensitivity-to-data-size.md) mean what they claim to mean.
|
|
|
|
## The worker step is nothing special
|
|
|
|
```java
|
|
@Bean
|
|
public Step workerStep(JobRepository jobRepository, PlatformTransactionManager transactionManager,
|
|
FlatFileItemReader<Order> shardReader, ItemProcessor<Order, RiskScoredOrder> riskProcessor,
|
|
ItemWriter<RiskScoredOrder> riskWriter, PartitionStatsListener partitionStatsListener) {
|
|
return new StepBuilder("ordersWorkerStep", jobRepository)
|
|
.<Order, RiskScoredOrder>chunk(1000)
|
|
.transactionManager(transactionManager)
|
|
.reader(shardReader)
|
|
.processor(riskProcessor)
|
|
.writer(riskWriter)
|
|
.listener(partitionStatsListener)
|
|
.build();
|
|
}
|
|
```
|
|
|
|
Chunk size 1000, one processor ([chapter 6](06-why-cpu-bound-not-io-bound.md)), one writer
|
|
([chapter 4](04-the-writer-and-the-beanmapped-trap.md)). The only partition-aware piece is the
|
|
`@StepScope` reader, which late-binds to whichever file the manager step assigned it:
|
|
|
|
```java
|
|
@Bean
|
|
@StepScope
|
|
public FlatFileItemReader<Order> shardReader(@Value("#{stepExecutionContext['fileName']}") Resource shardFile) {
|
|
return new FlatFileItemReaderBuilder<Order>()
|
|
.name("shardReader")
|
|
.resource(shardFile)
|
|
// ...
|
|
.build();
|
|
}
|
|
```
|
|
|
|
`stepExecutionContext['fileName']` reads the exact key `MultiResourcePartitioner.setKeyName("fileName")`
|
|
wrote into each partition's `ExecutionContext` back in chapter 1's diagram. `@StepScope` is what
|
|
makes the late binding possible: this reader bean is not constructed until a specific
|
|
`StepExecution` (a specific partition) starts, at which point Spring resolves `#{...}` against
|
|
*that* execution's context, not some shared default.
|
|
|
|
## Going deeper
|
|
|
|
- `@StepScope` and late binding in general:
|
|
[Spring Batch reference — Late Binding](https://docs.spring.io/spring-batch/reference/step/late-binding.html) (`rel="nofollow"`).
|
|
- `PartitionStepBuilder`'s full decompiled surface:
|
|
[`docs/output/03-package-repackaging-javap.txt`](output/03-package-repackaging-javap.txt).
|
|
|
|
[Next: What gridSize actually controls →](03-what-gridsize-actually-controls.md)
|