5.1 KiB
2. The anatomy of a partitioned step
← Previous | README | Next: What gridSize actually controls →
The whole manager/worker wiring is four beans in
BatchConfig:
@Bean
public Partitioner partitioner() throws IOException {
var resolver = new PathMatchingResourcePatternResolver();
Resource[] resources = resolver.getResources("file:" + shardsDir + "/*.csv");
// ... sort, then:
var partitioner = new MultiResourcePartitioner();
partitioner.setResources(resources);
partitioner.setKeyName("fileName");
return partitioner;
}
@Bean
public Step managerStep(JobRepository jobRepository, Partitioner partitioner, Step workerStep,
TaskExecutor partitionTaskExecutor) {
return new StepBuilder("ordersManagerStep", jobRepository)
.partitioner("ordersWorkerStep", partitioner)
.step(workerStep)
.gridSize(gridSize)
.taskExecutor(partitionTaskExecutor)
.build();
}
Full source: BatchConfig.java.
StepBuilder.partitioner(String, Partitioner) returns a PartitionStepBuilder
(org.springframework.batch.core.step.builder.PartitionStepBuilder, confirmed by decompiling
spring-batch-core-6.0.5.jar — see chapter 3),
which is where .step(...), .taskExecutor(...) and .gridSize(...) live. The "ordersWorkerStep"
string is the base name every partition's StepExecution is built from — it is why the logs
in docs/output/05-happy-path-4-partitions.txt show step
names like ordersWorkerStep:partition0.
Why there is no separately-coded "single-threaded baseline"
The single most load-bearing design decision in this module: --partition.grid-size=1 against a
one-shard directory drives the exact same partitioner() / workerStep() / managerStep()
code as --partition.grid-size=8 against an eight-shard directory. There is no if branch, no
separate @Profile, no hand-written "just call the reader directly" fallback. The only two things
that change between every run in
docs/output/09-full-scale-throughput.txt are the number of
shard files on disk and the gridSize/pool-size properties passed on the command line.
That matters because a partitioning benchmark that compares a hand-written single-threaded step against a partitioned one is comparing two variables at once: thread count, AND whatever incidental differences exist between the two code paths (a different reader class, a different transaction boundary, different connection pool behavior). Isolating gridSize as the only variable is what makes the numbers in chapter 6 and chapter 10 mean what they claim to mean.
The worker step is nothing special
@Bean
public Step workerStep(JobRepository jobRepository, PlatformTransactionManager transactionManager,
FlatFileItemReader<Order> shardReader, ItemProcessor<Order, RiskScoredOrder> riskProcessor,
ItemWriter<RiskScoredOrder> riskWriter, PartitionStatsListener partitionStatsListener) {
return new StepBuilder("ordersWorkerStep", jobRepository)
.<Order, RiskScoredOrder>chunk(1000)
.transactionManager(transactionManager)
.reader(shardReader)
.processor(riskProcessor)
.writer(riskWriter)
.listener(partitionStatsListener)
.build();
}
Chunk size 1000, one processor (chapter 6), one writer
(chapter 4). The only partition-aware piece is the
@StepScope reader, which late-binds to whichever file the manager step assigned it:
@Bean
@StepScope
public FlatFileItemReader<Order> shardReader(@Value("#{stepExecutionContext['fileName']}") Resource shardFile) {
return new FlatFileItemReaderBuilder<Order>()
.name("shardReader")
.resource(shardFile)
// ...
.build();
}
stepExecutionContext['fileName'] reads the exact key MultiResourcePartitioner.setKeyName("fileName")
wrote into each partition's ExecutionContext back in chapter 1's diagram. @StepScope is what
makes the late binding possible: this reader bean is not constructed until a specific
StepExecution (a specific partition) starts, at which point Spring resolves #{...} against
that execution's context, not some shared default.
Going deeper
@StepScopeand late binding in general: Spring Batch reference — Late Binding (rel="nofollow").PartitionStepBuilder's full decompiled surface:docs/output/03-package-repackaging-javap.txt.