Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job
A partitioned Spring Batch step scales a 10-million-row job across worker threads — until gridSize doesn’t mean what you think, a rejected partition gets stuck forever, and the real speedup tops out well short of 2x.
A chunk-oriented Spring Batch step is already fast — it reads an item, processes it, and touches the database once per chunk instead of once per row. It is also, by default, exactly one thread. Point that one thread at 10 million rows of real per-row work and it will finish correctly: restartably, with every fault-tolerance guarantee chunk-oriented processing gives you, using one CPU core for as long as that takes. On the two-vCPU sandbox this article’s companion project actually ran on, that was 70.5 seconds. On a bigger job, or a slower per-row computation, “one thread” stops being an implementation detail and starts being the reason the batch window doesn’t close in time.
Partitioning is Spring Batch’s answer: take that same step and run several copies of it in parallel, each against its own slice of the input, without giving up restartability. This article builds one, watches it fail in a way that does not look like failure, and measures exactly how much scaling it actually delivers — not the number a “twice the cores, twice the speed” slide promises.
Verified against: Spring Boot 4.1.1 (GA 20 Aug 2026), Spring Batch 6.0.5, Spring Framework 7.0.9, on Temurin JDK 25.0.4.1+1. Spring Batch 6.0.0 itself went GA on 19 Nov 2025. Every number in this article came from a real run of the companion project on a 2-vCPU sandbox — see the versions table and full transcripts in spring-batch-partitioning/.
This continues the chunk-oriented processing built in Spring Batch on Boot 4.1: Steps, Chunks and Restartability — read that first if terms like chunk, ItemReader, or restart-by-ExecutionContext are new. This article assumes them and adds exactly one new mechanism: more than one worker step running at once.
The smallest correct mental model: one manager, several identical workers
Partitioning does not change chunk-oriented processing at all. It takes the step you already know — reader, processor, writer, its own transaction, its own restart bookkeeping — and runs several copies of it side by side, each pointed at a different slice of the input. A new step type, the manager step, sits above them and does no item processing of its own: its entire job is to describe the slices, hand each one to a worker thread, wait, and roll the results up into one outcome.
The diagram is the whole mental model: a Partitioner never touches a row of data. It only produces descriptions of work — here, one CSV file path per partition, via Spring Batch’s built-in MultiResourcePartitioner. Reading, scoring, and writing all happen inside the worker step, run four separate times on four separate threads. Everything downstream of that diagram — whether four threads actually beats one, what happens when a thread can’t be found for a partition, how a failed partition restarts — is this article.
Four CSV files, each 5,000 orders, sitting in one directory. The manager/worker wiring is four beans:
@Bean
public Partitioner partitioner() throws IOException {
var resolver = new PathMatchingResourcePatternResolver();
Resource[] resources = resolver.getResources("file:" + shardsDir + "/*.csv");
// ... sort by filename, then:
var partitioner = new MultiResourcePartitioner();
partitioner.setResources(resources);
partitioner.setKeyName("fileName");
return partitioner;
}
@Bean
public Step managerStep(JobRepository jobRepository, Partitioner partitioner, Step workerStep,
TaskExecutor partitionTaskExecutor) {
return new StepBuilder("ordersManagerStep", jobRepository)
.partitioner("ordersWorkerStep", partitioner)
.step(workerStep)
.gridSize(gridSize)
.taskExecutor(partitionTaskExecutor)
.build();
}
Full source: BatchConfig.java. The worker step is nothing new — a chunk size of 1,000, a reader, a CPU-bound risk-scoring processor, a writer — except that its reader is @StepScope and late-binds to whichever file the manager assigned it, in the same file:
@Bean
@StepScope
public FlatFileItemReader<Order> shardReader(@Value("#{stepExecutionContext['fileName']}") Resource shardFile) {
return new FlatFileItemReaderBuilder<Order>()
.name("shardReader")
.resource(shardFile)
// ...
.build();
}
Same source as above: BatchConfig.java, the shardReader bean.
Run it, and four things happen within milliseconds of each other, on four distinct OS threads:
Full transcript, including a live query against a diagnostic endpoint this module adds specifically to make thread assignment observable rather than assumed: docs/output/05-happy-path-4-partitions.txt.
Configuration says four partitions should run; it does not say four partitions did run. This project’s PartitionStatsListener writes the actual thread name, start time, and duration for every partition to a table, and a small REST endpoint (/batch/partitions/{jobExecutionId}) exposes it. Delete both before shipping — they exist to make this article’s claims checkable, not because a production job should expose its own thread names over HTTP.
For readers who want the whole SPI — PartitionHandler, the built-in TaskExecutorPartitionHandler, wiring a custom remote handler for cross-machine partitioning — the manager/worker construction is walked bean-by-bean in chapter 2, and the diagnostic endpoint’s implementation is chapter 5.
The full worker-step wiring, including the writer: BatchConfig.java.
Every tutorial on partitioning, including the reference documentation, calls gridSize “the number of partitions.” Point that assumption at this module and it breaks immediately: set --partition.grid-size=10 against a directory holding three shard files, and exactly three worker steps execute. Not ten.
Full transcript: docs/output/10-real-job-gridsize-ignored.txt. Three “Executing step” lines, never a fourth, regardless of what gridSize says. The reason is in the partitioner this module uses, MultiResourcePartitioner: decompiling spring-batch-core-6.0.5.jar with javap -c shows its partition(int) method looping over its configured resources array and never once reading the int argument it was handed. A small unit test pins this directly — three real temp files, partition(10), three partitions back, never ten:
Full source, including why the test uses real FileSystemResource temp files rather than ByteArrayResource (the method also calls resource.getURL() on each one, which a byte array cannot satisfy): PartitionerGridSizeTest.java. Real output:
The number of partitions is decided by what Partitioner.partition(gridSize) returns — a Map — never by the int it was handed. A hand-written partitioner that divides a key range into gridSize pieces genuinely is controlled by this value. MultiResourcePartitioner simply isn’t one of those: for it, the partition count is however many files sit in partition.shards-dir, full stop. Size your shard files to your core count, not just the property.
For readers wiring their own custom partitioner rather than this module’s file-based one, this is the exact distinction to check first, and it is walked in full — including the second built-in implementation, SimplePartitioner, which always returns exactly one partition — in chapter 3.
The full chapter, with the decompiled bytecode: chapter 3.
The failure that doesn’t look like a failure
Here is the failure mode worth losing the most time to, because nothing about it announces itself as a failure. TaskExecutorPartitionHandler submits every partition’s worker step to its TaskExecutor up front, then waits for all of them. Undersize that executor — a pool of one thread, no queue, four shard files — and three of those four submissions get rejected before a single row of their input is ever read.
@Bean
@Profile("reject")
public TaskExecutor partitionTaskExecutorRejecting(@Value("${partition.reject.pool-size:1}") int smallPoolSize) {
var executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(smallPoolSize);
executor.setMaxPoolSize(smallPoolSize);
executor.setQueueCapacity(0);
executor.setThreadNamePrefix("order-partition-");
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
executor.initialize();
return executor;
}
Full source: BatchConfig.java. Run it against four shards with this executor active:
2026-09-14T09:11:21.375Z INFO 3074 --- [der-partition-1] o.s.batch.core.step.AbstractStep : Executing step: [ordersWorkerStep:partition1]
2026-09-14T09:11:21.946Z INFO 3074 --- [der-partition-1] o.s.batch.core.step.AbstractStep : Step: [ordersWorkerStep:partition1] executed in 570ms
org.springframework.batch.core.job.JobExecutionException: Partition handler returned an unsuccessful step
2026-09-14T09:11:21.961Z INFO 3074 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ordersManagerStep] executed in 601ms
JOB FINISHED: id=1 status=FAILED exitCode=FAILED
Full transcript: docs/output/06-rejected-partitions-stuck.txt. Only partition1 ever logs “Executing step.” The other three were rejected on submission, and the exception message never says so — no TaskRejectedException, no mention of threads or pools, anywhere in the log. TaskExecutorPartitionHandler catches whatever the executor throws on submission and folds it straight into the corresponding StepExecution‘s failure list instead of printing it. The only way to see what actually happened is to query the job repository’s own tables directly:
Same transcript as above: docs/output/06-rejected-partitions-stuck.txt. The manager step and the job both reach a clean FAILED. The three rejected worker steps do not — they are parked at STARTING/EXECUTING permanently, because the executor rejected them before Spring Batch’s own bookkeeping around that StepExecution ever began, so nothing in the job’s lifecycle ever transitions them again.
The diagram is the whole shape of this bug: a clean top-level failure sitting on top of three orphaned child rows that nothing will ever touch again on their own.
The rejection handler is the whole difference. Spring’s own default, CallerRunsPolicy, would have run the “extra” partitions on the submitting thread one at a time instead of rejecting them — slower, but never stuck. AbortPolicy at zero queue capacity is what turns an undersized pool from a performance problem into a permanently stuck job. Choose it deliberately, if at all, and only with an operational plan for what comes next — which is the rest of this section.
An ordinary failed step is exactly what restart exists for. This is not an ordinary failed step, and treating it as one is where the next hour goes.
ThreadPoolTaskExecutor‘s rejection handlers (AbortPolicy, CallerRunsPolicy, DiscardPolicy, DiscardOldestPolicy) and how differently each one fails: ThreadPoolTaskExecutor Javadoc.
Sizing a pool correctly relative to gridSize in the first place: chapter 3.
The restart that never comes
The instinct after a failed job is to restart it. Try that against the stuck job from the previous section, with the executor sized correctly this time:
Caused by: org.springframework.batch.core.launch.JobExecutionAlreadyRunningException: A job execution for this job is already running: JobExecution: id=1, version=3, startTime=2026-09-14T09:11:21.331474360, endTime=2026-09-14T09:11:21.965090481, lastUpdated=2026-09-14T09:11:21.966521177, status=FAILED, exitStatus=exitCode=FAILED;exitDescription=org.springframework.batch.core.job.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.PartitionStep.doExecute(PartitionStep.java:134)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:251)
Full transcript: docs/output/07-restart-throws-alreadyrunning.txt. Read that message closely: it says status=FAILED in the very same sentence that claims the execution is “already running.” That is not a contradiction in the exception, it’s a clue to what SimpleJobOperator actually checks — not “is the JobExecution status FAILED,” but something closer to “does this job instance have any StepExecution that has not reached a terminal status.” The three worker steps orphaned at STARTING/EXECUTING in the previous section satisfy that condition indefinitely.
There is no number of retries that fixes this on its own. Every subsequent attempt against the same identifying job parameters throws the identical exception. The job is not failed. It is stuck — a genuinely different state from an ordinary failure, and one that needed a mechanism that, before Spring Batch 6.0, didn’t exist.
The stuck state this restart attempt walked straight into: the previous section, and chapter 7.
What finally gets a job out of this state: the next section.
Spring Batch 6.0’s fix
Spring Batch 6.0 added exactly the operation this situation calls for: JobOperator.recover(JobExecution). A small runner, active under its own profile and ordered to run before the normal launch logic, fetches the stuck execution and calls it:
Full transcript: docs/output/08-recover-then-restart.txt. recover() did exactly one thing: it force-closed the three steps still at STARTING to FAILED. The already-COMPLETEDpartition1 was left untouched. Immediately afterward, in the same JVM, an ordinary restart against the same shard directory succeeds — a brand-new JobExecution, id 33.
What that restart actually re-ran, rather than what the reference docs merely promise it re-ran, is checkable against the same job repository:
Full transcript: docs/output/08-recover-then-restart.txt. Execution 33 has exactly three new worker rows — partition3, partition2, partition0, the ones recover() had just marked FAILED. There is no new row for partition1: it stayed COMPLETED from execution 1, at read count 5000, and the manager step’s own read count for execution 33 is 15000 — the sum of the three re-run partitions, not all four.
This works because partition names are stable across attempts: MultiResourcePartitioner names them partition0, partition1, and so on in the order its (explicitly sorted) resources array iterates, and restart resolves each partition’s step by that stable name against the same job instance — the identical mechanism ordinary, unpartitioned restart uses, applied once per partition rather than once per job.
A RejectedExecutionException is a different failure class from an ordinary step failure, and needs a different remedy. Sizing the pool correctly in the first place avoids the situation entirely. recover() is what a production runbook reaches for once a job has already gotten into it — call it before anyone attempts a restart, not after the restart has already thrown.
The full proof, with every BATCH_STEP_EXECUTION row: chapter 8 and chapter 9.
What partitioning actually buys you here
Everything above was about correctness under failure. This section is the number a “twice the cores, twice the speed” slide promises, measured rather than assumed, on the same 10,000,000-row dataset and the same 2-vCPU sandbox this whole module ran on:
grid-size
manager-step wall time
rows/sec
speedup vs. grid-size 1
1
70.485s
141,874
1.00x
2
49.647s
201,422
1.42x (best)
4
53.102s
188,317
1.33x
8
55.521s
180,112
1.27x
Full transcript: docs/output/09-full-scale-throughput.txt. Best result at grid-size 2 — exactly this sandbox’s physical core count. Past that, wall time gets worse with every doubling: more partitions than cores does not sit still, it actively costs scheduling and context-switch time with no additional compute to absorb it.
The same sweep at 300,000 rows instead of 10,000,000 tells a different story:
grid-size
wall time
speedup
1
5.127s
1.00x
2
4.677s
1.10x
4
5.158s
0.99x
8
6.812s
0.75x (worse than not partitioning at all)
Two lines, same shape at first, then diverging: at the smaller volume, grid-size 8 doesn’t just lose to grid-size 2, it loses to grid-size 1. The fixed cost of standing up one partition — opening its shard file, acquiring a JDBC connection, a thread handoff — is roughly the same number of milliseconds regardless of scale. At 10,000,000 rows there is enough real work per partition to make that cost irrelevant; at 300,000 rows split eight ways there isn’t.
Over-partitioning is a strictly worse mistake on a smaller job than on a larger one. The right gridSize is a function of data volume as well as core count, not core count alone — a value tuned against a large nightly batch can be actively harmful applied unchanged to a smaller one.
The full discussion, with the honest ceiling spelled out: chapter 10.
Why the speedup falls short of 2x
Even at the best-measured setting — grid-size 2, on 2 cores — speedup topped out at 1.42x, not 2x. Part of the answer is a deliberate design choice in this module’s own processor:
long acc = order.orderId() * 2654435761L + order.customerId();
for (int i = 0; i < iterations; i++) {
acc = (acc ^ (acc >>> 13)) * 2246822519L;
acc = (acc ^ (acc >>> 15)) * 3266489917L;
acc = acc ^ (acc >>> 16);
}
int score = (int) Math.floorMod(acc, 1000);
Full source: RiskScoringProcessor.java. No network call, no sleep — every millisecond this scorer spends is spent computing, on whichever core its thread lands on. That’s deliberate: an I/O-bound demo step would show a speedup even on a single core, because threads would spend their time blocked rather than competing for CPU, which is a real and useful effect but a different claim than “this uses more of the machine’s compute.” It’s also deterministic, which the article’s restart sections depend on:
Full transcript: docs/output/01-processor-determinism.txt. A partition that fails and re-runs has to produce the same score the second time, or a restart would silently change results depending on which attempt happened to write — the writer’s own idempotency (next section) only handles the “don’t duplicate the row” half of that guarantee.
A deliberately CPU-bound step still doesn’t fully explain a 1.42x ceiling on 2 cores. Raising the per-item iteration count 33x at a fixed data volume moved the grid-size-2 speedup from 1.10x to 1.26x — evidence, not proof, that at least part of the shortfall is time spent somewhere that does not scale with thread count. This module’s leading candidate is the shared H2 file all four worker threads write to: H2’s MVStore engine serializes writers against a single store, so however many threads are scoring orders in parallel, they still queue up one at a time to actually persist the result. This module didn’t isolate the writer completely enough to call that a closed case — a fair critique to have ready if you use this number yourself.
Amdahl’s law, for the general shape of “some fraction of the work cannot be parallelized”: Wikipedia — Amdahl’s law.
Two small decisions in the writer are worth calling out on their own, because both are traps a beginner walks into by doing the obviously convenient thing.
Order and RiskScoredOrder are Java records. The earlier, non-partitioned Spring Batch article already found once that JdbcBatchItemWriterBuilder.beanMapped() binds SQL parameters through BeanPropertySqlParameterSource, which looks for JavaBean-style getters — getOrderId() — and a record’s accessor is orderId(), no get prefix. beanMapped() against a record doesn’t throw. It silently writes every column NULL, because a missing getter reads as an absent property rather than an error. This module’s writer uses an explicit itemPreparedStatementSetter instead, on purpose, to avoid rediscovering that exact bug a second time:
@Bean
@StepScope
public ItemWriter<RiskScoredOrder> riskWriter(JdbcTemplate jdbcTemplate,
@Value("#{stepExecution.stepName}") String partitionName) {
return new JdbcBatchItemWriterBuilder<RiskScoredOrder>()
.dataSource(jdbcTemplate.getDataSource())
.sql("MERGE INTO ORDER_RISK_SUMMARY (order_id, customer_id, amount_cents, region, risk_score, " +
"high_risk, partition_name) KEY(order_id) VALUES (?, ?, ?, ?, ?, ?, ?)")
.itemPreparedStatementSetter((item, ps) -> {
ps.setLong(1, item.order().orderId());
// ... one setter per column, all seven
ps.setString(7, partitionName);
})
.assertUpdates(true)
.build();
}
Full source: BatchConfig.java. MERGE ... KEY(order_id), H2’s upsert syntax, rather than a plain INSERT, is the second deliberate choice: it makes re-running a single partition idempotent, which matters the moment a partition fails after writing some rows and then restarts under the mechanism from two sections ago.
The second trap is the one worth reading closely: populating partition_name — so a reader, or a bug report, can tell which worker wrote which row — needs the current step’s own name from inside a step-scoped bean. The first attempt reached for StepSynchronizationManager.getContext().getStepExecution().getStepName(), guessed at org.springframework.batch.core.step.StepSynchronizationManager by analogy with StepExecution living in core.step. It doesn’t compile: that package doesn’t contain that class in 6.0.5. javap against the real jar found where it actually moved to — org.springframework.batch.core.scope.context.StepSynchronizationManager, alongside JobScope and StepScope‘s own machinery, a more sensible home for it in hindsight; the guess just followed the wrong analogy.
The fix that shipped is simpler than getting the import right would have been.@Value("#{stepExecution.stepName}") late-binds the step name directly — the same late-binding mechanism the shard-file reader already uses — and needs no lookup class or package to get wrong in the first place.
Both traps share a shape worth generalizing: a Spring Batch API that fails silently (a record with the wrong accessor shape) or an import that fails loudly but for the wrong reason (a class that genuinely moved between major versions) are both cheaper to hit in a sandbox than in a nightly job.
BeanPropertySqlParameterSource, and why it fails silently rather than throwing: Spring Framework Javadoc.
Should you even do this? If your step is already fast enough within its batch window, no — partitioning trades a simple, single-threaded mental model for a genuinely harder one (gridSize that doesn’t mean what it says, a stuck-forever failure mode with its own remedy, a shared writer that caps your ceiling), and that trade should be made because you measured a real bottleneck, not because parallel sounds faster. If a single-threaded chunk-oriented step genuinely can’t fit your window, partitioning is the right tool and this module’s numbers are a fair expectation to set — a real but well-short-of-linear speedup, sensitive to both your core count and your data volume, that costs you a pool to size correctly and a writer to make idempotent. Measure your own job at your own data volume before picking a gridSize; the 300K-row sweep above is the whole argument for why “the same gridSize that worked at 10M rows” is not a safe assumption.
No Comments yet!