spring-boot-demo/spring-batch-partitioning

spring-batch-partitioning

Companion code for Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job on ankurm.com.

Verified against Spring Boot 4.1.1, Spring Batch 6.0.5, Spring Framework 7.0.9, on Temurin JDK 25.0.4.1+1, on a 2-vCPU sandbox.

One job, orderRiskJob: partition a directory of pre-sharded order CSVs across worker threads, score every order for risk with a deliberately CPU-bound processor, write the result to ORDER_RISK_SUMMARY. There is no separately-coded single-threaded baseline — --partition.grid-size=1 against a one-shard directory runs the identical code path as any other grid size (see chapter 2), so every number below differs by exactly one variable.

Run	What it demonstrates	Docs
--partition.grid-size=1 against a 1-shard dir	Single-threaded baseline, same code path	ch. 1, ch. 2
--partition.grid-size=2/4/8 against matching shard dirs	Partitioned scaling, and where it stops helping	ch. 3, ch. 10
--spring.profiles.active=reject	Undersized pool + AbortPolicy: partitions rejected, StepExecutions stuck at STARTING forever	ch. 7
--spring.profiles.active=recover --recover.job-execution-id=N	Spring Batch 6.0's JobOperator#recover, then a normal restart that reruns only the failed partitions	ch. 8, ch. 9

Documentation chapters

1. The problem, and the smallest correct mental model
2. The anatomy of a partitioned step
3. What gridSize actually controls
4. The writer, the beanMapped trap, and finding the partition's own name
5. The diagnostic endpoint
6. Why this module's work is CPU-bound, not I/O-bound
7. The failure that does not look like a failure: rejected partitions
8. Restart reruns only the failed partition — proved, not assumed
9. JobExecutionAlreadyRunningException, forever — and recover()
10. Scaling sensitivity to data size, and the honest ceiling
11. Production checklist

Captured output

Everything under docs/output/ was produced by a real run (or a real mvn test) and is quoted verbatim in the article and the chapters above:

File	What produced it
01-processor-determinism.txt, 02-gridsize-ignored.txt	JUnit tests, via mvn test
03-package-repackaging-javap.txt	javap / unzip -l against the real 6.0.5 and 5.2.6 jars
04-enforceuniquemethods-error.txt	A real startup failure, first draft of BatchConfig
05-happy-path-4-partitions.txt	4 shards, gridSize 4, plus the diagnostic endpoint
06-rejected-partitions-stuck.txt, 07-restart-throws-alreadyrunning.txt, 08-recover-then-restart.txt	The reject profile, a failed restart attempt, then the recover profile, all against the same H2 file across separate JVMs
09-full-scale-throughput.txt	The full grid-size sweep at 10,000,000 rows and at 300,000 rows

Running it

Needs a JDK 25 and Maven 3.9, plus Python 3 for the data generator.

export JAVA_HOME=/path/to/jdk-25
mvn -DskipTests package
python3 scripts/generate-shards.py ./data/shards 10000000 4   # 4 shard files, 2.5M rows each
java -jar target/spring-batch-partitioning-1.0.0.jar --partition.shards-dir=./data/shards --partition.grid-size=4

GET http://localhost:8081/batch/partitions/{jobExecutionId} (or /batch/partitions/latest) shows which thread ran which partition, and for how long — see chapter 5.

scripts/generate-shards.py <dir> <rows> <shards> [--corrupt-shard N] [--seed S] produces the sharded CSVs any of the above commands read; the same seed produces byte-identical row content regardless of how many shards it is split into, which is what makes the grid-size comparisons in chapter 10 apples-to-apples.

Licence

MIT — see the repository LICENSE.