asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity

Add spring-batch-partitioning: manager/worker partitioning, gridSize semantics, rejected-partition recovery, and a real 10M-row scaling sweep

Browse Source
This commit is contained in:
asmhatre
2026-09-14 09:36:35 +00:00
parent b81af72bc3
commit 34e9f5b243
40 changed files with 2071 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
spring-batch-partitioning/README.md
+73
View File
@@ -0,0 +1,73 @@
# spring-batch-partitioning
Companion code for **[Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job](https://ankurm.com/)**
on [ankurm.com](https://ankurm.com).
Verified against Spring Boot **4.1.1**, Spring Batch **6.0.5**, Spring Framework **7.0.9**, on
Temurin JDK **25.0.4.1+1**, on a 2-vCPU sandbox.
One job, `orderRiskJob`: partition a directory of pre-sharded order CSVs across worker threads,
score every order for risk with a deliberately CPU-bound processor, write the result to
`ORDER_RISK_SUMMARY`. There is no separately-coded single-threaded baseline — `--partition.grid-size=1`
against a one-shard directory runs the identical code path as any other grid size (see
[chapter 2](docs/02-anatomy-of-a-partitioned-step.md)), so every number below differs by exactly
one variable.
| Run | What it demonstrates | Docs |
|---|---|---|
| `--partition.grid-size=1` against a 1-shard dir | Single-threaded baseline, same code path | [ch. 1](docs/01-the-problem-and-mental-model.md), [ch. 2](docs/02-anatomy-of-a-partitioned-step.md) |
| `--partition.grid-size=2/4/8` against matching shard dirs | Partitioned scaling, and where it stops helping | [ch. 3](docs/03-what-gridsize-actually-controls.md), [ch. 10](docs/10-scaling-sensitivity-to-data-size.md) |
| `--spring.profiles.active=reject` | Undersized pool + `AbortPolicy`: partitions rejected, `StepExecution`s stuck at `STARTING` forever | [ch. 7](docs/07-the-rejectedexecutionexception.md) |
| `--spring.profiles.active=recover --recover.job-execution-id=N` | Spring Batch 6.0's `JobOperator#recover`, then a normal restart that reruns only the failed partitions | [ch. 8](docs/08-restart-reruns-only-the-failed-partition.md), [ch. 9](docs/09-jobexecutionalreadyrunning-and-recover.md) |
## Documentation chapters
1. [The problem, and the smallest correct mental model](docs/01-the-problem-and-mental-model.md)
2. [The anatomy of a partitioned step](docs/02-anatomy-of-a-partitioned-step.md)
3. [What gridSize actually controls](docs/03-what-gridsize-actually-controls.md)
4. [The writer, the beanMapped trap, and finding the partition's own name](docs/04-the-writer-and-the-beanmapped-trap.md)
5. [The diagnostic endpoint](docs/05-the-diagnostic-endpoint.md)
6. [Why this module's work is CPU-bound, not I/O-bound](docs/06-why-cpu-bound-not-io-bound.md)
7. [The failure that does not look like a failure: rejected partitions](docs/07-the-rejectedexecutionexception.md)
8. [Restart reruns only the failed partition — proved, not assumed](docs/08-restart-reruns-only-the-failed-partition.md)
9. [JobExecutionAlreadyRunningException, forever — and recover()](docs/09-jobexecutionalreadyrunning-and-recover.md)
10. [Scaling sensitivity to data size, and the honest ceiling](docs/10-scaling-sensitivity-to-data-size.md)
11. [Production checklist](docs/11-production-checklist.md)
## Captured output
Everything under [`docs/output/`](docs/output) was produced by a real run (or a real `mvn test`)
and is quoted verbatim in the article and the chapters above:
| File | What produced it |
|---|---|
| `01-processor-determinism.txt`, `02-gridsize-ignored.txt` | JUnit tests, via `mvn test` |
| `03-package-repackaging-javap.txt` | `javap` / `unzip -l` against the real 6.0.5 and 5.2.6 jars |
| `04-enforceuniquemethods-error.txt` | A real startup failure, first draft of `BatchConfig` |
| `05-happy-path-4-partitions.txt` | 4 shards, gridSize 4, plus the diagnostic endpoint |
| `06-rejected-partitions-stuck.txt`, `07-restart-throws-alreadyrunning.txt`, `08-recover-then-restart.txt` | The `reject` profile, a failed restart attempt, then the `recover` profile, all against the same H2 file across separate JVMs |
| `09-full-scale-throughput.txt` | The full grid-size sweep at 10,000,000 rows and at 300,000 rows |
## Running it
Needs a JDK 25 and Maven 3.9, plus Python 3 for the data generator.
```bash
export JAVA_HOME=/path/to/jdk-25
mvn -DskipTests package
python3 scripts/generate-shards.py ./data/shards 10000000 4 # 4 shard files, 2.5M rows each
java -jar target/spring-batch-partitioning-1.0.0.jar --partition.shards-dir=./data/shards --partition.grid-size=4
```
`GET http://localhost:8081/batch/partitions/{jobExecutionId}` (or `/batch/partitions/latest`)
shows which thread ran which partition, and for how long — see
[chapter 5](docs/05-the-diagnostic-endpoint.md).
`scripts/generate-shards.py <dir> <rows> <shards> [--corrupt-shard N] [--seed S]` produces the
sharded CSVs any of the above commands read; the same seed produces byte-identical row content
regardless of how many shards it is split into, which is what makes the grid-size comparisons in
[chapter 10](docs/10-scaling-sensitivity-to-data-size.md) apples-to-apples.
## Licence
MIT — see the repository [LICENSE](../LICENSE).