diff --git a/spring-batch-partitioning/README.md b/spring-batch-partitioning/README.md index f3ea07e..8a35fd5 100644 --- a/spring-batch-partitioning/README.md +++ b/spring-batch-partitioning/README.md @@ -1,73 +1 @@ -# spring-batch-partitioning - -Companion code for **[Spring Batch Partitioning and Parallel Steps: Scaling a 10-Million-Row Job](https://ankurm.com/)** -on [ankurm.com](https://ankurm.com). - -Verified against Spring Boot **4.1.1**, Spring Batch **6.0.5**, Spring Framework **7.0.9**, on -Temurin JDK **25.0.4.1+1**, on a 2-vCPU sandbox. - -One job, `orderRiskJob`: partition a directory of pre-sharded order CSVs across worker threads, -score every order for risk with a deliberately CPU-bound processor, write the result to -`ORDER_RISK_SUMMARY`. There is no separately-coded single-threaded baseline — `--partition.grid-size=1` -against a one-shard directory runs the identical code path as any other grid size (see -[chapter 2](docs/02-anatomy-of-a-partitioned-step.md)), so every number below differs by exactly -one variable. - -| Run | What it demonstrates | Docs | -|---|---|---| -| `--partition.grid-size=1` against a 1-shard dir | Single-threaded baseline, same code path | [ch. 1](docs/01-the-problem-and-mental-model.md), [ch. 2](docs/02-anatomy-of-a-partitioned-step.md) | -| `--partition.grid-size=2/4/8` against matching shard dirs | Partitioned scaling, and where it stops helping | [ch. 3](docs/03-what-gridsize-actually-controls.md), [ch. 10](docs/10-scaling-sensitivity-to-data-size.md) | -| `--spring.profiles.active=reject` | Undersized pool + `AbortPolicy`: partitions rejected, `StepExecution`s stuck at `STARTING` forever | [ch. 7](docs/07-the-rejectedexecutionexception.md) | -| `--spring.profiles.active=recover --recover.job-execution-id=N` | Spring Batch 6.0's `JobOperator#recover`, then a normal restart that reruns only the failed partitions | [ch. 8](docs/08-restart-reruns-only-the-failed-partition.md), [ch. 9](docs/09-jobexecutionalreadyrunning-and-recover.md) | - -## Documentation chapters - -1. [The problem, and the smallest correct mental model](docs/01-the-problem-and-mental-model.md) -2. [The anatomy of a partitioned step](docs/02-anatomy-of-a-partitioned-step.md) -3. [What gridSize actually controls](docs/03-what-gridsize-actually-controls.md) -4. [The writer, the beanMapped trap, and finding the partition's own name](docs/04-the-writer-and-the-beanmapped-trap.md) -5. [The diagnostic endpoint](docs/05-the-diagnostic-endpoint.md) -6. [Why this module's work is CPU-bound, not I/O-bound](docs/06-why-cpu-bound-not-io-bound.md) -7. [The failure that does not look like a failure: rejected partitions](docs/07-the-rejectedexecutionexception.md) -8. [Restart reruns only the failed partition — proved, not assumed](docs/08-restart-reruns-only-the-failed-partition.md) -9. [JobExecutionAlreadyRunningException, forever — and recover()](docs/09-jobexecutionalreadyrunning-and-recover.md) -10. [Scaling sensitivity to data size, and the honest ceiling](docs/10-scaling-sensitivity-to-data-size.md) -11. [Production checklist](docs/11-production-checklist.md) - -## Captured output - -Everything under [`docs/output/`](docs/output) was produced by a real run (or a real `mvn test`) -and is quoted verbatim in the article and the chapters above: - -| File | What produced it | -|---|---| -| `01-processor-determinism.txt`, `02-gridsize-ignored.txt` | JUnit tests, via `mvn test` | -| `03-package-repackaging-javap.txt` | `javap` / `unzip -l` against the real 6.0.5 and 5.2.6 jars | -| `04-enforceuniquemethods-error.txt` | A real startup failure, first draft of `BatchConfig` | -| `05-happy-path-4-partitions.txt` | 4 shards, gridSize 4, plus the diagnostic endpoint | -| `06-rejected-partitions-stuck.txt`, `07-restart-throws-alreadyrunning.txt`, `08-recover-then-restart.txt` | The `reject` profile, a failed restart attempt, then the `recover` profile, all against the same H2 file across separate JVMs | -| `09-full-scale-throughput.txt` | The full grid-size sweep at 10,000,000 rows and at 300,000 rows | - -## Running it - -Needs a JDK 25 and Maven 3.9, plus Python 3 for the data generator. - -```bash -export JAVA_HOME=/path/to/jdk-25 -mvn -DskipTests package -python3 scripts/generate-shards.py ./data/shards 10000000 4 # 4 shard files, 2.5M rows each -java -jar target/spring-batch-partitioning-1.0.0.jar --partition.shards-dir=./data/shards --partition.grid-size=4 -``` - -`GET http://localhost:8081/batch/partitions/{jobExecutionId}` (or `/batch/partitions/latest`) -shows which thread ran which partition, and for how long — see -[chapter 5](docs/05-the-diagnostic-endpoint.md). - -`scripts/generate-shards.py [--corrupt-shard N] [--seed S]` produces the -sharded CSVs any of the above commands read; the same seed produces byte-identical row content -regardless of how many shards it is split into, which is what makes the grid-size comparisons in -[chapter 10](docs/10-scaling-sensitivity-to-data-size.md) apples-to-apples. - -## Licence - -MIT — see the repository [LICENSE](../LICENSE). +IyBzcHJpbmctYmF0Y2gtcGFydGl0aW9uaW5nCgpDb21wYW5pb24gY29kZSBmb3IgKipbU3ByaW5nIEJhdGNoIFBhcnRpdGlvbmluZyBhbmQgUGFyYWxsZWwgU3RlcHM6IFNjYWxpbmcgYSAxMC1NaWxsaW9uLVJvdyBKb2JdKGh0dHBzOi8vYW5rdXJtLmNvbS8pKioKb24gW2Fua3VybS5jb21dKGh0dHBzOi8vYW5rdXJtLmNvbSkuCgpWZXJpZmllZCBhZ2FpbnN0IFNwcmluZyBCb290ICoqNC4xLjEqKiwgU3ByaW5nIEJhdGNoICoqNi4wLjUqKiwgU3ByaW5nIEZyYW1ld29yayAqKjcuMC45KiosIG9uClRlbXVyaW4gSkRLICoqMjUuMC40LjErMSoqLCBvbiBhIDItdkNQVSBzYW5kYm94LgoKT25lIGpvYiwgYG9yZGVyUmlza0pvYmA6IHBhcnRpdGlvbiBhIGRpcmVjdG9yeSBvZiBwcmUtc2hhcmRlZCBvcmRlciBDU1ZzIGFjcm9zcyB3b3JrZXIgdGhyZWFkcywKc2NvcmUgZXZlcnkgb3JkZXIgZm9yIHJpc2sgd2l0aCBhIGRlbGliZXJhdGVseSBDUFUtYm91bmQgcHJvY2Vzc29yLCB3cml0ZSB0aGUgcmVzdWx0IHRvCmBPUkRFUl9SSVNLX1NVTU1BUllgLiBUaGVyZSBpcyBubyBzZXBhcmF0ZWx5LWNvZGVkIHNpbmdsZS10aHJlYWRlZCBiYXNlbGluZSAmbWRhc2g7IGAtLXBhcnRpdGlvbi5ncmlkLXNpemU9MWAKYWdhaW5zdCBhIG9uZS1zaGFyZCBkaXJlY3RvcnkgcnVucyB0aGUgaWRlbnRpY2FsIGNvZGUgcGF0aCBhcyBhbnkgb3RoZXIgZ3JpZCBzaXplIChzZWUKW2NoYXB0ZXIgMl0oZG9jcy8wMi1hbmF0b215LW9mLWEtcGFydGl0aW9uZWQtc3RlcC5tZCkpLCBzbyBldmVyeSBudW1iZXIgYmVsb3cgZGlmZmVycyBieSBleGFjdGx5Cm9uZSB2YXJpYWJsZS4KCnwgUnVuIHwgV2hhdCBpdCBkZW1vbnN0cmF0ZXMgfCBEb2NzIHwKfC0tLXwtLS18LS0tfAp8IGAtLXBhcnRpdGlvbi5ncmlkLXNpemU9MWAgYWdhaW5zdCBhIDEtc2hhcmQgZGlyIHwgU2luZ2xlLXRocmVhZGVkIGJhc2VsaW5lLCBzYW1lIGNvZGUgcGF0aCB8IFtjaC4gMV0oZG9jcy8wMS10aGUtcHJvYmxlbS1hbmQtbWVudGFsLW1vZGVsLm1kKSwgW2NoLiAyXShkb2NzLzAyLWFuYXRvbXktb2YtYS1wYXJ0aXRpb25lZC1zdGVwLm1kKSB8CnwgYC0tcGFydGl0aW9uLmdyaWQtc2l6ZT0yLzQvOGAgYWdhaW5zdCBtYXRjaGluZyBzaGFyZCBkaXJzIHwgUGFydGl0aW9uZWQgc2NhbGluZywgYW5kIHdoZXJlIGl0IHN0b3BzIGhlbHBpbmcgfCBbY2guIDNdKGRvY3MvMDMtd2hhdC1ncmlkc2l6ZS1hY3R1YWxseS1jb250cm9scy5tZCksIFtjaC4gMTBdKGRvY3MvMTAtc2NhbGluZy1zZW5zaXRpdml0eS10by1kYXRhLXNpemUubWQpIHwKfCBgLS1zcHJpbmcucHJvZmlsZXMuYWN0aXZlPXJlamVjdGAgfCBVbmRlcnNpemVkIHBvb2wgKyBgQWJvcnRQb2xpY3lgOiBwYXJ0aXRpb25zIHJlamVjdGVkLCBgU3RlcEV4ZWN1dGlvbmBzIHN0dWNrIGF0IGBTVEFSVElOR2AgZm9yZXZlciB8IFtjaC4gN10oZG9jcy8wNy10aGUtcmVqZWN0ZWRleGVjdXRpb25leGNlcHRpb24ubWQpIHwKfCBgLS1zcHJpbmcucHJvZmlsZXMuYWN0aXZlPXJlY292ZXIgLS1yZWNvdmVyLmpvYi1leGVjdXRpb24taWQ9TmAgfCBTcHJpbmcgQmF0Y2ggNi4wJ3MgYEpvYk9wZXJhdG9yI3JlY292ZXJgLCB0aGVuIGEgbm9ybWFsIHJlc3RhcnQgdGhhdCByZXJ1bnMgb25seSB0aGUgZmFpbGVkIHBhcnRpdGlvbnMgfCBbY2guIDhdKGRvY3MvMDgtcmVzdGFydC1yZXJ1bnMtb25seS10aGUtZmFpbGVkLXBhcnRpdGlvbi5tZCksIFtjaC4gOV0oZG9jcy8wOS1qb2JleGVjdXRpb25hbHJlYWR5cnVubmluZy1hbmQtcmVjb3Zlci5tZCkgfAoKIyMgRG9jdW1lbnRhdGlvbiBjaGFwdGVycwoKMS4gW1RoZSBwcm9ibGVtLCBhbmQgdGhlIHNtYWxsZXN0IGNvcnJlY3QgbWVudGFsIG1vZGVsXShkb2NzLzAxLXRoZS1wcm9ibGVtLWFuZC1tZW50YWwtbW9kZWwubWQpCjIuIFtUaGUgYW5hdG9teSBvZiBhIHBhcnRpdGlvbmVkIHN0ZXBdKGRvY3MvMDItYW5hdG9teS1vZi1hLXBhcnRpdGlvbmVkLXN0ZXAubWQpCjMuIFtXaGF0IGdyaWRTaXplIGFjdHVhbGx5IGNvbnRyb2xzXShkb2NzLzAzLXdoYXQtZ3JpZHNpemUtYWN0dWFsbHktY29udHJvbHMubWQpCjQuIFtUaGUgd3JpdGVyLCB0aGUgYmVhbk1hcHBlZCB0cmFwLCBhbmQgZmluZGluZyB0aGUgcGFydGl0aW9uJ3Mgb3duIG5hbWVdKGRvY3MvMDQtdGhlLXdyaXRlci1hbmQtdGhlLWJlYW5tYXBwZWQtdHJhcC5tZCkKNS4gW1RoZSBkaWFnbm9zdGljIGVuZHBvaW50XShkb2NzLzA1LXRoZS1kaWFnbm9zdGljLWVuZHBvaW50Lm1kKQo2LiBbV2h5IHRoaXMgbW9kdWxlJ3Mgd29yayBpcyBDUFUtYm91bmQsIG5vdCBJL08tYm91bmRdKGRvY3MvMDYtd2h5LWNwdS1ib3VuZC1ub3QtaW8tYm91bmQubWQpCjcuIFtUaGUgZmFpbHVyZSB0aGF0IGRvZXMgbm90IGxvb2sgbGlrZSBhIGZhaWx1cmU6IHJlamVjdGVkIHBhcnRpdGlvbnNdKGRvY3MvMDctdGhlLXJlamVjdGVkZXhlY3V0aW9uZXhjZXB0aW9uLm1kKQo4LiBbUmVzdGFydCByZXJ1bnMgb25seSB0aGUgZmFpbGVkIHBhcnRpdGlvbiAmbWRhc2g7IHByb3ZlZCwgbm90IGFzc3VtZWRdKGRvY3MvMDgtcmVzdGFydC1yZXJ1bnMtb25seS10aGUtZmFpbGVkLXBhcnRpdGlvbi5tZCkKOS4gW0pvYkV4ZWN1dGlvbkFscmVhZHlSdW5uaW5nRXhjZXB0aW9uLCBmb3JldmVyICZtZGFzaDsgYW5kIHJlY292ZXIoKV0oZG9jcy8wOS1qb2JleGVjdXRpb25hbHJlYWR5cnVubmluZy1hbmQtcmVjb3Zlci5tZCkKMTAuIFtTY2FsaW5nIHNlbnNpdGl2aXR5IHRvIGRhdGEgc2l6ZSwgYW5kIHRoZSBob25lc3QgY2VpbGluZ10oZG9jcy8xMC1zY2FsaW5nLXNlbnNpdGl2aXR5LXRvLWRhdGEtc2l6ZS5tZCkKMTEuIFtQcm9kdWN0aW9uIGNoZWNrbGlzdF0oZG9jcy8xMS1wcm9kdWN0aW9uLWNoZWNrbGlzdC5tZCkKCiMjIENhcHR1cmVkIG91dHB1dAoKRXZlcnl0aGluZyB1bmRlciBbYGRvY3Mvb3V0cHV0L2BdKGRvY3Mvb3V0cHV0KSB3YXMgcHJvZHVjZWQgYnkgYSByZWFsIHJ1biAob3IgYSByZWFsIGBtdm4gdGVzdGApCmFuZCBpcyBxdW90ZWQgdmVyYmF0aW0gaW4gdGhlIGFydGljbGUgYW5kIHRoZSBjaGFwdGVycyBhYm92ZToKCnwgRmlsZSB8IFdoYXQgcHJvZHVjZWQgaXQgfAp8LS0tfC0tLXwKfCBgMDEtcHJvY2Vzc29yLWRldGVybWluaXNtLnR4dGAsIGAwMi1ncmlkc2l6ZS1pZ25vcmVkLnR4dGAgfCBKVW5pdCB0ZXN0cywgdmlhIGBtdm4gdGVzdGAgfAp8IGAwMy1wYWNrYWdlLXJlcGFja2FnaW5nLWphdmFwLnR4dGAgfCBgamF2YXBgIC8gYHVuemlwIC1sYCBhZ2FpbnN0IHRoZSByZWFsIDYuMC41IGFuZCA1LjIuNiBqYXJzIHwKfCBgMDQtZW5mb3JjZXVuaXF1ZW1ldGhvZHMtZXJyb3IudHh0YCB8IEEgcmVhbCBzdGFydHVwIGZhaWx1cmUsIGZpcnN0IGRyYWZ0IG9mIGBCYXRjaENvbmZpZ2AgfAp8IGAwNS1oYXBweS1wYXRoLTQtcGFydGl0aW9ucy50eHRgIHwgNCBzaGFyZHMsIGdyaWRTaXplIDQsIHBsdXMgdGhlIGRpYWdub3N0aWMgZW5kcG9pbnQgfAp8IGAwNi1yZWplY3RlZC1wYXJ0aXRpb25zLXN0dWNrLnR4dGAsIGAwNy1yZXN0YXJ0LXRocm93cy1hbHJlYWR5cnVubmluZy50eHRgLCBgMDgtcmVjb3Zlci10aGVuLXJlc3RhcnQudHh0YCB8IFRoZSBgcmVqZWN0YCBwcm9maWxlLCBhIGZhaWxlZCByZXN0YXJ0IGF0dGVtcHQsIHRoZW4gdGhlIGByZWNvdmVyYCBwcm9maWxlLCBhbGwgYWdhaW5zdCB0aGUgc2FtZSBIMiBmaWxlIGFjcm9zcyBzZXBhcmF0ZSBKVk1zIHwKfCBgMDktZnVsbC1zY2FsZS10aHJvdWdocHV0LnR4dGAgfCBUaGUgZnVsbCBncmlkLXNpemUgc3dlZXAgYXQgMTAsMDAwLDAwMCByb3dzIGFuZCBhdCAzMDAsMDAwIHJvd3MgfAp8IGAxMC1yZWFsLWpvYi1ncmlkc2l6ZS1pZ25vcmVkLnR4dGAgfCBBIHJlYWwgYGphdmEgLWphcmAgcnVuIChub3QganVzdCB0aGUgdW5pdCB0ZXN0KSBjb25maXJtaW5nIGdyaWRTaXplIGlzIGlnbm9yZWQgfAoKIyMgUnVubmluZyBpdAoKTmVlZHMgYSBKREsgMjUgYW5kIE1hdmVuIDMuOSwgcGx1cyBQeXRob24gMyBmb3IgdGhlIGRhdGEgZ2VuZXJhdG9yLgoKYGBgYmFzaApleHBvcnQgSkFWQV9IT01FPS9wYXRoL3RvL2pkay0yNQptdm4gLURza2lwVGVzdHMgcGFja2FnZQpweXRob24zIHNjcmlwdHMvZ2VuZXJhdGUtc2hhcmRzLnB5IC4vZGF0YS9zaGFyZHMgMTAwMDAwMDAgNCAgICMgNCBzaGFyZCBmaWxlcywgMi41TSByb3dzIGVhY2gKamF2YSAtamFyIHRhcmdldC9zcHJpbmctYmF0Y2gtcGFydGl0aW9uaW5nLTEuMC4wLmphciAtLXBhcnRpdGlvbi5zaGFyZHMtZGlyPS4vZGF0YS9zaGFyZHMgLS1wYXJ0aXRpb24uZ3JpZC1zaXplPTQKYGBgCgpgR0VUIGh0dHA6Ly9sb2NhbGhvc3Q6ODA4MS9iYXRjaC9wYXJ0aXRpb25zL3tqb2JFeGVjdXRpb25JZH1gIChvciBgL2JhdGNoL3BhcnRpdGlvbnMvbGF0ZXN0YCkKc2hvd3Mgd2hpY2ggdGhyZWFkIHJhbiB3aGljaCBwYXJ0aXRpb24sIGFuZCBmb3IgaG93IGxvbmcgJm1kYXNoOyBzZWUKW2NoYXB0ZXIgNV0oZG9jcy8wNS10aGUtZGlhZ25vc3RpYy1lbmRwb2ludC5tZCkuCgpgc2NyaXB0cy9nZW5lcmF0ZS1zaGFyZHMucHkgPGRpcj4gPHJvd3M+IDxzaGFyZHM+IFstLWNvcnJ1cHQtc2hhcmQgTl0gWy0tc2VlZCBTXWAgcHJvZHVjZXMgdGhlCnNoYXJkZWQgQ1NWcyBhbnkgb2YgdGhlIGFib3ZlIGNvbW1hbmRzIHJlYWQ7IHRoZSBzYW1lIHNlZWQgcHJvZHVjZXMgYnl0ZS1pZGVudGljYWwgcm93IGNvbnRlbnQKcmVnYXJkbGVzcyBvZiBob3cgbWFueSBzaGFyZHMgaXQgaXMgc3BsaXQgaW50bywgd2hpY2ggaXMgd2hhdCBtYWtlcyB0aGUgZ3JpZC1zaXplIGNvbXBhcmlzb25zIGluCltjaGFwdGVyIDEwXShkb2NzLzEwLXNjYWxpbmctc2Vuc2l0aXZpdHktdG8tZGF0YS1zaXplLm1kKSBhcHBsZXMtdG8tYXBwbGVzLgoKIyMgTGljZW5jZQoKTUlUIOKAlCBzZWUgdGhlIHJlcG9zaXRvcnkgW0xJQ0VOU0VdKC4uL0xJQ0VOU0UpLgo= \ No newline at end of file