spring-boot-demo/spring-batch-partitioning/docs/01-the-problem-and-mental-model.md

1. The problem, and the smallest correct mental model

README | Next: Anatomy of a partitioned step →

The problem

A chunk-oriented Spring Batch step is already fast: it reads an item, processes it, and only touches the database once per chunk, not once per row. But it is still one reader, one processor, one writer, one thread. Give it 10 million rows of CPU-bound scoring work and it will get through all of them correctly — restartably, with fault tolerance, with every guarantee chunk-oriented processing gives you — using exactly one CPU core, for as long as that takes. On the two-core sandbox this module was built and measured on, that is about 70 seconds for 10 million rows (see docs/output/09-full-scale-throughput.txt). On a job with real per-row work, or ten times the rows, "one thread" stops being a detail and starts being the bottleneck.

Partitioning is Spring Batch's answer to "make more than one thread do this work, without throwing away restartability." It does not change chunk-oriented processing at all — it takes the step itself and hands several copies of it, each with a different slice of input, to separate threads (or, wired differently, to separate JVMs on separate machines). Each copy is a completely ordinary chunk-oriented step: its own reader, its own processor, its own writer, its own transaction, its own restart bookkeeping in the job repository.

The smallest correct mental model

There are two steps in a partitioned job, not one, and they are structurally different:

• The manager step does no item processing. Its entire job is to ask a Partitioner for a set of input descriptions (in this module: which shard CSV file), hand each one to a worker thread as a separate StepExecution, wait for all of them, and roll the results up into one outcome.
• The worker step is the chunk-oriented step you already know from the earlier spring-batch article — reader, processor, writer, fault tolerance, all of it — run once per partition, against only that partition's slice of the input.

The Partitioner never sees a row of data. It only produces descriptions of work — in this module, one file path per partition, via the built-in MultiResourcePartitioner. The actual CSV parsing, risk scoring, and database writing all happen inside the worker step, four separate times, on four separate threads, each against its own file. See chapter 3 for exactly how many partitions actually run, which is a more interesting question than it sounds.

What this buys you, and what it does not

Partitioning parallelizes CPU-bound and I/O-bound work across threads (or machines) while keeping every restart, skip, and retry guarantee chunk-oriented processing already gives a single step — each partition restarts independently, as chapter 8 demonstrates concretely. It does not automatically parallelize a shared bottleneck: four threads writing to the same single-writer embedded database do not write four times as fast just because four threads are asking. Chapter 6 and chapter 10 measure exactly how much that shortfall is, on this hardware, with this writer.

Going deeper

• Manager/worker terminology and the full SPI: Spring Batch reference — Scaling and Parallel Processing (rel="nofollow").
• The chunk-oriented step this builds on: Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability.

Next: Anatomy of a partitioned step →