spring-boot-demo/spring-batch-partitioning/docs/01-the-problem-and-mental-model.md

# 1. The problem, and the smallest correct mental model

[README](../README.md) | [Next: Anatomy of a partitioned step →](02-anatomy-of-a-partitioned-step.md)

## The problem

A chunk-oriented Spring Batch step is already fast: it reads an item, processes it, and only
touches the database once per chunk, not once per row. But it is still **one reader, one
processor, one writer, one thread.** Give it 10 million rows of CPU-bound scoring work and it
will get through all of them correctly — restartably, with fault tolerance, with every
guarantee chunk-oriented processing gives you — using exactly one CPU core, for as long as
that takes. On the two-core sandbox this module was built and measured on, that is about 70
seconds for 10 million rows (see
[`docs/output/09-full-scale-throughput.txt`](output/09-full-scale-throughput.txt)). On a job with
real per-row work, or ten times the rows, "one thread" stops being a detail and starts being the
bottleneck.

Partitioning is Spring Batch's answer to "make more than one thread do this work, without
throwing away restartability." It does not change chunk-oriented processing at all — it
takes the *step itself* and hands several copies of it, each with a different slice of input, to
separate threads (or, wired differently, to separate JVMs on separate machines). Each copy is a
completely ordinary chunk-oriented step: its own reader, its own processor, its own writer, its
own transaction, its own restart bookkeeping in the job repository.

## The smallest correct mental model

There are two steps in a partitioned job, not one, and they are structurally different:

- **The manager step** does no item processing. Its entire job is to ask a `Partitioner` for a
  set of input descriptions (in this module: which shard CSV file), hand each one to a worker
  thread as a separate `StepExecution`, wait for all of them, and roll the results up into one
  outcome.
- **The worker step** is the chunk-oriented step you already know from
  [the earlier `spring-batch` article](https://ankurm.com/) — reader, processor, writer,
  fault tolerance, all of it — run once per partition, against only that partition's slice
  of the input.

<figure>
<svg viewBox="0 0 740 300" role="img" aria-label="One manager step asks a Partitioner for four ExecutionContexts, one per shard file, and hands each to an identical worker step running on its own thread; all four worker steps write to the same ORDER_RISK_SUMMARY table.">
<style>.h{font:600 12px sans-serif;fill:#1a1a1a}.c{font:11px sans-serif;fill:#4b5563}.m{font:11px monospace;fill:#1a1a1a}</style>
<rect x="300" y="16" width="160" height="40" rx="4" fill="#e8eefc" stroke="#5b7fc7"/>
<text x="326" y="41" class="h">ordersManagerStep</text>
<text x="130" y="90" class="c">Partitioner.partition(gridSize) returns one ExecutionContext per shard file</text>
<rect x="20" y="110" width="160" height="36" rx="4" fill="#f4f5f7" stroke="#b7bec9"/><text x="40" y="133" class="m">fileName=shard-00.csv</text>
<rect x="200" y="110" width="160" height="36" rx="4" fill="#f4f5f7" stroke="#b7bec9"/><text x="220" y="133" class="m">fileName=shard-01.csv</text>
<rect x="380" y="110" width="160" height="36" rx="4" fill="#f4f5f7" stroke="#b7bec9"/><text x="400" y="133" class="m">fileName=shard-02.csv</text>
<rect x="560" y="110" width="160" height="36" rx="4" fill="#f4f5f7" stroke="#b7bec9"/><text x="580" y="133" class="m">fileName=shard-03.csv</text>
<rect x="20" y="180" width="160" height="60" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="40" y="205" class="h">worker step</text><text x="40" y="222" class="c">partition0, thread A</text>
<rect x="200" y="180" width="160" height="60" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="220" y="205" class="h">worker step</text><text x="220" y="222" class="c">partition1, thread B</text>
<rect x="380" y="180" width="160" height="60" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="400" y="205" class="h">worker step</text><text x="400" y="222" class="c">partition2, thread C</text>
<rect x="560" y="180" width="160" height="60" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="580" y="205" class="h">worker step</text><text x="580" y="222" class="c">partition3, thread D</text>
<rect x="260" y="268" width="220" height="24" rx="4" fill="#fdeccf" stroke="#c9973f"/>
<text x="275" y="285" class="c">ORDER_RISK_SUMMARY (shared H2 file)</text>
</svg>
</figure>

The `Partitioner` never sees a row of data. It only produces *descriptions* of work &mdash; in
this module, one file path per partition, via the built-in `MultiResourcePartitioner`. The actual
CSV parsing, risk scoring, and database writing all happen inside the worker step, four separate
times, on four separate threads, each against its own file. See
[chapter 3](03-what-gridsize-actually-controls.md) for exactly how many partitions actually run,
which is a more interesting question than it sounds.

## What this buys you, and what it does not

Partitioning parallelizes CPU-bound and I/O-bound work across threads (or machines) while keeping
every restart, skip, and retry guarantee chunk-oriented processing already gives a single step
&mdash; each partition restarts independently, as [chapter 8](08-restart-reruns-only-the-failed-partition.md)
demonstrates concretely. It does **not** automatically parallelize a shared bottleneck: four
threads writing to the same single-writer embedded database do not write four times as fast just
because four threads are asking. [Chapter 6](06-why-cpu-bound-not-io-bound.md) and
[chapter 10](10-scaling-sensitivity-to-data-size.md) measure exactly how much that shortfall is,
on this hardware, with this writer.

## Going deeper

- Manager/worker terminology and the full SPI:
  [Spring Batch reference &mdash; Scaling and Parallel Processing](https://docs.spring.io/spring-batch/reference/scalability.html) (`rel="nofollow"`).
- The chunk-oriented step this builds on:
  [Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/).

[Next: Anatomy of a partitioned step &rarr;](02-anatomy-of-a-partitioned-step.md)