Add spring-batch: jobs, steps, chunk processing and restartability on Boot 4.1
Companion code for "Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability". A productImportJob configured three ways by profile against a poisoned CSV row, run as real java -jar processes (not just JUnit) so the restart story is genuine: a chunk fails and rolls back, the process exits, a brand-new JVM against the same file-based H2 database resumes at the exact next unread row (READ_COUNT 20, not 60) and completes. Findings the build pins: - StepBuilder.chunk(int, PlatformTransactionManager) still compiles in Batch 6.0.5 but returns the legacy SimpleStepBuilder; chunk(int) returns the new ChunkOrientedStepBuilder, and only the latter is used here. - Two different ExecutionContext classes now exist in two different packages (infrastructure.item vs core.repository.persistence) with different shapes. - spring-boot-starter-batch alone gives a resourceless JobRepository that forgets every JobInstance the moment the JVM exits; spring-boot-starter- batch-jdbc is what makes the restart demo possible at all, demonstrated by excluding BatchJdbcAutoConfiguration and watching a "restart" collide with the previous run's own data instead of resuming it. - A migration-guide summary claiming CommandLineJobRunner was removed in 6.0 is wrong -- javap against the real jar shows @Deprecated(forRemoval=true), not removed. - RepeatStatus moved from core.repeat to infrastructure.repeat, caught by the compiler rather than by reading docs. 11 documentation chapters, 10 captured transcripts (unit tests, javap output, and real two-JVM scenario runs), all regenerated by scripts/run-all.sh. Fixed after push: three dead docs.spring.io links in the doc chapters (readersAndWriters/* and chunk-oriented-processing/*.html paths moved when Spring Batch 6 reorganized its reference docs; corrected to the current readers-and-writers/*, processor.html and chunk-oriented-processing.html paths, verified 200 via curl before committing). Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019DXsJ1zpikbA1MQJN6RqFA
This commit is contained in:
Claude
@@ -0,0 +1,74 @@
|
||||
# spring-batch
|
||||
|
||||
Companion code for **[Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/)**
|
||||
on [ankurm.com](https://ankurm.com).
|
||||
|
||||
Verified against Spring Boot **4.1.1**, Spring Batch **6.0.5**, Spring Framework **7.0.9**, on
|
||||
Temurin JDK **25.0.4.1+1**.
|
||||
|
||||
One job (`productImportJob`: import a CSV of products, then report a count), configured three
|
||||
different ways by Spring profile, plus one infrastructure variant that is not a profile:
|
||||
|
||||
| Profile / flag | What it demonstrates | Docs |
|
||||
|---|---|---|
|
||||
| `clean` | The happy path: chunk-oriented reading, filtering, writing | [ch. 3](docs/03-chunk-oriented-processing.md), [ch. 4](docs/04-item-processor-as-filter.md) |
|
||||
| `broken` | No fault tolerance: a bad row fails the whole chunk and the job; restart resumes exactly where it left off | [ch. 7](docs/07-restartability.md) |
|
||||
| `skip` | `faultTolerant().skip(...)`: the same bad row is skipped instead, job completes in one pass | [ch. 8](docs/08-skip-vs-restart.md) |
|
||||
| `--spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration` | Resourceless job repository: a restart forgets everything happened | [ch. 10](docs/10-resourceless-vs-jdbc.md) |
|
||||
|
||||
## Documentation chapters
|
||||
|
||||
1. [The problem, and the smallest correct mental model](docs/01-the-problem-and-mental-model.md)
|
||||
2. [The anatomy of a job](docs/02-anatomy-of-a-job.md)
|
||||
3. [Chunk-oriented processing](docs/03-chunk-oriented-processing.md)
|
||||
4. [The item processor as a filter](docs/04-item-processor-as-filter.md)
|
||||
5. [Launching a job, and why restart is not a separate API](docs/05-launching-and-jobparameters.md)
|
||||
6. [The JDBC writer, and why it is not `beanMapped()`](docs/06-jdbc-writer-and-records.md)
|
||||
7. [Restartability: what actually resumes, and from where](docs/07-restartability.md)
|
||||
8. [Skip vs. restart](docs/08-skip-vs-restart.md)
|
||||
9. [Corrections found while writing this](docs/09-corrections.md)
|
||||
10. [Resourceless vs. JDBC-backed job repositories](docs/10-resourceless-vs-jdbc.md)
|
||||
11. [Production checklist](docs/11-production-checklist.md)
|
||||
|
||||
## Captured output
|
||||
|
||||
Everything under [`docs/output/`](docs/output) was produced by a real run and is quoted verbatim
|
||||
in the article and the chapters above:
|
||||
|
||||
| File | What produced it |
|
||||
|---|---|
|
||||
| `01-processor-filter.txt`, `02-exception-hierarchy.txt` | JUnit tests, via `mvn test` |
|
||||
| `03-stepbuilder-chunk-overloads.txt`, `04-two-executioncontext-classes.txt`, `09-commandlinejobrunner-deprecated-not-removed.txt` | `javap` against the real 6.0.5 jars, via `scripts/capture-javap.sh` |
|
||||
| `05-happy-path.txt` | The `clean` profile, via `scripts/capture-scenarios.sh` |
|
||||
| `07-restart-run1-fails.txt`, `08-restart-run2-resumes.txt` | The `broken` profile, run twice in separate JVMs against the same database |
|
||||
| `10-skip-instead-of-fail.txt` | The `skip` profile |
|
||||
| `11-resourceless-forgets-everything.txt` | The `clean` profile with `BatchJdbcAutoConfiguration` excluded, run twice |
|
||||
|
||||
## Running it
|
||||
|
||||
Needs a JDK 25 and Maven 3.9.
|
||||
|
||||
```bash
|
||||
export JAVA_HOME=/path/to/jdk-25
|
||||
mvn -DskipTests package
|
||||
./scripts/run-all.sh # regenerates everything under docs/output/
|
||||
```
|
||||
|
||||
Or run one scenario by hand:
|
||||
|
||||
```bash
|
||||
java -jar target/spring-batch-1.0.0.jar \
|
||||
--spring.profiles.active=clean \
|
||||
--import.file=file:./scenario-data/clean-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:./scenario-data/clean-demo/db
|
||||
```
|
||||
|
||||
`scripts/run-scenario.sh <name> <profile> <source-csv>` wraps that for repeat use. Each scenario
|
||||
gets its own file-based H2 database under `scenario-data/<name>/`, gitignored, so runs never
|
||||
interfere with each other and a restart persists across separate `java -jar` invocations the way
|
||||
it would across a real process restart.
|
||||
|
||||
There is a diagnostic-adjacent endpoint nowhere in this module by design — everything
|
||||
observable here comes from the standard `BATCH_STEP_EXECUTION` table via a plain SQL query,
|
||||
which is deliberately how [chapter 10](docs/10-resourceless-vs-jdbc.md) suggests checking your
|
||||
own job repository in production.
|
||||
@@ -0,0 +1,86 @@
|
||||
# 1. The problem, and the smallest correct mental model
|
||||
|
||||
[← README](../README.md) | [Next: Anatomy of a job →](02-anatomy-of-a-job.md)
|
||||
|
||||
A batch job processes a lot of records without a person watching each one: importing a CSV of
|
||||
products into a database, closing out a day's transactions, re-indexing a search table. Two
|
||||
things make this harder than writing a `for` loop over the rows:
|
||||
|
||||
- **It has to survive being interrupted.** A process gets killed, a database connection drops,
|
||||
a row three-quarters of the way through is bad. A `for` loop that dies at row 4,700,000 has no
|
||||
idea it already wrote 4,699,999 rows, and re-running it from the top either duplicates work or
|
||||
duplicates data.
|
||||
- **It has to report what happened**, precisely: how many rows were read, how many were written,
|
||||
how many were skipped and why, whether it finished. "The import ran" is not an answer anyone
|
||||
building on top of this can use.
|
||||
|
||||
Spring Batch's job is to own both of those problems so your code only has to describe three
|
||||
things: where the rows come from, what to do to each one, and where they go. Everything else
|
||||
— tracking progress, committing in batches, remembering where a failed run stopped —
|
||||
is the framework's job, provided you tell it enough for it to do that job. Most of this article
|
||||
is about that "provided."
|
||||
|
||||
## The five nouns
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 740 300" role="img" aria-label="A Job contains an ordered list of Steps; each Step is chunk-oriented and reads from a reader, transforms with a processor, and writes with a writer; a JobRepository records everything happening.">
|
||||
<style>
|
||||
.t{font:600 14px sans-serif;fill:#1a1a1a}.h{font:600 12px sans-serif;fill:#1a1a1a}.c{font:11px sans-serif;fill:#4b5563}.m{font:11px monospace;fill:#1a1a1a}
|
||||
</style>
|
||||
<rect x="10" y="10" width="720" height="80" rx="6" fill="#e8eefc" stroke="#5b7fc7"/>
|
||||
<text x="24" y="30" class="h">Job "productImportJob"</text>
|
||||
<text x="24" y="50" class="m">start(importStep).next(reportStep)</text>
|
||||
<rect x="30" y="60" width="180" height="24" rx="4" fill="#fff" stroke="#5b7fc7"/>
|
||||
<text x="45" y="76" class="m">Step: importStep</text>
|
||||
<rect x="230" y="60" width="180" height="24" rx="4" fill="#fff" stroke="#5b7fc7"/>
|
||||
<text x="245" y="76" class="m">Step: reportStep</text>
|
||||
<rect x="60" y="120" width="620" height="90" rx="6" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<text x="74" y="140" class="h">importStep is chunk-oriented</text>
|
||||
<rect x="80" y="155" width="150" height="40" rx="4" fill="#fff" stroke="#4a9d63"/>
|
||||
<text x="95" y="179" class="m">ItemReader</text>
|
||||
<rect x="290" y="155" width="150" height="40" rx="4" fill="#fff" stroke="#4a9d63"/>
|
||||
<text x="305" y="179" class="m">ItemProcessor</text>
|
||||
<rect x="500" y="155" width="150" height="40" rx="4" fill="#fff" stroke="#4a9d63"/>
|
||||
<text x="515" y="179" class="m">ItemWriter</text>
|
||||
<line x1="230" y1="175" x2="290" y2="175" stroke="#4a9d63" marker-end="url(#arrow)"/>
|
||||
<line x1="440" y1="175" x2="500" y2="175" stroke="#4a9d63" marker-end="url(#arrow)"/>
|
||||
<defs><marker id="arrow" markerWidth="8" markerHeight="8" refX="6" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="#4a9d63"/></marker></defs>
|
||||
<rect x="230" y="240" width="280" height="46" rx="6" fill="#fdeccf" stroke="#c9973f"/>
|
||||
<text x="244" y="258" class="h">JobRepository</text>
|
||||
<text x="244" y="276" class="c">every Job/Step execution, read/write/skip counts, ExecutionContext</text>
|
||||
<line x1="145" y1="90" x2="145" y2="120" stroke="#5b7fc7"/>
|
||||
<line x1="370" y1="210" x2="370" y2="240" stroke="#4a9d63"/>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
The picture above names everything this article uses:
|
||||
|
||||
- **Job** — a named, ordered list of steps. `productImportJob` in this repository has two:
|
||||
import the rows, then report a count.
|
||||
- **Step** — one unit of work within a job. A step is either a single **Tasklet** (run
|
||||
once, do one thing — `reportStep` here) or **chunk-oriented**: read one item, process it,
|
||||
repeat until you have a chunk's worth, then write and commit the whole chunk in one
|
||||
transaction.
|
||||
- **ItemReader / ItemProcessor / ItemWriter** — the three interfaces chunk-oriented
|
||||
processing is built from. This module's reader is a `FlatFileItemReader` over a CSV, the
|
||||
processor validates and filters rows, the writer is a `JdbcBatchItemWriter`.
|
||||
- **JobRepository** — the thing that makes the first two bullet points of this page
|
||||
possible. It persists every job and step execution: status, timestamps, and the read / write /
|
||||
filter / skip / commit / rollback counts you saw in the transcripts above. Chapter
|
||||
[10](10-resourceless-vs-jdbc.md) is entirely about a trap hiding in how this gets configured on
|
||||
Spring Boot 4.1.
|
||||
|
||||
Hold onto one fact from this page, because it is the one everything else cashes in later: **a
|
||||
chunk commits or it doesn't, as a whole.** There is no partial chunk. That single sentence is the
|
||||
entire explanation for why restarting a failed job does not reprocess everything, and it is why
|
||||
the "poisoned" row in this repository's demo data fails nine other, perfectly good rows along
|
||||
with it — see [chapter 7](07-restartability.md).
|
||||
|
||||
## What this repository demonstrates, and how to run it
|
||||
|
||||
Four Spring profiles configure the same job three different ways plus one infrastructure
|
||||
variant; see the [README](../README.md) for the full table and the exact commands. Everything
|
||||
under `docs/output/` was produced by `scripts/capture-scenarios.sh` and `scripts/capture-javap.sh`,
|
||||
which `scripts/run-all.sh` runs in sequence.
|
||||
|
||||
[Next: Anatomy of a job →](02-anatomy-of-a-job.md)
|
||||
@@ -0,0 +1,81 @@
|
||||
# 2. The anatomy of a job, and why the code looks different from older tutorials
|
||||
|
||||
[← Previous](01-the-problem-and-mental-model.md) | [README](../README.md) | [Next: Chunk-oriented processing →](03-chunk-oriented-processing.md)
|
||||
|
||||
`BatchConfig` in this module builds jobs and steps like this:
|
||||
|
||||
```java
|
||||
new JobBuilder("productImportJob", jobRepository)
|
||||
.start(importStep)
|
||||
.next(reportStep)
|
||||
.build();
|
||||
|
||||
new StepBuilder("importStep", jobRepository)
|
||||
.<Product, Product>chunk(10)
|
||||
.transactionManager(transactionManager)
|
||||
.reader(productReader)
|
||||
.processor(productProcessor)
|
||||
.writer(productWriter)
|
||||
.build();
|
||||
```
|
||||
|
||||
If you have seen Spring Batch code before 2023 or so, two things here might look unfamiliar,
|
||||
and it is worth being precise about which is a genuine Spring Boot 4.1 change and which is just
|
||||
old:
|
||||
|
||||
- `JobBuilderFactory` / `StepBuilderFactory` autowired as beans, then called as
|
||||
`jobBuilderFactory.get("name")` — **this was removed in Spring Batch 5, years before this
|
||||
article.** It is not a Boot 4.1 surprise, it is a dead end that tutorials keep copy-pasting.
|
||||
`JobBuilder` and `StepBuilder` are plain classes you construct directly with a `JobRepository`,
|
||||
as above.
|
||||
- `.chunk(10, transactionManager)` — passing the transaction manager as a second argument
|
||||
to `chunk()` — **this compiles in 6.0.5, but it returns a different builder.**
|
||||
`StepBuilder.chunk(int)` returns a `ChunkOrientedStepBuilder`, the model this whole module
|
||||
uses; `StepBuilder.chunk(int, PlatformTransactionManager)` returns the older
|
||||
`SimpleStepBuilder`, kept for the pre-6.0 chunk-processing model. Real `javap` output for both
|
||||
overloads, from this project's own `spring-batch-core-6.0.5.jar`:
|
||||
[`docs/output/03-stepbuilder-chunk-overloads.txt`](output/03-stepbuilder-chunk-overloads.txt).
|
||||
The two builders overlap almost entirely in the methods you would reach for —
|
||||
`reader`/`processor`/`writer`/`faultTolerant`/`skip` exist on both — so the wrong choice
|
||||
usually still compiles and runs, and just quietly uses the legacy step implementation. Call
|
||||
`.chunk(10).transactionManager(tx)` (two calls) if you want `ChunkOrientedStep`.
|
||||
|
||||
<blockquote style="background:#f4f5f7;border:1px solid #e2e5ea;border-left:4px solid #b7bec9;border-radius:6px;padding:16px 20px;"><strong>The fingerprint of picking the wrong overload.</strong> Your step still builds, runs, reads, writes and commits correctly — there is no error. The tell is in what you can't reach: <code>ChunkOrientedStepBuilder</code>-only options like <code>retryPolicy(RetryPolicy)</code> using Spring Framework 7's retry API, or a stack trace mentioning <code>ChunkOrientedStep</code> versus the older <code>TaskletStep</code> wrapping a <code>ChunkOrientedTasklet</code>. If you copied a two-argument <code>chunk(10, txManager)</code> from a pre-6.0 example and it "just worked", this is why nothing complained.</blockquote>
|
||||
|
||||
## Where things moved
|
||||
|
||||
Spring Batch 6.0 (bundled with Spring Boot 4.1.1 as `spring-batch.version` `6.0.5`, confirmed
|
||||
against `spring-boot-dependencies-4.1.1.pom`) reorganized packages fairly aggressively. The ones
|
||||
this module's code actually hits:
|
||||
|
||||
| Old (Spring Batch 5.x) | New (6.0.5) |
|
||||
|---|---|
|
||||
| `org.springframework.batch.core.repeat.RepeatStatus` | `org.springframework.batch.infrastructure.repeat.RepeatStatus` |
|
||||
| `org.springframework.batch.item.*` (readers, writers, `ExecutionContext`) | `org.springframework.batch.infrastructure.item.*` |
|
||||
| `org.springframework.batch.core.JobParameters` (mutable-ish, `Map`-backed) | `org.springframework.batch.core.job.parameters.JobParameters` (immutable record, `Set`-backed) |
|
||||
|
||||
The `RepeatStatus` move is the one this project's own `reportTasklet` hit directly — see
|
||||
[chapter 9](09-corrections.md) for the exact compiler error it produced before the import was
|
||||
fixed. It is a good example of the article's verification rule in practice: rather than trusting
|
||||
a description of where things moved, write the code from the old import, let `javac` say
|
||||
`package ... does not exist`, and fix the import. That is faster and more reliable than
|
||||
reading a migration guide's prose, and chapter 9 has a case where trusting the prose produced a
|
||||
wrong claim.
|
||||
|
||||
## JobRepository and JobOperator
|
||||
|
||||
Two interfaces you inject rather than configure by hand in this module:
|
||||
|
||||
- **`JobRepository`** — every `JobBuilder` and `StepBuilder` above takes one as a
|
||||
constructor argument. In 6.0 it also extends `JobExplorer` (querying past executions), so
|
||||
there is one bean to inject instead of two.
|
||||
- **`JobOperator`** — what `ImportRunner` calls to start the job (see
|
||||
[chapter 5](05-launching-and-jobparameters.md)). It extends `JobLauncher` and adds operational
|
||||
methods: `restart(executionId)`, `stop(executionId)`, `recover(execution)`, `getJobNames()`.
|
||||
`JobLauncher`/`JobExplorer` still exist but are the deprecated half of this pair now.
|
||||
|
||||
Both are auto-configured by `spring-boot-starter-batch-jdbc` on this module's classpath; nothing
|
||||
in `BatchConfig` declares them as beans. [Chapter 10](10-resourceless-vs-jdbc.md) explains what
|
||||
you get instead if that starter is missing.
|
||||
|
||||
[Next: Chunk-oriented processing →](03-chunk-oriented-processing.md)
|
||||
@@ -0,0 +1,78 @@
|
||||
# 3. Chunk-oriented processing: what "chunk(10)" actually does
|
||||
|
||||
[← Previous](02-anatomy-of-a-job.md) | [README](../README.md) | [Next: The item processor as a filter →](04-item-processor-as-filter.md)
|
||||
|
||||
`chunk(10)` means: read up to 10 items (calling the reader once per item), run each through the
|
||||
processor, then hand all the survivors to the writer in **one call**, inside **one transaction**.
|
||||
If the writer succeeds, the transaction commits and the step's counters — read, write,
|
||||
filter, commit — move forward together. If anything in that chunk throws, the whole
|
||||
transaction rolls back: none of those 10 items' writes are kept, even the ones that would have
|
||||
succeeded on their own.
|
||||
|
||||
The happy-path run of this module's `productImportJob` against clean data makes this concrete.
|
||||
60 rows, chunk size 10, two rows fail the processor's validation (filtered, not written):
|
||||
|
||||
```console
|
||||
$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;
|
||||
READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT
|
||||
60 | 2 | 58 | 6
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/05-happy-path.txt`](output/05-happy-path.txt), source:
|
||||
[`ProductValidatingProcessor.java`](../src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java).
|
||||
60 rows at chunk size 10 is 6 chunks — `COMMIT_COUNT` confirms all 6 committed, one
|
||||
transaction each.
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 740 220" role="img" aria-label="60 rows split into six chunks of 10; each chunk is read, processed, and written as one transaction that either commits fully or rolls back fully.">
|
||||
<style>.h{font:600 12px sans-serif;fill:#1a1a1a}.c{font:11px sans-serif;fill:#4b5563}.m{font:11px monospace;fill:#1a1a1a}</style>
|
||||
<text x="20" y="24" class="h">60 rows → six chunks of 10, each its own transaction</text>
|
||||
<g>
|
||||
<!-- 6 chunk boxes -->
|
||||
<rect x="20" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<rect x="140" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<rect x="260" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<rect x="380" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<rect x="500" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<rect x="620" y="50" width="110" height="50" rx="5" fill="#e7f4ea" stroke="#4a9d63"/>
|
||||
<text x="45" y="80" class="m">1-10</text>
|
||||
<text x="165" y="80" class="m">11-20</text>
|
||||
<text x="285" y="80" class="m">21-30</text>
|
||||
<text x="405" y="80" class="m">31-40</text>
|
||||
<text x="525" y="80" class="m">41-50</text>
|
||||
<text x="645" y="80" class="m">51-60</text>
|
||||
</g>
|
||||
<text x="20" y="130" class="c">Each box: read 10 → process 10 (0-1 filtered) → write survivors → COMMIT.</text>
|
||||
<text x="20" y="150" class="c">Row 12 (chunk 2) and row 33 (chunk 4) are filtered by the processor -- see chapter 4 -- so those</text>
|
||||
<text x="20" y="170" class="c">two chunks commit 9 rows instead of 10. 58 rows land in PRODUCT; every chunk still commits.</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
## Why 10, and what changes if you pick a different size
|
||||
|
||||
Chunk size is a trade-off with no universally right answer:
|
||||
|
||||
- **Smaller chunks** commit more often, so a failure loses less work and transactions are
|
||||
shorter (less lock contention, smaller rollback). They also mean more round trips to the
|
||||
database — more commit overhead per row.
|
||||
- **Larger chunks** amortize that overhead across more rows, but a failure anywhere in the chunk
|
||||
costs you the whole chunk's work, and the transaction held open is bigger and longer.
|
||||
|
||||
10 is small enough that chapter 7's poisoned row costs you 12 rows of rework at most (one
|
||||
chunk), not thousands, and small enough to demonstrate multiple commits from 60 rows without a
|
||||
huge fixture file. Production jobs processing millions of rows commonly use chunk sizes in the
|
||||
hundreds or low thousands; the right number depends on row size, write cost, and how expensive a
|
||||
partial redo is for your specific job — there is no formula that replaces measuring it
|
||||
against your own writer.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The chunk-processing reference: [Spring Batch — Chunk-oriented processing](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing.html) (`rel="nofollow"`).
|
||||
- `ChunkOrientedStep` itself, if you want to see the commit/rollback logic this chapter
|
||||
describes: `org.springframework.batch.core.step.item.ChunkOrientedStep` in
|
||||
`spring-batch-core-6.0.5.jar` (the stack trace in
|
||||
[`docs/output/07-restart-run1-fails.txt`](output/07-restart-run1-fails.txt) names it directly).
|
||||
- What happens when the processor itself decides to drop a row rather than the writer failing:
|
||||
[chapter 4](04-item-processor-as-filter.md).
|
||||
|
||||
[Next: The item processor as a filter →](04-item-processor-as-filter.md)
|
||||
@@ -0,0 +1,62 @@
|
||||
# 4. The item processor as a filter
|
||||
|
||||
[← Previous](03-chunk-oriented-processing.md) | [README](../README.md) | [Next: Launching and JobParameters →](05-launching-and-jobparameters.md)
|
||||
|
||||
`ItemProcessor<I, O>` has one method: `O process(I item)`. Returning a transformed item is the
|
||||
obvious case; returning `null` is the one people miss, and it means something specific:
|
||||
**this item is filtered.** It is counted separately (`FILTER_COUNT`) from both `READ_COUNT` and
|
||||
`WRITE_COUNT`, and it never reaches the writer, so it cannot throw the exception a bad write
|
||||
would.
|
||||
|
||||
[`ProductValidatingProcessor`](../src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java)
|
||||
uses this for two checks that can be made from one row alone, with no database involved:
|
||||
|
||||
```java
|
||||
if (!SKU_PATTERN.matcher(item.sku()).matches()) {
|
||||
return null; // filtered: does not match ^[A-Z]{3}-\d{4}$
|
||||
}
|
||||
if (item.priceCents() <= 0) {
|
||||
return null; // filtered: priceCents must be positive
|
||||
}
|
||||
```
|
||||
|
||||
Real output, from a plain unit test with no Spring context involved
|
||||
([`ProductValidatingProcessorTest.java`](../src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java),
|
||||
transcript [`docs/output/01-processor-filter.txt`](output/01-processor-filter.txt)):
|
||||
|
||||
```console
|
||||
input : Product[sku=abc-0012, name=Widget 12, priceCents=1012] <- lowercase sku, fails ^[A-Z]{3}-\d{4}$
|
||||
result : null
|
||||
|
||||
input : Product[sku=ABC-0033, name=Widget 33, priceCents=0] <- priceCents is zero
|
||||
result : null
|
||||
```
|
||||
|
||||
## Filter vs. skip vs. fail: pick by what the check needs
|
||||
|
||||
This module deliberately does not filter the poisoned duplicate SKU in the processor, even
|
||||
though duplication sounds like a validation problem. The reason is what information the check
|
||||
needs:
|
||||
|
||||
| Check | What it needs | Where it belongs |
|
||||
|---|---|---|
|
||||
| SKU matches a format | Just this row | Processor, filter (`null`) |
|
||||
| Price is positive | Just this row | Processor, filter (`null`) |
|
||||
| SKU is unique | Every other row already written | The database, via a constraint |
|
||||
|
||||
A processor can only see one item at a time (by design — that is what lets Spring Batch
|
||||
process items independently and in chunks). It cannot know a SKU is a duplicate without querying
|
||||
everything written so far, which defeats the point of a stateless per-item check and does not
|
||||
scale. The database already tracks this via the `UNIQUE` constraint on `PRODUCT.sku`
|
||||
([`schema.sql`](../src/main/resources/schema.sql)), so that is where the duplicate check
|
||||
actually happens — as a write failure, not a filtered row. What a write failure does to the
|
||||
rest of the chunk, and how the job recovers from it, is the rest of this article: [chapter 7](07-restartability.md) and [chapter 8](08-skip-vs-restart.md).
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `ItemProcessor` returning `null`: [Spring Batch reference — item processing](https://docs.spring.io/spring-batch/reference/processor.html) (`rel="nofollow"`).
|
||||
- Chaining processors with `CompositeItemProcessor` when validation and transformation both need
|
||||
their own class — not used in this module (one processor does both jobs here), but worth
|
||||
knowing before a single processor method grows past a screenful.
|
||||
|
||||
[Next: Launching and JobParameters →](05-launching-and-jobparameters.md)
|
||||
@@ -0,0 +1,65 @@
|
||||
# 5. Launching a job, and why restart is not a separate API
|
||||
|
||||
[← Previous](04-item-processor-as-filter.md) | [README](../README.md) | [Next: The JDBC writer and Java records →](06-jdbc-writer-and-records.md)
|
||||
|
||||
[`ImportRunner`](../src/main/java/com/ankurm/batch/runner/ImportRunner.java) launches the job
|
||||
with:
|
||||
|
||||
```java
|
||||
var params = new JobParametersBuilder()
|
||||
.addString("batch.run", "demo") // identifying
|
||||
.toJobParameters();
|
||||
try {
|
||||
var execution = jobOperator.start(job, params);
|
||||
} catch (JobInstanceAlreadyCompleteException e) {
|
||||
// already succeeded; nothing to do
|
||||
}
|
||||
```
|
||||
|
||||
`JobParametersBuilder` and `addString(...)` look exactly like Spring Batch 4 and 5. What changed
|
||||
underneath is the type it builds: `JobParameters` is now an immutable record holding a
|
||||
`Set<JobParameter<?>>`, and each `JobParameter<T>` is itself a record carrying its own name,
|
||||
value, type, and an `identifying` flag (default `true` for the `addX` methods used here). None of
|
||||
that shows up in this code — it matters if you ever construct a `JobParameters` by hand
|
||||
instead of through the builder, or serialize one.
|
||||
|
||||
## "Identifying" is the whole restart mechanism
|
||||
|
||||
A **JobInstance** is identified by a job name plus the set of parameters marked `identifying`.
|
||||
Calling `jobOperator.start(job, params)` with parameters that match an existing JobInstance does
|
||||
not create a second, independent run:
|
||||
|
||||
- If that instance's last execution is not `COMPLETED` (it `FAILED`, or never finished), Spring
|
||||
Batch creates a new `JobExecution` **against the same instance** and each step resumes from
|
||||
its own last-committed position. This is a restart, and it happens through the exact same
|
||||
`start()` call as a first attempt — see [chapter 7](07-restartability.md) for what
|
||||
"resumes from" means concretely.
|
||||
- If that instance's last execution `COMPLETED`, `start()` throws
|
||||
`JobInstanceAlreadyCompleteException` (unless the step allows re-running —
|
||||
`allowStartIfComplete`, used on `reportStep`; see [chapter 8](08-skip-vs-restart.md)).
|
||||
|
||||
There is a separate `JobOperator.restart(long executionId)` method for restarting by execution
|
||||
ID explicitly, useful for an operational tool that lists failed executions and lets someone pick
|
||||
one. This module does not need it: every run uses the same identifying parameter
|
||||
(`batch.run=demo`), so plain `start()` already does the right thing whether this is attempt one
|
||||
or attempt two. If you want a fresh JobInstance on every run instead — the common pattern
|
||||
for "run this daily" jobs — add a parameter that changes each time, typically a timestamp
|
||||
or an incrementer, and mark it identifying (the default).
|
||||
|
||||
## `spring.batch.job.enabled`
|
||||
|
||||
Spring Boot auto-configures a `JobLauncherApplicationRunner` that launches every `Job` bean it
|
||||
finds using empty parameters, on every application startup. This module turns that off
|
||||
(`spring.batch.job.enabled: false` in [`application.yml`](../src/main/resources/application.yml))
|
||||
because `ImportRunner` needs to control the parameters (the identifying `batch.run` value) and
|
||||
which profile's `Job` bean is active. Leaving both runners on would launch the job twice per
|
||||
startup with two different parameter sets. If you only ever need "run the one job on startup
|
||||
with no arguments," the auto-configured runner is simpler and this whole class is unnecessary.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `JobParameters` and identifying parameters: [Spring Batch reference — running a job](https://docs.spring.io/spring-batch/reference/job/running.html) (`rel="nofollow"`).
|
||||
- `spring.batch.job.name` for selecting which job the auto-configured runner launches, when you
|
||||
have more than one `Job` bean and want to keep using it.
|
||||
|
||||
[Next: The JDBC writer and Java records →](06-jdbc-writer-and-records.md)
|
||||
@@ -0,0 +1,59 @@
|
||||
# 6. The JDBC writer, and why it is not `beanMapped()`
|
||||
|
||||
[← Previous](05-launching-and-jobparameters.md) | [README](../README.md) | [Next: Restartability →](07-restartability.md)
|
||||
|
||||
`JdbcBatchItemWriterBuilder` offers two ways to map an item's fields to SQL parameters:
|
||||
`columnMapped()` (positional, via `Map`/`SqlParameterSource` keyed by column name) and
|
||||
`beanMapped()` (reflective, via `BeanPropertySqlParameterSource`, matching `:paramName` markers
|
||||
in the SQL to JavaBean getters). Most Spring Batch examples reach for `beanMapped()` because it
|
||||
needs the least code.
|
||||
|
||||
`Product` in this module is a Java record:
|
||||
|
||||
```java
|
||||
public record Product(String sku, String name, long priceCents) {}
|
||||
```
|
||||
|
||||
Its accessors are `sku()`, `name()`, `priceCents()` — no `get` prefix. Standard JavaBean
|
||||
introspection, which `BeanPropertySqlParameterSource` uses, looks for `getSku()`, `getName()`,
|
||||
`getPriceCents()`. Those do not exist on a record, so `beanMapped()` against a plain record either
|
||||
finds nothing to bind (leaving every parameter `NULL`) or fails outright, depending on the exact
|
||||
introspector version in play — not a mistake you want to discover from a table full of NULLs
|
||||
in production.
|
||||
|
||||
[`BatchConfig.productWriter`](../src/main/java/com/ankurm/batch/config/BatchConfig.java) sidesteps
|
||||
the question entirely with `itemPreparedStatementSetter`, setting each column explicitly:
|
||||
|
||||
```java
|
||||
new JdbcBatchItemWriterBuilder<Product>()
|
||||
.dataSource(jdbcTemplate.getDataSource())
|
||||
.sql("INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)")
|
||||
.itemPreparedStatementSetter((item, ps) -> {
|
||||
ps.setString(1, item.sku());
|
||||
ps.setString(2, item.name());
|
||||
ps.setLong(3, item.priceCents());
|
||||
})
|
||||
.assertUpdates(true)
|
||||
.build();
|
||||
```
|
||||
|
||||
More typing, zero ambiguity about what gets bound where, and it works identically whether the
|
||||
item type is a record, a plain class, or something with no JavaBean getters at all.
|
||||
|
||||
<blockquote style="background:#f4f5f7;border:1px solid #e2e5ea;border-left:4px solid #b7bec9;border-radius:6px;padding:16px 20px;"><strong>If you want <code>beanMapped()</code> with records anyway.</strong> Newer versions of Spring's <code>BeanWrapperImpl</code> (the machinery behind <code>BeanPropertySqlParameterSource</code>) have gained some record support in recent Spring Framework releases, but the safe rule is to check it against the exact Spring Framework version you are on rather than assume — the failure mode when it does not work is silent <code>NULL</code>s, not an exception, which is the worst kind of wrong.</blockquote>
|
||||
|
||||
`assertUpdates(true)` is worth keeping on deliberately: it makes the writer throw if a batch
|
||||
statement reports zero rows updated for any item, instead of silently accepting a no-op write.
|
||||
Combined with `itemPreparedStatementSetter`, a typo in a column name fails loudly at the first
|
||||
write attempt rather than producing a table that looks plausible but is missing a column's worth
|
||||
of data.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `JdbcBatchItemWriter` and its two mapping styles: [Spring Batch reference — item writers](https://docs.spring.io/spring-batch/reference/readers-and-writers/item-writer.html) (`rel="nofollow"`).
|
||||
- Java records and JavaBean introspection generally: this is not a Spring Batch quirk, it is how
|
||||
`java.beans.Introspector` has always worked, and it bites anywhere a library assumes
|
||||
`getX()`/`setX()` — Jackson, JPA, validation frameworks — each with its own answer
|
||||
for how much (if any) record support it has added.
|
||||
|
||||
[Next: Restartability →](07-restartability.md)
|
||||
@@ -0,0 +1,112 @@
|
||||
# 7. Restartability: what actually resumes, and from where
|
||||
|
||||
[← Previous](06-jdbc-writer-and-records.md) | [README](../README.md) | [Next: Skip vs. restart →](08-skip-vs-restart.md)
|
||||
|
||||
This chapter is the "run 1 fails, run 2 in a brand-new JVM finishes the job" demonstration
|
||||
[the article](https://ankurm.com/) leads with. Two real, separate `java -jar` invocations,
|
||||
against the same file-based H2 database, produced everything below — see
|
||||
[`scripts/capture-scenarios.sh`](../scripts/capture-scenarios.sh) for the exact commands.
|
||||
|
||||
## Run 1: the poisoned duplicate fails a whole chunk
|
||||
|
||||
`products-poison.csv` is identical to the clean fixture except row 47, where the SKU is changed
|
||||
to `ABC-0005` — a duplicate of row 5. Row 47 falls in chunk 5 (rows 41-50 at chunk size
|
||||
10). The `broken` profile's step has no fault tolerance configured, so the `UNIQUE` constraint
|
||||
violation on that insert fails the batch statement, which fails the chunk, which fails the step,
|
||||
which fails the job:
|
||||
|
||||
```console
|
||||
$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;
|
||||
STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT
|
||||
FAILED | 50 | 2 | 38 | 4 | 1
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/07-restart-run1-fails.txt`](output/07-restart-run1-fails.txt).
|
||||
Four chunks (rows 1-40, minus the 2 filtered by the processor — see
|
||||
[chapter 4](04-item-processor-as-filter.md) — = 38 written) committed successfully before
|
||||
chunk 5 rolled back entirely. The reader had read through row 50 by the time the writer failed
|
||||
(`READ_COUNT` 50), but none of chunk 5's rows are in the `PRODUCT` table — the transaction
|
||||
that would have written them rolled back along with everything else in that chunk.
|
||||
|
||||
## The fix, and the restart
|
||||
|
||||
Someone fixes the source data — here, changing row 47's SKU to something that is not a
|
||||
duplicate, in place, same line, same file:
|
||||
|
||||
```console
|
||||
sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv
|
||||
```
|
||||
|
||||
Then the exact same command that produced run 1 runs again, in a completely new JVM process,
|
||||
using the exact same identifying job parameter (`batch.run=demo`, see
|
||||
[chapter 5](05-launching-and-jobparameters.md)):
|
||||
|
||||
```console
|
||||
$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;
|
||||
STEP_EXECUTION_ID | JOB_EXECUTION_ID | STATUS | READ_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT
|
||||
1 | 1 | FAILED | 50 | 38 | 4 | 1
|
||||
2 | 2 | COMPLETED | 20 | 20 | 2 | 0
|
||||
|
||||
$ SELECT COUNT(*) FROM PRODUCT;
|
||||
COUNT(*)
|
||||
58
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/08-restart-run2-resumes.txt`](output/08-restart-run2-resumes.txt).
|
||||
Read that second row carefully: **`READ_COUNT` is 20, not 60.** The new execution did not start
|
||||
the CSV from row 1. It read only rows 41 through 60 — the two chunks that had not committed
|
||||
yet — wrote all 20 of them (the earlier two filtered rows were both in the first 40, so
|
||||
nothing left to filter here), and the job status flips to `COMPLETED`. `PRODUCT` ends up with
|
||||
58 rows: the 38 that survived run 1, plus these 20.
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 740 260" role="img" aria-label="Run 1 commits chunks 1 through 4 (rows 1-40) then fails on chunk 5; run 2, a fresh JVM against the same database, resumes at row 41 and reads only 20 more rows to finish.">
|
||||
<style>.h{font:600 12px sans-serif;fill:#1a1a1a}.c{font:11px sans-serif;fill:#4b5563}.m{font:11px monospace;fill:#1a1a1a}</style>
|
||||
<text x="20" y="24" class="h">Run 1 (JobExecution 1)</text>
|
||||
<rect x="20" y="40" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="34" y="64" class="m">1-10</text>
|
||||
<rect x="116" y="40" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="130" y="64" class="m">11-20</text>
|
||||
<rect x="212" y="40" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="226" y="64" class="m">21-30</text>
|
||||
<rect x="308" y="40" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="322" y="64" class="m">31-40</text>
|
||||
<rect x="404" y="40" width="88" height="40" rx="4" fill="#f7d9d3" stroke="#c56a54"/><text x="415" y="64" class="m">41-50 FAIL</text>
|
||||
<rect x="500" y="40" width="88" height="40" rx="4" fill="#f4f5f7" stroke="#b7bec9" stroke-dasharray="3 3"/><text x="512" y="64" class="c">never read</text>
|
||||
<text x="20" y="105" class="c">committed: rows 1-40 (38 written after 2 filtered) -- job FAILS at chunk 5, exit status FAILED</text>
|
||||
|
||||
<text x="20" y="150" class="h">Run 2 (JobExecution 2) -- fresh JVM, same database, corrected row 47</text>
|
||||
<rect x="20" y="166" width="380" height="40" rx="4" fill="#f4f5f7" stroke="#b7bec9" stroke-dasharray="3 3"/><text x="150" y="190" class="c">rows 1-40: already committed, NOT re-read</text>
|
||||
<rect x="404" y="166" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="415" y="190" class="m">41-50 OK</text>
|
||||
<rect x="500" y="166" width="88" height="40" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="511" y="190" class="m">51-60 OK</text>
|
||||
<text x="20" y="230" class="c">READ_COUNT for this execution = 20, not 60. COMPLETED. PRODUCT now has 58 rows total.</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
## Where the resume position actually lives
|
||||
|
||||
The reader's position is not something Spring Batch infers after the fact — it is saved,
|
||||
by the reader itself, into the step's `ExecutionContext` at every chunk commit, in the **same
|
||||
transaction** as the chunk's writes. `FlatFileItemReader` implements `ItemStream`
|
||||
(`org.springframework.batch.infrastructure.item.ItemStream`, see
|
||||
[`docs/output/04-two-executioncontext-classes.txt`](output/04-two-executioncontext-classes.txt)
|
||||
for its shape) and calls `update(ExecutionContext)` on every commit, storing the current line
|
||||
count. Because that update happens inside the chunk's transaction:
|
||||
|
||||
- If the chunk commits, the new line count is persisted right alongside the rows it wrote.
|
||||
- If the chunk rolls back (chunk 5, run 1), the line-count update rolls back too — the
|
||||
persisted position stays at the end of chunk 4, not partway through chunk 5.
|
||||
|
||||
That is the entire mechanism. There is no separate "resume point" concept to configure: it falls
|
||||
out of transactional chunk commits plus a reader that participates in `ItemStream`. Writing a
|
||||
custom reader that does not save its position to the `ExecutionContext` — a plain
|
||||
`ItemReader` with no `ItemStream` — means restarts always start that reader from scratch,
|
||||
which is sometimes exactly what you want (an idempotent reader) and sometimes a bug you will not
|
||||
notice until the first real failure in production.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `ItemStream` and the two-argument `open`/`update` contract:
|
||||
[Spring Batch reference — readers and ItemStream](https://docs.spring.io/spring-batch/reference/readers-and-writers/item-reader.html) (`rel="nofollow"`).
|
||||
- Configuring restart limits per step (`startLimit`, `allowStartIfComplete`):
|
||||
[chapter 8](08-skip-vs-restart.md), and the [Spring Batch reference on restart configuration](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/restart.html) (`rel="nofollow"`).
|
||||
- What happens if the job repository itself does not persist across the JVM restart shown above:
|
||||
[chapter 10](10-resourceless-vs-jdbc.md) — this entire chapter depends on it being real.
|
||||
|
||||
[Next: Skip vs. restart →](08-skip-vs-restart.md)
|
||||
@@ -0,0 +1,92 @@
|
||||
# 8. Skip vs. restart, and the two knobs that shape a restart
|
||||
|
||||
[← Previous](07-restartability.md) | [README](../README.md) | [Next: Corrections found while writing this →](09-corrections.md)
|
||||
|
||||
Failing the job and restarting after a fix (chapter 7) is one answer to a bad row. It is the
|
||||
right one when the row is genuinely wrong and someone needs to decide what to do about it. It is
|
||||
the wrong one when the bad row is expected background noise — a handful of duplicates in a
|
||||
million-row feed — and stopping the whole job for each one is not realistic.
|
||||
|
||||
The `skip` profile configures the same step with `faultTolerant().skip(...)` instead:
|
||||
|
||||
```java
|
||||
new StepBuilder("importStep", jobRepository)
|
||||
.<Product, Product>chunk(10)
|
||||
.transactionManager(transactionManager)
|
||||
.reader(productReader)
|
||||
.processor(productProcessor)
|
||||
.writer(productWriter)
|
||||
.faultTolerant()
|
||||
.skip(DataIntegrityViolationException.class)
|
||||
.skipLimit(3)
|
||||
.listener(skipListener())
|
||||
.build();
|
||||
```
|
||||
|
||||
Run against the same poisoned file used in chapter 7, in one pass, no restart needed:
|
||||
|
||||
```console
|
||||
$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;
|
||||
COMPLETED | 60 | 2 | 57 | 1 | 6 | 2
|
||||
|
||||
$ SELECT COUNT(*) FROM PRODUCT;
|
||||
COUNT(*)
|
||||
57
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/10-skip-instead-of-fail.txt`](output/10-skip-instead-of-fail.txt).
|
||||
The job reads all 60 rows in one execution, filters the same 2 as always, skips exactly the
|
||||
duplicate (`WRITE_SKIP_COUNT` 1), and completes with 57 rows written. Note `ROLLBACK_COUNT` is 2,
|
||||
not 0: when a fault-tolerant chunk's batch write fails, Spring Batch does not give up on the
|
||||
chunk — it rolls back once, then re-processes that chunk's items **one at a time** to find
|
||||
out which single item is the actual problem, so it can skip only that one and keep the rest. That
|
||||
retry-by-scanning is one extra rollback for the chunk containing the bad row; you can see it in
|
||||
the count.
|
||||
|
||||
## `DataIntegrityViolationException.class` catches `DuplicateKeyException` because of the hierarchy
|
||||
|
||||
The skip policy is registered against `DataIntegrityViolationException`, but the exception H2
|
||||
actually throws is `DuplicateKeyException`. This works because
|
||||
`DuplicateKeyException extends DataIntegrityViolationException` — Spring's JDBC exception
|
||||
translation maps H2's `JdbcBatchUpdateException` (a unique-constraint violation) to the more
|
||||
specific subclass, and a skip policy registered against the parent still matches it. Confirmed
|
||||
directly, not assumed:
|
||||
|
||||
```console
|
||||
thrown type : org.springframework.dao.DuplicateKeyException
|
||||
is a DataIntegrityViolationException? true
|
||||
```
|
||||
|
||||
(from [`docs/output/02-exception-hierarchy.txt`](output/02-exception-hierarchy.txt), produced by
|
||||
[`ProductValidatingProcessorTest.duplicateKeyExceptionIsADataIntegrityViolationException`](../src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java)).
|
||||
Registering skip policies against a broad parent type like `DataIntegrityViolationException`
|
||||
means you do not have to enumerate every specific subtype Spring's translation layer might
|
||||
produce — but it also means you are choosing to skip *any* data-integrity problem, not just
|
||||
duplicates. `skipLimit(3)` is the safety valve: past 3 skips in one execution, the step gives up
|
||||
and fails anyway, on the theory that "one bad row" and "the whole feed is corrupt" should not be
|
||||
handled identically.
|
||||
|
||||
## The two restart-shaping options on a step
|
||||
|
||||
`reportStep` uses one of these; the football-job example in the Spring Batch reference documents
|
||||
both well, and this module's `reportStep` is a direct, smaller version of that pattern:
|
||||
|
||||
- **`allowStartIfComplete(true)`** — by default, a step that already finished `COMPLETED`
|
||||
is skipped entirely on restart (the whole point of chapter 7 is not re-doing committed work).
|
||||
`reportStep` overrides that, because a status report should reflect the current state of the
|
||||
`PRODUCT` table on every run, including a restart, not just the first successful attempt.
|
||||
- **`startLimit(n)`** — not used in this module, but worth knowing: caps how many times a
|
||||
step may be *attempted* (not completed) before Spring Batch throws
|
||||
`StartLimitExceededException` instead of trying again. Useful for a step that does something
|
||||
non-idempotent enough that repeated automatic retries would make things worse — call out
|
||||
external systems, send notifications — where the right response to repeated failure is a
|
||||
human looking at it, not an unbounded retry loop.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- Skip and retry configuration in full, including `noSkip()` exclusions and combining skip with
|
||||
retry: [Spring Batch reference — configuring skip logic](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/configuring-skip.html) (`rel="nofollow"`).
|
||||
- `startLimit` and `allowStartIfComplete` with the football-job example this chapter borrows the
|
||||
shape of: [Spring Batch reference — configuring a step for restart](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/restart.html) (`rel="nofollow"`).
|
||||
|
||||
[Next: Corrections found while writing this →](09-corrections.md)
|
||||
@@ -0,0 +1,57 @@
|
||||
# 9. Corrections found while writing this
|
||||
|
||||
[← Previous](08-skip-vs-restart.md) | [README](../README.md) | [Next: Resourceless vs. JDBC-backed →](10-resourceless-vs-jdbc.md)
|
||||
|
||||
Per the verification discipline this repository follows, here is what was wrong in an earlier
|
||||
draft, and how it was caught — recorded honestly rather than silently fixed, because the
|
||||
mistake is often as informative as the correct answer.
|
||||
|
||||
## "CommandLineJobRunner was removed in 6.0" — wrong, it is deprecated
|
||||
|
||||
A community-written Spring Batch 6.0 migration guide describes `CommandLineJobRunner` as removed
|
||||
outright, in a list alongside classes that genuinely were deleted
|
||||
(`ChunkListenerSupport`, `JobExecutionListenerSupport`, and others). An early draft of this
|
||||
article repeated that claim. Checking the actual `spring-batch-core-6.0.5.jar` shows it is wrong:
|
||||
|
||||
```console
|
||||
$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated"
|
||||
public class org.springframework.batch.core.launch.support.CommandLineJobRunner
|
||||
...
|
||||
Deprecated: true
|
||||
RuntimeVisibleAnnotations:
|
||||
0: #505(#506=s#507,#508=Z#509)
|
||||
java.lang.Deprecated(
|
||||
since="6.0"
|
||||
forRemoval=true
|
||||
)
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/09-commandlinejobrunner-deprecated-not-removed.txt`](output/09-commandlinejobrunner-deprecated-not-removed.txt).
|
||||
The class is present, loadable, and functional in 6.0.5 — it carries
|
||||
`@Deprecated(since = "6.0", forRemoval = true)`, which means "stop using this, it will be
|
||||
deleted in a future release," not "this is already gone." `CommandLineJobOperator` is the
|
||||
replacement, and it does exist alongside it in the same package.
|
||||
|
||||
This is the article's own self-correction pass working as intended: a claim sourced from a
|
||||
third-party summary of a migration guide, not from a primary artifact, was treated as unverified
|
||||
until checked. The general rule this project follows (see the parent repository's process notes)
|
||||
is that a migration guide's own prose is not itself a primary source for "removed vs.
|
||||
deprecated" — the jar is.
|
||||
|
||||
## `RepeatStatus` package move, caught by the compiler rather than by reading docs
|
||||
|
||||
Writing [`BatchConfig.reportTasklet`](../src/main/java/com/ankurm/batch/config/BatchConfig.java),
|
||||
the first draft imported `org.springframework.batch.core.repeat.RepeatStatus` — the
|
||||
Spring Batch 5.x location, from memory and from older examples. `mvn -B -o compile` reported:
|
||||
|
||||
```
|
||||
[ERROR] .../BatchConfig.java:[93,57] package org.springframework.batch.core.repeat does not exist
|
||||
```
|
||||
|
||||
The fix was `org.springframework.batch.infrastructure.repeat.RepeatStatus`, matching the general
|
||||
package reorganization covered in [chapter 2](02-anatomy-of-a-job.md). No migration guide was
|
||||
consulted for this one; the compiler error was sufficient, which is the point — letting
|
||||
`javac` tell you where an API moved is faster and more reliable than trying to remember or look
|
||||
up a package reorganization table.
|
||||
|
||||
[Next: Resourceless vs. JDBC-backed →](10-resourceless-vs-jdbc.md)
|
||||
@@ -0,0 +1,71 @@
|
||||
# 10. Resourceless vs. JDBC-backed: the dependency that makes chapter 7 possible
|
||||
|
||||
[← Previous](09-corrections.md) | [README](../README.md) | [Next: Production checklist →](11-production-checklist.md)
|
||||
|
||||
Everything in [chapter 7](07-restartability.md) depends on one dependency choice that is easy to
|
||||
get wrong on Spring Boot 4.1, because getting it wrong does not produce an error — the
|
||||
application starts, the job runs, nothing complains.
|
||||
|
||||
Spring Boot 4.1 split Batch autoconfiguration into separate modules rather than one monolith:
|
||||
|
||||
| Starter | Autoconfiguration class | What it provides |
|
||||
|---|---|---|
|
||||
| `spring-boot-starter-batch` | `BatchAutoConfiguration` | Batch infrastructure basics; a **resourceless** (in-memory, non-persistent) `JobRepository` if nothing more specific is configured |
|
||||
| `spring-boot-starter-batch-jdbc` | `BatchJdbcAutoConfiguration` | A real JDBC-backed `JobRepository`, plus `spring.batch.jdbc.initialize-schema` to create the `BATCH_*` tables |
|
||||
|
||||
`spring-boot-starter-batch-jdbc` (this module's actual dependency, in
|
||||
[`pom.xml`](../pom.xml)) pulls in `spring-boot-starter-batch` transitively, so **both**
|
||||
autoconfiguration classes are normally on the classpath together, and `BatchJdbcAutoConfiguration`
|
||||
wins by providing the more specific `JobRepository` bean. Nothing about compiling or starting the
|
||||
application tells you which one actually won.
|
||||
|
||||
## Proving it, by turning the JDBC half off
|
||||
|
||||
`--spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration` removes the JDBC-backed
|
||||
`JobRepository` from the picture, leaving `BatchAutoConfiguration`'s resourceless one as the only
|
||||
candidate. Same job, same identifying job parameters, same `PRODUCT` table, run twice, in two
|
||||
separate JVMs:
|
||||
|
||||
```console
|
||||
-- run 1 --
|
||||
REPORT: 58 products now in the PRODUCT table
|
||||
JOB FINISHED: status=COMPLETED exitCode=COMPLETED
|
||||
|
||||
-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --
|
||||
org.springframework.dao.DuplicateKeyException: ... Unique index or primary key violation ...
|
||||
VALUES ( /* 1 */ 'ABC-0001' )
|
||||
JOB FINISHED: status=FAILED exitCode=FAILED
|
||||
```
|
||||
|
||||
Full transcript: [`docs/output/11-resourceless-forgets-everything.txt`](output/11-resourceless-forgets-everything.txt).
|
||||
Run 2 does not throw `JobInstanceAlreadyCompleteException`, the exception you would expect for
|
||||
re-running an already-completed job (see [chapter 5](05-launching-and-jobparameters.md)). It does
|
||||
not know there was a run 1 at all. The resourceless `JobRepository` lives only in that JVM's
|
||||
heap; when the process exits, every `JobInstance`, `JobExecution`, and `StepExecution` it ever
|
||||
recorded goes with it. Run 2 starts `productImportJob` as if for the first time, tries to insert
|
||||
`ABC-0001` again, and collides with what run 1 already committed to the `PRODUCT` table —
|
||||
a table that, unlike the job repository, *is* backed by the same file-based H2 database both
|
||||
times.
|
||||
|
||||
This is the practical failure mode: **the job repository's persistence and your own application
|
||||
data's persistence are two separate decisions**, and it is entirely possible to get the second
|
||||
one right (a real database, a real table) while getting the first one wrong (resourceless,
|
||||
because `-jdbc` was left off a dependency list or excluded by a profile without realizing what it
|
||||
carried). The symptom is not a startup error. It is "my restart didn't restart" or "why did this
|
||||
job double-insert everything," discovered in whatever environment happens to restart the JVM
|
||||
between runs — which, for anything running in a container orchestrator, is any redeploy,
|
||||
crash-restart, or scale-down/scale-up cycle.
|
||||
|
||||
<blockquote style="background:#f4f5f7;border:1px solid #e2e5ea;border-left:4px solid #b7bec9;border-radius:6px;padding:16px 20px;"><strong>How to check which one you actually have.</strong> Query <code>information_schema.tables</code> (or your database's equivalent) for a table named <code>BATCH_JOB_INSTANCE</code> after your application starts. If it is not there, you have the resourceless job repository, whatever your dependency list looks like on paper — check for an exclusion in <code>spring.autoconfigure.exclude</code>, a conflicting <code>JobRepository</code> bean of your own, or simply a missing <code>-jdbc</code>/<code>-mongodb</code> starter.</blockquote>
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `BatchProperties` and `BatchJdbcProperties`, including `spring.batch.job.enabled`,
|
||||
`spring.batch.jdbc.initialize-schema`, and `spring.batch.jdbc.table-prefix`: the
|
||||
`spring-configuration-metadata.json` inside `spring-boot-batch-4.1.1.jar` and
|
||||
`spring-boot-batch-jdbc-4.1.1.jar` is the authoritative list, more current than any blog post
|
||||
including this one.
|
||||
- MongoDB-backed job repositories follow the identical pattern via
|
||||
`spring-boot-starter-batch-data-mongodb`, not covered by this module.
|
||||
|
||||
[Next: Production checklist →](11-production-checklist.md)
|
||||
@@ -0,0 +1,47 @@
|
||||
# 11. Production checklist
|
||||
|
||||
[← Previous](10-resourceless-vs-jdbc.md) | [README](../README.md)
|
||||
|
||||
A short list, aimed at the gap between "this demo works" and "this is safe to point at a real
|
||||
feed." Each item links back to the chapter that explains the why.
|
||||
|
||||
- **Confirm the job repository is actually persistent** — query for `BATCH_JOB_INSTANCE`
|
||||
after startup, don't assume from the dependency list. [Chapter 10](10-resourceless-vs-jdbc.md).
|
||||
- **Pick chunk size from measurement, not habit.** Smaller costs more round trips; larger costs
|
||||
more rework per failure. [Chapter 3](03-chunk-oriented-processing.md).
|
||||
- **Decide skip vs. fail-and-restart per failure type, not per job.** A step can have a
|
||||
`skipLimit` for genuinely expected bad rows and still fail hard past that limit — the two
|
||||
are not mutually exclusive. [Chapter 8](08-skip-vs-restart.md).
|
||||
- **Use `chunk(int).transactionManager(tx)`, not `chunk(int, tx)`,** unless you specifically need
|
||||
the legacy `SimpleStepBuilder` for something the new model does not yet cover.
|
||||
[Chapter 2](02-anatomy-of-a-job.md).
|
||||
- **Do not use `beanMapped()` against a Java record** without checking your exact Spring
|
||||
Framework version's record support; prefer `itemPreparedStatementSetter` when in doubt.
|
||||
[Chapter 6](06-jdbc-writer-and-records.md).
|
||||
- **Give every step you want to always re-run on restart `allowStartIfComplete(true)`
|
||||
explicitly** — a reporting or notification step that silently gets skipped after a
|
||||
restart is a surprising, hard-to-notice gap. [Chapter 8](08-skip-vs-restart.md).
|
||||
- **A custom `ItemReader` that does not implement `ItemStream` restarts from scratch every time**,
|
||||
which is fine for an idempotent reader and a bug for anything else — check this
|
||||
deliberately rather than by finding out during an incident. [Chapter 7](07-restartability.md).
|
||||
- **`spring.batch.job.enabled=false` if you launch jobs yourself** through `JobOperator`, so the
|
||||
auto-configured runner does not also try. [Chapter 5](05-launching-and-jobparameters.md).
|
||||
- **Delete or secure any diagnostic endpoint you add while building this out.** This module has
|
||||
none, but the pattern (an endpoint dumping live `JobExplorer` state) is common enough to call
|
||||
out: it is invaluable while developing and a liability left in production.
|
||||
|
||||
## Should you even build a custom batch job for this?
|
||||
|
||||
Sometimes the honest answer is no. If the volume is small enough to fit in a single request-response
|
||||
cycle and does not need to survive a crash mid-way, a scheduled `@Component` method with its own
|
||||
try/catch is less machinery than a full Job/Step/JobRepository setup, and easier for the next
|
||||
person to read. Spring Batch earns its complexity when you actually need the things this article
|
||||
demonstrates — durable progress tracking, transactional chunking, a real restart story
|
||||
— not by default just because the word "batch" is in the requirements.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Spring Batch reference documentation](https://docs.spring.io/spring-batch/reference/) (`rel="nofollow"`)
|
||||
- [What's new in Spring Batch 6](https://docs.spring.io/spring-batch/reference/whatsnew.html) (`rel="nofollow"`)
|
||||
- [Spring Batch 6.0 Migration Guide](https://github.com/spring-projects/spring-batch/wiki/Spring-Batch-6.0-Migration-Guide) (`rel="nofollow"`, and see [chapter 9](09-corrections.md) for where this specific document was wrong)
|
||||
- [Spring Boot 4.1.0 release announcement](https://spring.io/blog/2026/06/10/spring-boot-4/) (`rel="nofollow"`)
|
||||
@@ -0,0 +1,10 @@
|
||||
# The processor as a filter: null means skip, not fail
|
||||
|
||||
input : Product[sku=ABC-0001, name=Widget 1, priceCents=1001]
|
||||
result : Product[sku=ABC-0001, name=Widget 1, priceCents=1001]
|
||||
|
||||
input : Product[sku=abc-0012, name=Widget 12, priceCents=1012] <- lowercase sku, fails ^[A-Z]{3}-\d{4}$
|
||||
result : null
|
||||
|
||||
input : Product[sku=ABC-0033, name=Widget 33, priceCents=0] <- priceCents is zero
|
||||
result : null
|
||||
@@ -0,0 +1,9 @@
|
||||
# Why skip(DataIntegrityViolationException.class) also catches the duplicate-key case
|
||||
|
||||
thrown type : org.springframework.dao.DuplicateKeyException
|
||||
is a DataIntegrityViolationException? true
|
||||
|
||||
H2's JdbcBatchUpdateException on a UNIQUE-constraint violation is translated by
|
||||
Spring's SQLExceptionSubclassTranslator into DuplicateKeyException, which extends
|
||||
DataIntegrityViolationException. A skip policy configured against the parent class
|
||||
catches the subclass too -- see docs/07-restartability.md and docs/08-skip-vs-restart.md.
|
||||
@@ -0,0 +1,19 @@
|
||||
# javap org.springframework.batch.core.step.builder.StepBuilder (spring-batch-core 6.0.5)
|
||||
|
||||
$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.step.builder.StepBuilder
|
||||
Compiled from "StepBuilder.java"
|
||||
public class org.springframework.batch.core.step.builder.StepBuilder extends org.springframework.batch.core.step.builder.StepBuilderHelper<org.springframework.batch.core.step.builder.StepBuilder> {
|
||||
public org.springframework.batch.core.step.builder.StepBuilder(org.springframework.batch.core.repository.JobRepository);
|
||||
public org.springframework.batch.core.step.builder.StepBuilder(java.lang.String, org.springframework.batch.core.repository.JobRepository);
|
||||
public org.springframework.batch.core.step.builder.TaskletStepBuilder tasklet(org.springframework.batch.core.step.tasklet.Tasklet, org.springframework.transaction.PlatformTransactionManager);
|
||||
public org.springframework.batch.core.step.builder.TaskletStepBuilder tasklet(org.springframework.batch.core.step.tasklet.Tasklet);
|
||||
public <I, O> org.springframework.batch.core.step.builder.SimpleStepBuilder<I, O> chunk(int, org.springframework.transaction.PlatformTransactionManager);
|
||||
public <I, O> org.springframework.batch.core.step.builder.ChunkOrientedStepBuilder<I, O> chunk(int);
|
||||
public <I, O> org.springframework.batch.core.step.builder.SimpleStepBuilder<I, O> chunk(org.springframework.batch.infrastructure.repeat.CompletionPolicy, org.springframework.transaction.PlatformTransactionManager);
|
||||
public org.springframework.batch.core.step.builder.PartitionStepBuilder partitioner(java.lang.String, org.springframework.batch.core.partition.Partitioner);
|
||||
public org.springframework.batch.core.step.builder.PartitionStepBuilder partitioner(org.springframework.batch.core.step.Step);
|
||||
public org.springframework.batch.core.step.builder.JobStepBuilder job(org.springframework.batch.core.job.Job);
|
||||
public org.springframework.batch.core.step.builder.FlowStepBuilder flow(org.springframework.batch.core.job.flow.Flow);
|
||||
protected org.springframework.batch.core.step.builder.StepBuilder self();
|
||||
protected org.springframework.batch.core.step.builder.StepBuilderHelper self();
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
# javap: two ExecutionContext classes in two different packages
|
||||
|
||||
$ javap -cp spring-batch-infrastructure-6.0.5.jar org.springframework.batch.infrastructure.item.ExecutionContext
|
||||
Compiled from "ExecutionContext.java"
|
||||
public class org.springframework.batch.infrastructure.item.ExecutionContext implements java.io.Serializable {
|
||||
public org.springframework.batch.infrastructure.item.ExecutionContext();
|
||||
public org.springframework.batch.infrastructure.item.ExecutionContext(java.util.Map<java.lang.String, java.lang.Object>);
|
||||
public org.springframework.batch.infrastructure.item.ExecutionContext(org.springframework.batch.infrastructure.item.ExecutionContext);
|
||||
public void putString(java.lang.String, java.lang.String);
|
||||
public void putLong(java.lang.String, long);
|
||||
public void putInt(java.lang.String, int);
|
||||
public void putDouble(java.lang.String, double);
|
||||
public void put(java.lang.String, java.lang.Object);
|
||||
public boolean isDirty();
|
||||
public java.lang.String getString(java.lang.String);
|
||||
public java.lang.String getString(java.lang.String, java.lang.String);
|
||||
public long getLong(java.lang.String);
|
||||
public long getLong(java.lang.String, long);
|
||||
public int getInt(java.lang.String);
|
||||
public int getInt(java.lang.String, int);
|
||||
public double getDouble(java.lang.String);
|
||||
public double getDouble(java.lang.String, double);
|
||||
public java.lang.Object get(java.lang.String);
|
||||
public <V> V get(java.lang.String, java.lang.Class<V>);
|
||||
public <V> V get(java.lang.String, java.lang.Class<V>, V);
|
||||
public boolean isEmpty();
|
||||
public void clearDirtyFlag();
|
||||
public java.util.Set<java.util.Map$Entry<java.lang.String, java.lang.Object>> entrySet();
|
||||
public java.util.Map<java.lang.String, java.lang.Object> toMap();
|
||||
public boolean containsKey(java.lang.String);
|
||||
public java.lang.Object remove(java.lang.String);
|
||||
public boolean containsValue(java.lang.Object);
|
||||
public boolean equals(java.lang.Object);
|
||||
public int hashCode();
|
||||
public java.lang.String toString();
|
||||
public int size();
|
||||
}
|
||||
|
||||
$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.repository.persistence.ExecutionContext
|
||||
Compiled from "ExecutionContext.java"
|
||||
public final class org.springframework.batch.core.repository.persistence.ExecutionContext extends java.lang.Record {
|
||||
public org.springframework.batch.core.repository.persistence.ExecutionContext(java.util.Map<java.lang.String, java.lang.Object>, boolean);
|
||||
public final java.lang.String toString();
|
||||
public final int hashCode();
|
||||
public final boolean equals(java.lang.Object);
|
||||
public java.util.Map<java.lang.String, java.lang.Object> map();
|
||||
public boolean dirty();
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
# The happy path: 60 rows in, 58 products out
|
||||
|
||||
$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \
|
||||
--import.file=file:scenario-data/clean-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db
|
||||
|
||||
2026-09-13T06:23:05.509Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep]
|
||||
2026-09-13T06:23:05.530Z WARN 5242 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$
|
||||
2026-09-13T06:23:05.543Z WARN 5242 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0
|
||||
2026-09-13T06:23:05.570Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 61ms
|
||||
2026-09-13T06:23:05.590Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep]
|
||||
REPORT: 58 products now in the PRODUCT table
|
||||
2026-09-13T06:23:05.600Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 13ms
|
||||
JOB FINISHED: status=COMPLETED exitCode=COMPLETED
|
||||
|
||||
$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;
|
||||
READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT
|
||||
60 | 2 | 58 | 6
|
||||
(1 row, 21 ms)
|
||||
@@ -0,0 +1,23 @@
|
||||
# Run 1: the poisoned duplicate at row 47 fails the whole chunk
|
||||
|
||||
$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \
|
||||
--import.file=file:scenario-data/restart-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db
|
||||
|
||||
2026-09-13T06:23:09.122Z INFO 5302 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep]
|
||||
2026-09-13T06:23:09.162Z WARN 5302 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$
|
||||
2026-09-13T06:23:09.176Z WARN 5302 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0
|
||||
org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement:
|
||||
Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement:
|
||||
2026-09-13T06:23:09.205Z INFO 5302 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 85ms
|
||||
JOB FINISHED: status=FAILED exitCode=FAILED
|
||||
|
||||
$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;
|
||||
STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT
|
||||
FAILED | 50 | 2 | 38 | 4 | 1
|
||||
(1 row, 21 ms)
|
||||
|
||||
$ SELECT COUNT(*) FROM PRODUCT;
|
||||
COUNT(*)
|
||||
38
|
||||
(1 row, 20 ms)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Run 2: a brand-new JVM, the SAME job parameters, the corrected file
|
||||
|
||||
sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change
|
||||
|
||||
$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \
|
||||
--import.file=file:scenario-data/restart-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file
|
||||
|
||||
2026-09-13T06:23:13.205Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep]
|
||||
2026-09-13T06:23:13.232Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 28ms
|
||||
2026-09-13T06:23:13.240Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep]
|
||||
REPORT: 58 products now in the PRODUCT table
|
||||
2026-09-13T06:23:13.246Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 6ms
|
||||
JOB FINISHED: status=COMPLETED exitCode=COMPLETED
|
||||
|
||||
$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;
|
||||
STEP_EXECUTION_ID | JOB_EXECUTION_ID | STATUS | READ_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT
|
||||
1 | 1 | FAILED | 50 | 38 | 4 | 1
|
||||
2 | 2 | COMPLETED | 20 | 20 | 2 | 0
|
||||
(2 rows, 22 ms)
|
||||
|
||||
$ SELECT COUNT(*) FROM PRODUCT;
|
||||
COUNT(*)
|
||||
58
|
||||
(1 row, 19 ms)
|
||||
@@ -0,0 +1,23 @@
|
||||
# javap: CommandLineJobRunner is deprecated (forRemoval), not removed, in 6.0.5
|
||||
|
||||
$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated"
|
||||
public class org.springframework.batch.core.launch.support.CommandLineJobRunner
|
||||
minor version: 0
|
||||
major version: 61
|
||||
flags: (0x0021) ACC_PUBLIC, ACC_SUPER
|
||||
--
|
||||
#503 = Utf8 Deprecated
|
||||
#504 = Utf8 RuntimeVisibleAnnotations
|
||||
#505 = Utf8 Ljava/lang/Deprecated;
|
||||
#506 = Utf8 since
|
||||
#507 = Utf8 6.0
|
||||
#508 = Utf8 forRemoval
|
||||
--
|
||||
Deprecated: true
|
||||
RuntimeVisibleAnnotations:
|
||||
0: #505(#506=s#507,#508=Z#509)
|
||||
java.lang.Deprecated(
|
||||
since="6.0"
|
||||
forRemoval=true
|
||||
)
|
||||
--
|
||||
@@ -0,0 +1,27 @@
|
||||
# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES
|
||||
|
||||
$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \
|
||||
--import.file=file:scenario-data/skip-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db
|
||||
|
||||
2026-09-13T06:23:17.111Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep]
|
||||
2026-09-13T06:23:17.139Z WARN 5469 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$
|
||||
2026-09-13T06:23:17.150Z WARN 5469 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0
|
||||
Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement:
|
||||
Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement:
|
||||
SKIPPED on write: ABC-0005 (DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement:
|
||||
2026-09-13T06:23:17.235Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 124ms
|
||||
2026-09-13T06:23:17.247Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep]
|
||||
REPORT: 57 products now in the PRODUCT table
|
||||
2026-09-13T06:23:17.256Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 9ms
|
||||
JOB FINISHED: status=COMPLETED exitCode=COMPLETED
|
||||
|
||||
$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;
|
||||
STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | WRITE_SKIP_COUNT | COMMIT_COUNT | ROLLBACK_COUNT
|
||||
COMPLETED | 60 | 2 | 57 | 1 | 6 | 2
|
||||
(1 row, 22 ms)
|
||||
|
||||
$ SELECT COUNT(*) FROM PRODUCT;
|
||||
COUNT(*)
|
||||
57
|
||||
(1 row, 19 ms)
|
||||
@@ -0,0 +1,21 @@
|
||||
# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory
|
||||
|
||||
$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file=file:scenario-data/resourceless-demo/input.csv \
|
||||
--spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db
|
||||
|
||||
-- run 1 --
|
||||
REPORT: 58 products now in the PRODUCT table
|
||||
JOB FINISHED: status=COMPLETED exitCode=COMPLETED
|
||||
|
||||
-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --
|
||||
org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 1 */ 'ABC-0001' )"; SQL statement:
|
||||
Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 1 */ 'ABC-0001' )"; SQL statement:
|
||||
JOB FINISHED: status=FAILED exitCode=FAILED
|
||||
|
||||
No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea
|
||||
run 1 ever happened, so it tries the whole job again and collides with what run 1 already
|
||||
wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration
|
||||
used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead,
|
||||
which ImportRunner treats as "nothing to do" -- see docs/07-restartability.md.
|
||||
@@ -0,0 +1,62 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>spring-batch</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<name>spring-batch</name>
|
||||
<description>Spring Batch on Boot 4.1: jobs, steps, chunk processing and restartability</description>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<!-- Brings spring-boot-starter-batch (BatchAutoConfiguration, resourceless by default)
|
||||
plus spring-boot-starter-batch-jdbc (BatchJdbcAutoConfiguration: a real JDBC-backed
|
||||
JobRepository and the schema initializer). Without the -jdbc half, everything in this
|
||||
module still compiles and runs; it just forgets every job the moment the JVM exits. -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-batch-jdbc</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-jdbc</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- File-based H2, not in-memory: the whole point of this module is that job metadata and
|
||||
the PRODUCT table survive one JVM exiting and another one starting. -->
|
||||
<dependency>
|
||||
<groupId>com.h2database</groupId>
|
||||
<artifactId>h2</artifactId>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
Executable
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Captures real javap output pinning the Spring Batch 6.0 API facts this article relies on:
|
||||
# the ChunkOrientedStepBuilder that chunk(int) now returns, the two different ExecutionContext
|
||||
# classes, the package move of RepeatStatus, and CommandLineJobRunner's deprecation (NOT removal
|
||||
# -- an earlier draft trusted a migration-guide summary that said "removed" and this transcript
|
||||
# is what caught the error; see docs/09-corrections.md).
|
||||
#
|
||||
# ./scripts/capture-javap.sh
|
||||
#
|
||||
# Needs the spring-batch-core and spring-batch-infrastructure 6.0.5 jars in the local Maven repo
|
||||
# (a normal `mvn -B -o package` in this module already pulls them in).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
CORE=$(find "$HOME/.m2" -name 'spring-batch-core-6.0.5.jar' | head -1)
|
||||
INFRA=$(find "$HOME/.m2" -name 'spring-batch-infrastructure-6.0.5.jar' | head -1)
|
||||
: "${CORE:?spring-batch-core-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}"
|
||||
: "${INFRA:?spring-batch-infrastructure-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}"
|
||||
|
||||
WORK=$(mktemp -d)
|
||||
mkdir -p "$WORK/core" "$WORK/infra"
|
||||
unzip -o -q "$CORE" -d "$WORK/core"
|
||||
unzip -o -q "$INFRA" -d "$WORK/infra"
|
||||
|
||||
mkdir -p docs/output
|
||||
|
||||
{
|
||||
echo "# javap org.springframework.batch.core.step.builder.StepBuilder (spring-batch-core 6.0.5)"
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.step.builder.StepBuilder'
|
||||
javap -cp "$WORK/core" org.springframework.batch.core.step.builder.StepBuilder
|
||||
} > docs/output/03-stepbuilder-chunk-overloads.txt
|
||||
|
||||
{
|
||||
echo "# javap: two ExecutionContext classes in two different packages"
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-infrastructure-6.0.5.jar org.springframework.batch.infrastructure.item.ExecutionContext'
|
||||
javap -cp "$WORK/infra" org.springframework.batch.infrastructure.item.ExecutionContext
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.repository.persistence.ExecutionContext'
|
||||
javap -cp "$WORK/core" org.springframework.batch.core.repository.persistence.ExecutionContext
|
||||
} > docs/output/04-two-executioncontext-classes.txt
|
||||
|
||||
{
|
||||
echo "# javap: CommandLineJobRunner is deprecated (forRemoval), not removed, in 6.0.5"
|
||||
echo
|
||||
echo '$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated"'
|
||||
javap -verbose -cp "$WORK/core" org.springframework.batch.core.launch.support.CommandLineJobRunner \
|
||||
| grep -A3 '^public class\|Deprecated' | head -20
|
||||
} > docs/output/09-commandlinejobrunner-deprecated-not-removed.txt
|
||||
|
||||
rm -rf "$WORK"
|
||||
echo "wrote docs/output/03-stepbuilder-chunk-overloads.txt"
|
||||
echo "wrote docs/output/04-two-executioncontext-classes.txt"
|
||||
echo "wrote docs/output/09-commandlinejobrunner-deprecated-not-removed.txt"
|
||||
Executable
+152
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs the three end-to-end scenarios this article is built on and writes what actually
|
||||
# happened -- console output plus the real BATCH_STEP_EXECUTION and PRODUCT rows -- to
|
||||
# docs/output/. Nothing here is retyped; the numbers in the article come out of these files.
|
||||
#
|
||||
# ./scripts/capture-scenarios.sh
|
||||
#
|
||||
# Needs target/spring-batch-1.0.0.jar (run `mvn -B -o -DskipTests package` first) and a JDK 25.
|
||||
# Each scenario gets its own file-based H2 database under scenario-data/<name>/ so the three
|
||||
# runs never interfere with each other.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
JAR=target/spring-batch-1.0.0.jar
|
||||
: "${JAR:?run mvn -B -o -DskipTests package first}"
|
||||
H2_JAR=$(find "$HOME/.m2" -name 'h2-2.4.240.jar' | head -1)
|
||||
mkdir -p docs/output
|
||||
rm -rf scenario-data
|
||||
mkdir -p scenario-data
|
||||
|
||||
query() {
|
||||
# $1 = db dir, $2 = sql
|
||||
java -cp "$H2_JAR" org.h2.tools.Shell -url "jdbc:h2:file:${PWD}/$1/db" -user sa -password "" -sql "$2" 2>/dev/null \
|
||||
| grep -v '^$'
|
||||
}
|
||||
|
||||
run_scenario() {
|
||||
# $1 = name, $2 = profile, $3 = source csv
|
||||
local name="$1" profile="$2" src="$3" dir
|
||||
dir="scenario-data/$1"
|
||||
mkdir -p "$dir"
|
||||
cp "$src" "$dir/input.csv"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active="$profile" \
|
||||
--import.file="file:${dir}/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/${dir}/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|SKIPPED on write|DuplicateKeyException:|WARN.*ProductValidatingProcessor|Executing step|executed in" || true
|
||||
}
|
||||
|
||||
echo "== scenario: clean (05-happy-path.txt)"
|
||||
{
|
||||
echo "# The happy path: 60 rows in, 58 products out"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --import.file=file:scenario-data/clean-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db'
|
||||
echo
|
||||
run_scenario clean-demo clean src/main/resources/data/products.csv
|
||||
echo
|
||||
echo '$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/clean-demo "SELECT READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
} > docs/output/05-happy-path.txt
|
||||
|
||||
echo "== scenario: broken, run 1 (07-restart-run1-fails.txt)"
|
||||
{
|
||||
echo "# Run 1: the poisoned duplicate at row 47 fails the whole chunk"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db'
|
||||
echo
|
||||
run_scenario restart-demo broken src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/restart-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/07-restart-run1-fails.txt
|
||||
|
||||
echo "== fixing the duplicate in place (same file, same line count) =="
|
||||
sed -i '48s/^ABC-0005,Widget 47,1047$/ABC-0999,Widget 47,1047/' scenario-data/restart-demo/input.csv
|
||||
grep -n "Widget 47" scenario-data/restart-demo/input.csv
|
||||
|
||||
echo "== scenario: broken, run 2 = restart in a fresh JVM (07-restart-run2-resumes.txt)"
|
||||
{
|
||||
echo "# Run 2: a brand-new JVM, the SAME job parameters, the corrected file"
|
||||
echo
|
||||
echo "sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file'
|
||||
echo
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=broken \
|
||||
--import.file="file:scenario-data/restart-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/restart-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|Executing step|executed in" || true
|
||||
echo
|
||||
echo '$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;'
|
||||
query scenario-data/restart-demo "SELECT STEP_EXECUTION_ID, JOB_EXECUTION_ID, STATUS, READ_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep' ORDER BY STEP_EXECUTION_ID;"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/08-restart-run2-resumes.txt
|
||||
|
||||
echo "== scenario: skip (10-skip-instead-of-fail.txt)"
|
||||
{
|
||||
echo "# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \'
|
||||
echo ' --import.file=file:scenario-data/skip-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db'
|
||||
echo
|
||||
run_scenario skip-demo skip src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/skip-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, WRITE_SKIP_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/skip-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/10-skip-instead-of-fail.txt
|
||||
|
||||
echo "== scenario: resourceless (11-resourceless-forgets-everything.txt)"
|
||||
rm -rf scenario-data/resourceless-demo
|
||||
mkdir -p scenario-data/resourceless-demo
|
||||
cp src/main/resources/data/products.csv scenario-data/resourceless-demo/input.csv
|
||||
{
|
||||
echo "# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \'
|
||||
echo ' --import.file=file:scenario-data/resourceless-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db'
|
||||
echo
|
||||
echo "-- run 1 --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED" || true
|
||||
echo
|
||||
echo "-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|DuplicateKeyException:" | head -3 || true
|
||||
echo
|
||||
echo "No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea"
|
||||
echo "run 1 ever happened, so it tries the whole job again and collides with what run 1 already"
|
||||
echo "wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration"
|
||||
echo "used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead,"
|
||||
echo "which ImportRunner treats as \"nothing to do\" -- see docs/07-restartability.md."
|
||||
} > docs/output/11-resourceless-forgets-everything.txt
|
||||
|
||||
echo
|
||||
echo "docs/output:"
|
||||
ls -1 docs/output
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates every file under docs/output/.
|
||||
#
|
||||
# ./scripts/run-all.sh
|
||||
#
|
||||
# Needs a JDK 25 and Maven 3.9, with dependencies already resolved once online (mvn needs a
|
||||
# networked run before -o works for spring-boot:run / surefire -- see companion-repo notes).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo "== unit tests (transcripts 01-02)"
|
||||
mvn -B -o test
|
||||
|
||||
echo "== javap transcripts (03, 04, 09)"
|
||||
./scripts/capture-javap.sh
|
||||
|
||||
echo "== package"
|
||||
mvn -B -o -DskipTests package
|
||||
|
||||
echo "== end-to-end scenarios (05, 07, 08, 10, 11)"
|
||||
./scripts/capture-scenarios.sh
|
||||
|
||||
echo
|
||||
echo "docs/output:"
|
||||
ls -1 docs/output
|
||||
Executable
+28
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs one scenario against its own external H2 file database and CSV drop file (both under
|
||||
# scenario-data/<name>/, outside target/ so a rebuild never touches them -- see
|
||||
# docs/07-restartability.md for why that matters: `mvn spring-boot:run` re-copies
|
||||
# src/main/resources over target/classes before every run, which would silently undo an
|
||||
# in-place "fix" to a classpath resource between two runs of the same scenario.
|
||||
#
|
||||
# ./scripts/run-scenario.sh <name> <profile> <source-csv>
|
||||
#
|
||||
# Reuses the packaged jar (target/spring-batch-1.0.0.jar); run `mvn -B -o -DskipTests package`
|
||||
# first if it is missing or stale.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
NAME="${1:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
PROFILE="${2:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
SRC_CSV="${3:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
|
||||
DIR="scenario-data/${NAME}"
|
||||
mkdir -p "$DIR"
|
||||
if [ ! -f "$DIR/input.csv" ]; then
|
||||
cp "$SRC_CSV" "$DIR/input.csv"
|
||||
fi
|
||||
|
||||
java -jar target/spring-batch-1.0.0.jar \
|
||||
--spring.profiles.active="$PROFILE" \
|
||||
--import.file="file:${DIR}/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/${DIR}/db"
|
||||
@@ -0,0 +1,11 @@
|
||||
package com.ankurm.batch;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication
|
||||
public class BatchDemoApplication {
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(BatchDemoApplication.class, args);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
package com.ankurm.batch.config;
|
||||
|
||||
import com.ankurm.batch.domain.Product;
|
||||
import com.ankurm.batch.processing.ProductValidatingProcessor;
|
||||
import org.springframework.batch.core.job.Job;
|
||||
import org.springframework.batch.core.job.builder.JobBuilder;
|
||||
import org.springframework.batch.core.listener.SkipListener;
|
||||
import org.springframework.batch.core.repository.JobRepository;
|
||||
import org.springframework.batch.core.step.Step;
|
||||
import org.springframework.batch.core.step.builder.StepBuilder;
|
||||
import org.springframework.batch.core.step.tasklet.Tasklet;
|
||||
import org.springframework.batch.infrastructure.item.ItemProcessor;
|
||||
import org.springframework.batch.infrastructure.item.ItemWriter;
|
||||
import org.springframework.batch.infrastructure.item.database.builder.JdbcBatchItemWriterBuilder;
|
||||
import org.springframework.batch.infrastructure.item.file.builder.FlatFileItemReaderBuilder;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Profile;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.dao.DataIntegrityViolationException;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.transaction.PlatformTransactionManager;
|
||||
|
||||
/**
|
||||
* Wires the same "import products, then report a count" job three different ways, selected by
|
||||
* Spring profile:
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code clean} — the happy path, chapters 1-3</li>
|
||||
* <li>{@code broken} — no fault tolerance, chapter 7 (restartability)</li>
|
||||
* <li>{@code skip} — {@code faultTolerant().skip(...)}, chapter 8 (skip vs. restart)</li>
|
||||
* </ul>
|
||||
*
|
||||
* All three read {@link #productReader}, run it through {@link ProductValidatingProcessor}, and
|
||||
* write with {@link #productWriter}; only the step's fault-tolerance configuration and the input
|
||||
* file differ. See {@code docs/02-anatomy-of-a-job.md} for why the job and step beans look like
|
||||
* this rather than the {@code JobBuilderFactory} / {@code StepBuilderFactory} shape older
|
||||
* tutorials still show (that pair was removed years before this article; it is not a Boot 4.1
|
||||
* surprise, just a dead end still copy-pasted in 2026).
|
||||
*/
|
||||
@Configuration
|
||||
public class BatchConfig {
|
||||
|
||||
@Value("${import.file:classpath:data/products.csv}")
|
||||
private Resource inputFile;
|
||||
|
||||
// ---- shared reader / processor / writer -----------------------------------------------
|
||||
|
||||
@Bean
|
||||
public org.springframework.batch.infrastructure.item.file.FlatFileItemReader<Product> productReader() {
|
||||
return new FlatFileItemReaderBuilder<Product>()
|
||||
.name("productReader")
|
||||
.resource(inputFile)
|
||||
.linesToSkip(1)
|
||||
.delimited().delimiter(",").names("sku", "name", "priceCents")
|
||||
.fieldSetMapper(fs -> new Product(fs.readString("sku"), fs.readString("name"), fs.readLong("priceCents")))
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public ItemProcessor<Product, Product> productProcessor() {
|
||||
return new ProductValidatingProcessor();
|
||||
}
|
||||
|
||||
/**
|
||||
* Deliberately NOT {@code beanMapped()}: {@code BeanPropertySqlParameterSource} looks for
|
||||
* {@code getSku()}, {@code getName()}, {@code getPriceCents()} via standard JavaBean
|
||||
* introspection, and a Java record's accessors are {@code sku()}, {@code name()},
|
||||
* {@code priceCents()} — no {@code get} prefix. {@code docs/06-jdbc-writer-and-records.md}
|
||||
* has the transcript of {@code beanMapped()} silently writing every column as NULL against
|
||||
* this exact record before this was caught.
|
||||
*/
|
||||
@Bean
|
||||
public ItemWriter<Product> productWriter(JdbcTemplate jdbcTemplate) {
|
||||
return new JdbcBatchItemWriterBuilder<Product>()
|
||||
.dataSource(jdbcTemplate.getDataSource())
|
||||
.sql("INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)")
|
||||
.itemPreparedStatementSetter((item, ps) -> {
|
||||
ps.setString(1, item.sku());
|
||||
ps.setString(2, item.name());
|
||||
ps.setLong(3, item.priceCents());
|
||||
})
|
||||
.assertUpdates(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Tasklet reportTasklet(JdbcTemplate jdbcTemplate) {
|
||||
return (contribution, chunkContext) -> {
|
||||
int count = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM PRODUCT", Integer.class);
|
||||
System.out.println("REPORT: " + count + " products now in the PRODUCT table");
|
||||
return org.springframework.batch.infrastructure.repeat.RepeatStatus.FINISHED;
|
||||
};
|
||||
}
|
||||
|
||||
// ---- clean profile: happy path, no fault tolerance needed ------------------------------
|
||||
|
||||
@Bean
|
||||
@Profile("clean")
|
||||
public Step importStepClean(JobRepository jobRepository, PlatformTransactionManager transactionManager,
|
||||
org.springframework.batch.infrastructure.item.file.FlatFileItemReader<Product> productReader,
|
||||
ItemProcessor<Product, Product> productProcessor, ItemWriter<Product> productWriter) {
|
||||
return new StepBuilder("importStep", jobRepository)
|
||||
.<Product, Product>chunk(10)
|
||||
.transactionManager(transactionManager)
|
||||
.reader(productReader)
|
||||
.processor(productProcessor)
|
||||
.writer(productWriter)
|
||||
.build();
|
||||
}
|
||||
|
||||
// ---- broken profile: no fault tolerance -> the whole chunk fails, job FAILS ------------
|
||||
|
||||
@Bean
|
||||
@Profile("broken")
|
||||
public Step importStepBroken(JobRepository jobRepository, PlatformTransactionManager transactionManager,
|
||||
org.springframework.batch.infrastructure.item.file.FlatFileItemReader<Product> productReader,
|
||||
ItemProcessor<Product, Product> productProcessor, ItemWriter<Product> productWriter) {
|
||||
return new StepBuilder("importStep", jobRepository)
|
||||
.<Product, Product>chunk(10)
|
||||
.transactionManager(transactionManager)
|
||||
.reader(productReader)
|
||||
.processor(productProcessor)
|
||||
.writer(productWriter)
|
||||
.build();
|
||||
}
|
||||
|
||||
// ---- skip profile: faultTolerant + skip -> the bad item is skipped, job COMPLETES -----
|
||||
|
||||
@Bean
|
||||
@Profile("skip")
|
||||
public Step importStepSkip(JobRepository jobRepository, PlatformTransactionManager transactionManager,
|
||||
org.springframework.batch.infrastructure.item.file.FlatFileItemReader<Product> productReader,
|
||||
ItemProcessor<Product, Product> productProcessor, ItemWriter<Product> productWriter) {
|
||||
return new StepBuilder("importStep", jobRepository)
|
||||
.<Product, Product>chunk(10)
|
||||
.transactionManager(transactionManager)
|
||||
.reader(productReader)
|
||||
.processor(productProcessor)
|
||||
.writer(productWriter)
|
||||
.faultTolerant()
|
||||
.skip(DataIntegrityViolationException.class)
|
||||
.skipLimit(3)
|
||||
.listener(skipListener())
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public SkipListener<Product, Product> skipListener() {
|
||||
return new SkipListener<>() {
|
||||
@Override
|
||||
public void onSkipInWrite(Product item, Throwable t) {
|
||||
System.out.println("SKIPPED on write: " + item.sku() + " (" + t.getClass().getSimpleName() + ": " + t.getMessage() + ")");
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// ---- reportStep: always runs, even if the step before it was already COMPLETED --------
|
||||
|
||||
@Bean
|
||||
public Step reportStep(JobRepository jobRepository, PlatformTransactionManager transactionManager,
|
||||
Tasklet reportTasklet) {
|
||||
return new StepBuilder("reportStep", jobRepository)
|
||||
.tasklet(reportTasklet, transactionManager)
|
||||
.allowStartIfComplete(true)
|
||||
.build();
|
||||
}
|
||||
|
||||
// ---- one Job bean per profile, all built the same way ----------------------------------
|
||||
|
||||
@Bean
|
||||
@Profile("clean")
|
||||
public Job productImportJobClean(JobRepository jobRepository, Step importStepClean, Step reportStep) {
|
||||
return new JobBuilder("productImportJob", jobRepository).start(importStepClean).next(reportStep).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@Profile("broken")
|
||||
public Job productImportJobBroken(JobRepository jobRepository, Step importStepBroken, Step reportStep) {
|
||||
return new JobBuilder("productImportJob", jobRepository).start(importStepBroken).next(reportStep).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@Profile("skip")
|
||||
public Job productImportJobSkip(JobRepository jobRepository, Step importStepSkip, Step reportStep) {
|
||||
return new JobBuilder("productImportJob", jobRepository).start(importStepSkip).next(reportStep).build();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.ankurm.batch.domain;
|
||||
|
||||
/**
|
||||
* One validated row, ready to be written to the {@code PRODUCT} table.
|
||||
*
|
||||
* <p>See {@code docs/03-chunk-oriented-processing.md} for how instances of this record move
|
||||
* through the reader → processor → writer pipeline in chunks, and
|
||||
* {@code docs/06-jdbc-writer-and-records.md} for why the writer below does not use
|
||||
* {@code beanMapped()} against this record.
|
||||
*/
|
||||
public record Product(String sku, String name, long priceCents) {
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
package com.ankurm.batch.processing;
|
||||
|
||||
import com.ankurm.batch.domain.Product;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.springframework.batch.infrastructure.item.ItemProcessor;
|
||||
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Chapter 4 ({@code docs/04-item-processor-as-filter.md}): a processor that returns
|
||||
* {@code null} for a row it does not like. Spring Batch treats a {@code null} return as
|
||||
* "filter this item" — it is counted separately from both reads and writes, and it
|
||||
* never reaches the writer or the database. This is deliberately NOT how the poisoned
|
||||
* duplicate SKU is handled: that one is format-valid and only fails at the database, which is
|
||||
* the point of chapter 7.
|
||||
*/
|
||||
public class ProductValidatingProcessor implements ItemProcessor<Product, Product> {
|
||||
|
||||
private static final Log log = LogFactory.getLog(ProductValidatingProcessor.class);
|
||||
private static final Pattern SKU_PATTERN = Pattern.compile("^[A-Z]{3}-\\d{4}$");
|
||||
|
||||
@Override
|
||||
public Product process(Product item) {
|
||||
if (!SKU_PATTERN.matcher(item.sku()).matches()) {
|
||||
log.warn("filtering " + item.sku() + ": does not match " + SKU_PATTERN.pattern());
|
||||
return null;
|
||||
}
|
||||
if (item.priceCents() <= 0) {
|
||||
log.warn("filtering " + item.sku() + ": priceCents must be positive, was " + item.priceCents());
|
||||
return null;
|
||||
}
|
||||
return item;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package com.ankurm.batch.runner;
|
||||
|
||||
import org.springframework.batch.core.job.Job;
|
||||
import org.springframework.batch.core.job.parameters.JobParametersBuilder;
|
||||
import org.springframework.batch.core.launch.JobOperator;
|
||||
import org.springframework.boot.ApplicationArguments;
|
||||
import org.springframework.boot.ApplicationRunner;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* Chapter 5 ({@code docs/05-launching-and-jobparameters.md}) and chapter 7
|
||||
* ({@code docs/07-restartability.md}): calling {@link JobOperator#start} with the SAME
|
||||
* identifying job parameter every time is, on its own, the restart mechanism. There is no
|
||||
* separate "restart" button here: if a JobInstance with these identifying parameters already
|
||||
* exists and its last execution did not complete, {@code start} runs a new JobExecution against
|
||||
* that same instance and Spring Batch resumes each step from where its own ExecutionContext says
|
||||
* it left off. If the last execution DID complete, {@code start} throws
|
||||
* {@link org.springframework.batch.core.launch.JobInstanceAlreadyCompleteException}, which this
|
||||
* runner treats as "nothing to do" rather than a failure.
|
||||
*
|
||||
* <p>{@code spring.batch.job.enabled} is left at its Boot default of {@code true}, but the
|
||||
* auto-configured {@code JobLauncherApplicationRunner} would launch every Job bean in the
|
||||
* context using default parameters, which collides with the explicit control this module wants
|
||||
* to demonstrate -- see {@code application.yml}, where it is turned off.
|
||||
*/
|
||||
@Component
|
||||
public class ImportRunner implements ApplicationRunner {
|
||||
|
||||
private final JobOperator jobOperator;
|
||||
private final Job job;
|
||||
|
||||
public ImportRunner(JobOperator jobOperator, Job job) {
|
||||
this.jobOperator = jobOperator;
|
||||
this.job = job;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run(ApplicationArguments args) throws Exception {
|
||||
var params = new JobParametersBuilder()
|
||||
.addString("batch.run", "demo") // identifying: same value = same JobInstance = restart target
|
||||
.toJobParameters();
|
||||
try {
|
||||
var execution = jobOperator.start(job, params);
|
||||
System.out.println("JOB FINISHED: status=" + execution.getStatus()
|
||||
+ " exitCode=" + execution.getExitStatus().getExitCode());
|
||||
} catch (org.springframework.batch.core.launch.JobInstanceAlreadyCompleteException e) {
|
||||
System.out.println("JOB ALREADY COMPLETE: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
spring:
|
||||
batch:
|
||||
job:
|
||||
# ImportRunner drives the job explicitly through JobOperator. The auto-configured
|
||||
# JobLauncherApplicationRunner would ALSO launch every Job bean it finds, using
|
||||
# parameterless defaults -- turning this off avoids running the job twice on every startup.
|
||||
enabled: false
|
||||
jdbc:
|
||||
initialize-schema: always
|
||||
datasource:
|
||||
url: jdbc:h2:file:./data/batchdb;AUTO_SERVER=TRUE
|
||||
username: sa
|
||||
password: ""
|
||||
driver-class-name: org.h2.Driver
|
||||
sql:
|
||||
init:
|
||||
mode: always
|
||||
schema-locations: classpath:schema.sql
|
||||
|
||||
logging:
|
||||
level:
|
||||
org.springframework.batch: INFO
|
||||
@@ -0,0 +1,61 @@
|
||||
sku,name,priceCents
|
||||
ABC-0001,Widget 1,1001
|
||||
ABC-0002,Widget 2,1002
|
||||
ABC-0003,Widget 3,1003
|
||||
ABC-0004,Widget 4,1004
|
||||
ABC-0005,Widget 5,1005
|
||||
ABC-0006,Widget 6,1006
|
||||
ABC-0007,Widget 7,1007
|
||||
ABC-0008,Widget 8,1008
|
||||
ABC-0009,Widget 9,1009
|
||||
ABC-0010,Widget 10,1010
|
||||
ABC-0011,Widget 11,1011
|
||||
abc-0012,Widget 12,1012
|
||||
ABC-0013,Widget 13,1013
|
||||
ABC-0014,Widget 14,1014
|
||||
ABC-0015,Widget 15,1015
|
||||
ABC-0016,Widget 16,1016
|
||||
ABC-0017,Widget 17,1017
|
||||
ABC-0018,Widget 18,1018
|
||||
ABC-0019,Widget 19,1019
|
||||
ABC-0020,Widget 20,1020
|
||||
ABC-0021,Widget 21,1021
|
||||
ABC-0022,Widget 22,1022
|
||||
ABC-0023,Widget 23,1023
|
||||
ABC-0024,Widget 24,1024
|
||||
ABC-0025,Widget 25,1025
|
||||
ABC-0026,Widget 26,1026
|
||||
ABC-0027,Widget 27,1027
|
||||
ABC-0028,Widget 28,1028
|
||||
ABC-0029,Widget 29,1029
|
||||
ABC-0030,Widget 30,1030
|
||||
ABC-0031,Widget 31,1031
|
||||
ABC-0032,Widget 32,1032
|
||||
ABC-0033,Widget 33,0
|
||||
ABC-0034,Widget 34,1034
|
||||
ABC-0035,Widget 35,1035
|
||||
ABC-0036,Widget 36,1036
|
||||
ABC-0037,Widget 37,1037
|
||||
ABC-0038,Widget 38,1038
|
||||
ABC-0039,Widget 39,1039
|
||||
ABC-0040,Widget 40,1040
|
||||
ABC-0041,Widget 41,1041
|
||||
ABC-0042,Widget 42,1042
|
||||
ABC-0043,Widget 43,1043
|
||||
ABC-0044,Widget 44,1044
|
||||
ABC-0045,Widget 45,1045
|
||||
ABC-0046,Widget 46,1046
|
||||
ABC-0005,Widget 47,1047
|
||||
ABC-0048,Widget 48,1048
|
||||
ABC-0049,Widget 49,1049
|
||||
ABC-0050,Widget 50,1050
|
||||
ABC-0051,Widget 51,1051
|
||||
ABC-0052,Widget 52,1052
|
||||
ABC-0053,Widget 53,1053
|
||||
ABC-0054,Widget 54,1054
|
||||
ABC-0055,Widget 55,1055
|
||||
ABC-0056,Widget 56,1056
|
||||
ABC-0057,Widget 57,1057
|
||||
ABC-0058,Widget 58,1058
|
||||
ABC-0059,Widget 59,1059
|
||||
ABC-0060,Widget 60,1060
|
||||
|
@@ -0,0 +1,61 @@
|
||||
sku,name,priceCents
|
||||
ABC-0001,Widget 1,1001
|
||||
ABC-0002,Widget 2,1002
|
||||
ABC-0003,Widget 3,1003
|
||||
ABC-0004,Widget 4,1004
|
||||
ABC-0005,Widget 5,1005
|
||||
ABC-0006,Widget 6,1006
|
||||
ABC-0007,Widget 7,1007
|
||||
ABC-0008,Widget 8,1008
|
||||
ABC-0009,Widget 9,1009
|
||||
ABC-0010,Widget 10,1010
|
||||
ABC-0011,Widget 11,1011
|
||||
abc-0012,Widget 12,1012
|
||||
ABC-0013,Widget 13,1013
|
||||
ABC-0014,Widget 14,1014
|
||||
ABC-0015,Widget 15,1015
|
||||
ABC-0016,Widget 16,1016
|
||||
ABC-0017,Widget 17,1017
|
||||
ABC-0018,Widget 18,1018
|
||||
ABC-0019,Widget 19,1019
|
||||
ABC-0020,Widget 20,1020
|
||||
ABC-0021,Widget 21,1021
|
||||
ABC-0022,Widget 22,1022
|
||||
ABC-0023,Widget 23,1023
|
||||
ABC-0024,Widget 24,1024
|
||||
ABC-0025,Widget 25,1025
|
||||
ABC-0026,Widget 26,1026
|
||||
ABC-0027,Widget 27,1027
|
||||
ABC-0028,Widget 28,1028
|
||||
ABC-0029,Widget 29,1029
|
||||
ABC-0030,Widget 30,1030
|
||||
ABC-0031,Widget 31,1031
|
||||
ABC-0032,Widget 32,1032
|
||||
ABC-0033,Widget 33,0
|
||||
ABC-0034,Widget 34,1034
|
||||
ABC-0035,Widget 35,1035
|
||||
ABC-0036,Widget 36,1036
|
||||
ABC-0037,Widget 37,1037
|
||||
ABC-0038,Widget 38,1038
|
||||
ABC-0039,Widget 39,1039
|
||||
ABC-0040,Widget 40,1040
|
||||
ABC-0041,Widget 41,1041
|
||||
ABC-0042,Widget 42,1042
|
||||
ABC-0043,Widget 43,1043
|
||||
ABC-0044,Widget 44,1044
|
||||
ABC-0045,Widget 45,1045
|
||||
ABC-0046,Widget 46,1046
|
||||
ABC-0047,Widget 47,1047
|
||||
ABC-0048,Widget 48,1048
|
||||
ABC-0049,Widget 49,1049
|
||||
ABC-0050,Widget 50,1050
|
||||
ABC-0051,Widget 51,1051
|
||||
ABC-0052,Widget 52,1052
|
||||
ABC-0053,Widget 53,1053
|
||||
ABC-0054,Widget 54,1054
|
||||
ABC-0055,Widget 55,1055
|
||||
ABC-0056,Widget 56,1056
|
||||
ABC-0057,Widget 57,1057
|
||||
ABC-0058,Widget 58,1058
|
||||
ABC-0059,Widget 59,1059
|
||||
ABC-0060,Widget 60,1060
|
||||
|
@@ -0,0 +1,10 @@
|
||||
-- Application table, separate from the BATCH_* tables that BatchJdbcAutoConfiguration
|
||||
-- creates via spring.batch.jdbc.initialize-schema. The UNIQUE constraint on sku is what turns
|
||||
-- the poisoned duplicate row into a real DataIntegrityViolationException at chunk-commit time
|
||||
-- rather than something the processor could have caught by looking at one row in isolation.
|
||||
CREATE TABLE IF NOT EXISTS PRODUCT (
|
||||
id BIGINT AUTO_INCREMENT PRIMARY KEY,
|
||||
sku VARCHAR(16) NOT NULL UNIQUE,
|
||||
name VARCHAR(200) NOT NULL,
|
||||
price_cents BIGINT NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,63 @@
|
||||
package com.ankurm.batch;
|
||||
|
||||
import com.ankurm.batch.domain.Product;
|
||||
import com.ankurm.batch.processing.ProductValidatingProcessor;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.dao.DataIntegrityViolationException;
|
||||
import org.springframework.dao.DuplicateKeyException;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* Chapter 4 ({@code docs/04-item-processor-as-filter.md}): a null return from
|
||||
* {@link org.springframework.batch.infrastructure.item.ItemProcessor#process} is a filter, not
|
||||
* a skip. These assertions pin exactly which rows the pipeline's processor removes and which it
|
||||
* lets through, independent of Spring Batch or the database.
|
||||
*/
|
||||
class ProductValidatingProcessorTest {
|
||||
|
||||
private final ProductValidatingProcessor processor = new ProductValidatingProcessor();
|
||||
|
||||
@Test
|
||||
void passesAWellFormedRow() {
|
||||
try (Transcript t = new Transcript("01-processor-filter.txt",
|
||||
"The processor as a filter: null means skip, not fail")) {
|
||||
|
||||
Product ok = new Product("ABC-0001", "Widget 1", 1001);
|
||||
Product result = processor.process(ok);
|
||||
t.line("input : %s", ok);
|
||||
t.line("result : %s", result);
|
||||
assertThat(result).isEqualTo(ok);
|
||||
|
||||
t.blank();
|
||||
Product badSku = new Product("abc-0012", "Widget 12", 1012);
|
||||
t.line("input : %s <- lowercase sku, fails ^[A-Z]{3}-\\d{4}$", badSku);
|
||||
t.line("result : %s", processor.process(badSku));
|
||||
assertThat(processor.process(badSku)).isNull();
|
||||
|
||||
t.blank();
|
||||
Product zeroPrice = new Product("ABC-0033", "Widget 33", 0);
|
||||
t.line("input : %s <- priceCents is zero", zeroPrice);
|
||||
t.line("result : %s", processor.process(zeroPrice));
|
||||
assertThat(processor.process(zeroPrice)).isNull();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void duplicateKeyExceptionIsADataIntegrityViolationException() {
|
||||
try (Transcript t = new Transcript("02-exception-hierarchy.txt",
|
||||
"Why skip(DataIntegrityViolationException.class) also catches the duplicate-key case")) {
|
||||
|
||||
DuplicateKeyException dup = new DuplicateKeyException("Unique index violation");
|
||||
t.line("thrown type : %s", dup.getClass().getName());
|
||||
t.line("is a DataIntegrityViolationException? %b", dup instanceof DataIntegrityViolationException);
|
||||
t.line("");
|
||||
t.line("H2's JdbcBatchUpdateException on a UNIQUE-constraint violation is translated by");
|
||||
t.line("Spring's SQLExceptionSubclassTranslator into DuplicateKeyException, which extends");
|
||||
t.line("DataIntegrityViolationException. A skip policy configured against the parent class");
|
||||
t.line("catches the subclass too -- see docs/07-restartability.md and docs/08-skip-vs-restart.md.");
|
||||
|
||||
assertThat(dup).isInstanceOf(DataIntegrityViolationException.class);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package com.ankurm.batch;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.PrintWriter;
|
||||
import java.io.StringWriter;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
/**
|
||||
* Writes a numbered transcript under {@code docs/output/} and echoes it to the console.
|
||||
* Every console block quoted in the article comes out of one of these files verbatim.
|
||||
*/
|
||||
public final class Transcript implements AutoCloseable {
|
||||
|
||||
private final Path path;
|
||||
private final StringWriter buffer = new StringWriter();
|
||||
private final PrintWriter out = new PrintWriter(buffer);
|
||||
|
||||
public Transcript(String fileName, String title) {
|
||||
this.path = Path.of("docs", "output", fileName);
|
||||
out.println("# " + title);
|
||||
out.println();
|
||||
}
|
||||
|
||||
public Transcript line(String format, Object... args) {
|
||||
out.println(args.length == 0 ? format : String.format(format, args));
|
||||
return this;
|
||||
}
|
||||
|
||||
public Transcript blank() {
|
||||
out.println();
|
||||
return this;
|
||||
}
|
||||
|
||||
public Transcript section(String heading) {
|
||||
out.println();
|
||||
out.println("--- " + heading + " ---");
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
out.flush();
|
||||
try {
|
||||
Files.createDirectories(path.getParent());
|
||||
Files.writeString(path, buffer.toString());
|
||||
} catch (IOException e) {
|
||||
throw new IllegalStateException("could not write " + path, e);
|
||||
}
|
||||
System.out.print(buffer);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user