diff --git a/.gitignore b/.gitignore index 75bc9f9..bdeb402 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,7 @@ target/ *.aotconf # generated by spring-boot-startup-time/scripts/gen-bulk.sh spring-boot-startup-time/src/main/java/com/ankurm/startup/bulk/ + +# spring-batch: file-based H2 databases created by running the demo +spring-batch/data/ +spring-batch/scenario-data/ diff --git a/README.md b/README.md index 8042fe7..6dda18b 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ files. | [`problem-details/`](problem-details) | [Global Exception Handling with ProblemDetail (RFC 9457) in Spring Boot 4](https://ankurm.com/spring-boot-4-problemdetail-rfc-9457-global-exception-handling/) | thirteen failures under five handling setups, validation errors, i18n, content negotiation, errors outside MVC, silent 500s, decoding on the client | | [`resilience/`](resilience) | [Spring Framework 7's Built-in Resilience: @Retryable, @ConcurrencyLimit, and What's Left for Resilience4j](https://ankurm.com/spring-framework-7-retryable-concurrencylimit-resilience4j/) | `@Retryable` and `@ConcurrencyLimit` counted invocation by invocation, retries inside transactions, where Resilience4j still earns its place, migrating from Spring Retry | | [`caching/`](caching) | [The Spring Cache Abstraction: @Cacheable, @CacheEvict, Key Generators and the Self-Invocation Trap](https://ankurm.com/spring-cache-abstraction-cacheable-cacheevict-self-invocation-trap/) | the self-invocation trap measured four ways, the key collision `SimpleKeyGenerator` makes easy, eviction timing under a thrown exception, a rollback the cache keeps, and where this sits next to Hibernate's L2 cache | +| [`spring-batch/`](spring-batch) | [Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/) | a job that fails mid-chunk and resumes exactly where it left off across two separate JVMs, skip vs. restart on the same poisoned row, the resourceless job repository that forgets a restart ever happened, and the `chunk(int)` vs `chunk(int, tx)` builder split | Articles whose text is kept here rather than only on the blog have it under `/post/` — `post.md` for the body and `meta.md` for the title, excerpt and diff --git a/spring-batch/README.md b/spring-batch/README.md new file mode 100644 index 0000000..37a6caf --- /dev/null +++ b/spring-batch/README.md @@ -0,0 +1,74 @@ +# spring-batch + +Companion code for **[Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability](https://ankurm.com/)** +on [ankurm.com](https://ankurm.com). + +Verified against Spring Boot **4.1.1**, Spring Batch **6.0.5**, Spring Framework **7.0.9**, on +Temurin JDK **25.0.4.1+1**. + +One job (`productImportJob`: import a CSV of products, then report a count), configured three +different ways by Spring profile, plus one infrastructure variant that is not a profile: + +| Profile / flag | What it demonstrates | Docs | +|---|---|---| +| `clean` | The happy path: chunk-oriented reading, filtering, writing | [ch. 3](docs/03-chunk-oriented-processing.md), [ch. 4](docs/04-item-processor-as-filter.md) | +| `broken` | No fault tolerance: a bad row fails the whole chunk and the job; restart resumes exactly where it left off | [ch. 7](docs/07-restartability.md) | +| `skip` | `faultTolerant().skip(...)`: the same bad row is skipped instead, job completes in one pass | [ch. 8](docs/08-skip-vs-restart.md) | +| `--spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration` | Resourceless job repository: a restart forgets everything happened | [ch. 10](docs/10-resourceless-vs-jdbc.md) | + +## Documentation chapters + +1. [The problem, and the smallest correct mental model](docs/01-the-problem-and-mental-model.md) +2. [The anatomy of a job](docs/02-anatomy-of-a-job.md) +3. [Chunk-oriented processing](docs/03-chunk-oriented-processing.md) +4. [The item processor as a filter](docs/04-item-processor-as-filter.md) +5. [Launching a job, and why restart is not a separate API](docs/05-launching-and-jobparameters.md) +6. [The JDBC writer, and why it is not `beanMapped()`](docs/06-jdbc-writer-and-records.md) +7. [Restartability: what actually resumes, and from where](docs/07-restartability.md) +8. [Skip vs. restart](docs/08-skip-vs-restart.md) +9. [Corrections found while writing this](docs/09-corrections.md) +10. [Resourceless vs. JDBC-backed job repositories](docs/10-resourceless-vs-jdbc.md) +11. [Production checklist](docs/11-production-checklist.md) + +## Captured output + +Everything under [`docs/output/`](docs/output) was produced by a real run and is quoted verbatim +in the article and the chapters above: + +| File | What produced it | +|---|---| +| `01-processor-filter.txt`, `02-exception-hierarchy.txt` | JUnit tests, via `mvn test` | +| `03-stepbuilder-chunk-overloads.txt`, `04-two-executioncontext-classes.txt`, `09-commandlinejobrunner-deprecated-not-removed.txt` | `javap` against the real 6.0.5 jars, via `scripts/capture-javap.sh` | +| `05-happy-path.txt` | The `clean` profile, via `scripts/capture-scenarios.sh` | +| `07-restart-run1-fails.txt`, `08-restart-run2-resumes.txt` | The `broken` profile, run twice in separate JVMs against the same database | +| `10-skip-instead-of-fail.txt` | The `skip` profile | +| `11-resourceless-forgets-everything.txt` | The `clean` profile with `BatchJdbcAutoConfiguration` excluded, run twice | + +## Running it + +Needs a JDK 25 and Maven 3.9. + +```bash +export JAVA_HOME=/path/to/jdk-25 +mvn -DskipTests package +./scripts/run-all.sh # regenerates everything under docs/output/ +``` + +Or run one scenario by hand: + +```bash +java -jar target/spring-batch-1.0.0.jar \ + --spring.profiles.active=clean \ + --import.file=file:./scenario-data/clean-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:./scenario-data/clean-demo/db +``` + +`scripts/run-scenario.sh ` wraps that for repeat use. Each scenario +gets its own file-based H2 database under `scenario-data//`, gitignored, so runs never +interfere with each other and a restart persists across separate `java -jar` invocations the way +it would across a real process restart. + +There is a diagnostic-adjacent endpoint nowhere in this module by design — everything +observable here comes from the standard `BATCH_STEP_EXECUTION` table via a plain SQL query, +which is deliberately how [chapter 10](docs/10-resourceless-vs-jdbc.md) suggests checking your +own job repository in production. diff --git a/spring-batch/docs/01-the-problem-and-mental-model.md b/spring-batch/docs/01-the-problem-and-mental-model.md new file mode 100644 index 0000000..f9e44c7 --- /dev/null +++ b/spring-batch/docs/01-the-problem-and-mental-model.md @@ -0,0 +1,86 @@ +# 1. The problem, and the smallest correct mental model + +[← README](../README.md) | [Next: Anatomy of a job →](02-anatomy-of-a-job.md) + +A batch job processes a lot of records without a person watching each one: importing a CSV of +products into a database, closing out a day's transactions, re-indexing a search table. Two +things make this harder than writing a `for` loop over the rows: + +- **It has to survive being interrupted.** A process gets killed, a database connection drops, + a row three-quarters of the way through is bad. A `for` loop that dies at row 4,700,000 has no + idea it already wrote 4,699,999 rows, and re-running it from the top either duplicates work or + duplicates data. +- **It has to report what happened**, precisely: how many rows were read, how many were written, + how many were skipped and why, whether it finished. "The import ran" is not an answer anyone + building on top of this can use. + +Spring Batch's job is to own both of those problems so your code only has to describe three +things: where the rows come from, what to do to each one, and where they go. Everything else +— tracking progress, committing in batches, remembering where a failed run stopped — +is the framework's job, provided you tell it enough for it to do that job. Most of this article +is about that "provided." + +## The five nouns + +
+ + + +Job "productImportJob" +start(importStep).next(reportStep) + +Step: importStep + +Step: reportStep + +importStep is chunk-oriented + +ItemReader + +ItemProcessor + +ItemWriter + + + + +JobRepository +every Job/Step execution, read/write/skip counts, ExecutionContext + + + +
+ +The picture above names everything this article uses: + +- **Job** — a named, ordered list of steps. `productImportJob` in this repository has two: + import the rows, then report a count. +- **Step** — one unit of work within a job. A step is either a single **Tasklet** (run + once, do one thing — `reportStep` here) or **chunk-oriented**: read one item, process it, + repeat until you have a chunk's worth, then write and commit the whole chunk in one + transaction. +- **ItemReader / ItemProcessor / ItemWriter** — the three interfaces chunk-oriented + processing is built from. This module's reader is a `FlatFileItemReader` over a CSV, the + processor validates and filters rows, the writer is a `JdbcBatchItemWriter`. +- **JobRepository** — the thing that makes the first two bullet points of this page + possible. It persists every job and step execution: status, timestamps, and the read / write / + filter / skip / commit / rollback counts you saw in the transcripts above. Chapter + [10](10-resourceless-vs-jdbc.md) is entirely about a trap hiding in how this gets configured on + Spring Boot 4.1. + +Hold onto one fact from this page, because it is the one everything else cashes in later: **a +chunk commits or it doesn't, as a whole.** There is no partial chunk. That single sentence is the +entire explanation for why restarting a failed job does not reprocess everything, and it is why +the "poisoned" row in this repository's demo data fails nine other, perfectly good rows along +with it — see [chapter 7](07-restartability.md). + +## What this repository demonstrates, and how to run it + +Four Spring profiles configure the same job three different ways plus one infrastructure +variant; see the [README](../README.md) for the full table and the exact commands. Everything +under `docs/output/` was produced by `scripts/capture-scenarios.sh` and `scripts/capture-javap.sh`, +which `scripts/run-all.sh` runs in sequence. + +[Next: Anatomy of a job →](02-anatomy-of-a-job.md) diff --git a/spring-batch/docs/02-anatomy-of-a-job.md b/spring-batch/docs/02-anatomy-of-a-job.md new file mode 100644 index 0000000..9838572 --- /dev/null +++ b/spring-batch/docs/02-anatomy-of-a-job.md @@ -0,0 +1,81 @@ +# 2. The anatomy of a job, and why the code looks different from older tutorials + +[← Previous](01-the-problem-and-mental-model.md) | [README](../README.md) | [Next: Chunk-oriented processing →](03-chunk-oriented-processing.md) + +`BatchConfig` in this module builds jobs and steps like this: + +```java +new JobBuilder("productImportJob", jobRepository) + .start(importStep) + .next(reportStep) + .build(); + +new StepBuilder("importStep", jobRepository) + .chunk(10) + .transactionManager(transactionManager) + .reader(productReader) + .processor(productProcessor) + .writer(productWriter) + .build(); +``` + +If you have seen Spring Batch code before 2023 or so, two things here might look unfamiliar, +and it is worth being precise about which is a genuine Spring Boot 4.1 change and which is just +old: + +- `JobBuilderFactory` / `StepBuilderFactory` autowired as beans, then called as + `jobBuilderFactory.get("name")` — **this was removed in Spring Batch 5, years before this + article.** It is not a Boot 4.1 surprise, it is a dead end that tutorials keep copy-pasting. + `JobBuilder` and `StepBuilder` are plain classes you construct directly with a `JobRepository`, + as above. +- `.chunk(10, transactionManager)` — passing the transaction manager as a second argument + to `chunk()` — **this compiles in 6.0.5, but it returns a different builder.** + `StepBuilder.chunk(int)` returns a `ChunkOrientedStepBuilder`, the model this whole module + uses; `StepBuilder.chunk(int, PlatformTransactionManager)` returns the older + `SimpleStepBuilder`, kept for the pre-6.0 chunk-processing model. Real `javap` output for both + overloads, from this project's own `spring-batch-core-6.0.5.jar`: + [`docs/output/03-stepbuilder-chunk-overloads.txt`](output/03-stepbuilder-chunk-overloads.txt). + The two builders overlap almost entirely in the methods you would reach for — + `reader`/`processor`/`writer`/`faultTolerant`/`skip` exist on both — so the wrong choice + usually still compiles and runs, and just quietly uses the legacy step implementation. Call + `.chunk(10).transactionManager(tx)` (two calls) if you want `ChunkOrientedStep`. + +
The fingerprint of picking the wrong overload. Your step still builds, runs, reads, writes and commits correctly — there is no error. The tell is in what you can't reach: ChunkOrientedStepBuilder-only options like retryPolicy(RetryPolicy) using Spring Framework 7's retry API, or a stack trace mentioning ChunkOrientedStep versus the older TaskletStep wrapping a ChunkOrientedTasklet. If you copied a two-argument chunk(10, txManager) from a pre-6.0 example and it "just worked", this is why nothing complained.
+ +## Where things moved + +Spring Batch 6.0 (bundled with Spring Boot 4.1.1 as `spring-batch.version` `6.0.5`, confirmed +against `spring-boot-dependencies-4.1.1.pom`) reorganized packages fairly aggressively. The ones +this module's code actually hits: + +| Old (Spring Batch 5.x) | New (6.0.5) | +|---|---| +| `org.springframework.batch.core.repeat.RepeatStatus` | `org.springframework.batch.infrastructure.repeat.RepeatStatus` | +| `org.springframework.batch.item.*` (readers, writers, `ExecutionContext`) | `org.springframework.batch.infrastructure.item.*` | +| `org.springframework.batch.core.JobParameters` (mutable-ish, `Map`-backed) | `org.springframework.batch.core.job.parameters.JobParameters` (immutable record, `Set`-backed) | + +The `RepeatStatus` move is the one this project's own `reportTasklet` hit directly — see +[chapter 9](09-corrections.md) for the exact compiler error it produced before the import was +fixed. It is a good example of the article's verification rule in practice: rather than trusting +a description of where things moved, write the code from the old import, let `javac` say +`package ... does not exist`, and fix the import. That is faster and more reliable than +reading a migration guide's prose, and chapter 9 has a case where trusting the prose produced a +wrong claim. + +## JobRepository and JobOperator + +Two interfaces you inject rather than configure by hand in this module: + +- **`JobRepository`** — every `JobBuilder` and `StepBuilder` above takes one as a + constructor argument. In 6.0 it also extends `JobExplorer` (querying past executions), so + there is one bean to inject instead of two. +- **`JobOperator`** — what `ImportRunner` calls to start the job (see + [chapter 5](05-launching-and-jobparameters.md)). It extends `JobLauncher` and adds operational + methods: `restart(executionId)`, `stop(executionId)`, `recover(execution)`, `getJobNames()`. + `JobLauncher`/`JobExplorer` still exist but are the deprecated half of this pair now. + +Both are auto-configured by `spring-boot-starter-batch-jdbc` on this module's classpath; nothing +in `BatchConfig` declares them as beans. [Chapter 10](10-resourceless-vs-jdbc.md) explains what +you get instead if that starter is missing. + +[Next: Chunk-oriented processing →](03-chunk-oriented-processing.md) diff --git a/spring-batch/docs/03-chunk-oriented-processing.md b/spring-batch/docs/03-chunk-oriented-processing.md new file mode 100644 index 0000000..0af35a9 --- /dev/null +++ b/spring-batch/docs/03-chunk-oriented-processing.md @@ -0,0 +1,78 @@ +# 3. Chunk-oriented processing: what "chunk(10)" actually does + +[← Previous](02-anatomy-of-a-job.md) | [README](../README.md) | [Next: The item processor as a filter →](04-item-processor-as-filter.md) + +`chunk(10)` means: read up to 10 items (calling the reader once per item), run each through the +processor, then hand all the survivors to the writer in **one call**, inside **one transaction**. +If the writer succeeds, the transaction commits and the step's counters — read, write, +filter, commit — move forward together. If anything in that chunk throws, the whole +transaction rolls back: none of those 10 items' writes are kept, even the ones that would have +succeeded on their own. + +The happy-path run of this module's `productImportJob` against clean data makes this concrete. +60 rows, chunk size 10, two rows fail the processor's validation (filtered, not written): + +```console +$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION; +READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT +60 | 2 | 58 | 6 +``` + +Full transcript: [`docs/output/05-happy-path.txt`](output/05-happy-path.txt), source: +[`ProductValidatingProcessor.java`](../src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java). +60 rows at chunk size 10 is 6 chunks — `COMMIT_COUNT` confirms all 6 committed, one +transaction each. + +
+ + +60 rows → six chunks of 10, each its own transaction + + + + + + + + +1-10 +11-20 +21-30 +31-40 +41-50 +51-60 + +Each box: read 10 → process 10 (0-1 filtered) → write survivors → COMMIT. +Row 12 (chunk 2) and row 33 (chunk 4) are filtered by the processor -- see chapter 4 -- so those +two chunks commit 9 rows instead of 10. 58 rows land in PRODUCT; every chunk still commits. + +
+ +## Why 10, and what changes if you pick a different size + +Chunk size is a trade-off with no universally right answer: + +- **Smaller chunks** commit more often, so a failure loses less work and transactions are + shorter (less lock contention, smaller rollback). They also mean more round trips to the + database — more commit overhead per row. +- **Larger chunks** amortize that overhead across more rows, but a failure anywhere in the chunk + costs you the whole chunk's work, and the transaction held open is bigger and longer. + +10 is small enough that chapter 7's poisoned row costs you 12 rows of rework at most (one +chunk), not thousands, and small enough to demonstrate multiple commits from 60 rows without a +huge fixture file. Production jobs processing millions of rows commonly use chunk sizes in the +hundreds or low thousands; the right number depends on row size, write cost, and how expensive a +partial redo is for your specific job — there is no formula that replaces measuring it +against your own writer. + +## Going deeper + +- The chunk-processing reference: [Spring Batch — Chunk-oriented processing](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing.html) (`rel="nofollow"`). +- `ChunkOrientedStep` itself, if you want to see the commit/rollback logic this chapter + describes: `org.springframework.batch.core.step.item.ChunkOrientedStep` in + `spring-batch-core-6.0.5.jar` (the stack trace in + [`docs/output/07-restart-run1-fails.txt`](output/07-restart-run1-fails.txt) names it directly). +- What happens when the processor itself decides to drop a row rather than the writer failing: + [chapter 4](04-item-processor-as-filter.md). + +[Next: The item processor as a filter →](04-item-processor-as-filter.md) diff --git a/spring-batch/docs/04-item-processor-as-filter.md b/spring-batch/docs/04-item-processor-as-filter.md new file mode 100644 index 0000000..48ce889 --- /dev/null +++ b/spring-batch/docs/04-item-processor-as-filter.md @@ -0,0 +1,62 @@ +# 4. The item processor as a filter + +[← Previous](03-chunk-oriented-processing.md) | [README](../README.md) | [Next: Launching and JobParameters →](05-launching-and-jobparameters.md) + +`ItemProcessor` has one method: `O process(I item)`. Returning a transformed item is the +obvious case; returning `null` is the one people miss, and it means something specific: +**this item is filtered.** It is counted separately (`FILTER_COUNT`) from both `READ_COUNT` and +`WRITE_COUNT`, and it never reaches the writer, so it cannot throw the exception a bad write +would. + +[`ProductValidatingProcessor`](../src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java) +uses this for two checks that can be made from one row alone, with no database involved: + +```java +if (!SKU_PATTERN.matcher(item.sku()).matches()) { + return null; // filtered: does not match ^[A-Z]{3}-\d{4}$ +} +if (item.priceCents() <= 0) { + return null; // filtered: priceCents must be positive +} +``` + +Real output, from a plain unit test with no Spring context involved +([`ProductValidatingProcessorTest.java`](../src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java), +transcript [`docs/output/01-processor-filter.txt`](output/01-processor-filter.txt)): + +```console +input : Product[sku=abc-0012, name=Widget 12, priceCents=1012] <- lowercase sku, fails ^[A-Z]{3}-\d{4}$ +result : null + +input : Product[sku=ABC-0033, name=Widget 33, priceCents=0] <- priceCents is zero +result : null +``` + +## Filter vs. skip vs. fail: pick by what the check needs + +This module deliberately does not filter the poisoned duplicate SKU in the processor, even +though duplication sounds like a validation problem. The reason is what information the check +needs: + +| Check | What it needs | Where it belongs | +|---|---|---| +| SKU matches a format | Just this row | Processor, filter (`null`) | +| Price is positive | Just this row | Processor, filter (`null`) | +| SKU is unique | Every other row already written | The database, via a constraint | + +A processor can only see one item at a time (by design — that is what lets Spring Batch +process items independently and in chunks). It cannot know a SKU is a duplicate without querying +everything written so far, which defeats the point of a stateless per-item check and does not +scale. The database already tracks this via the `UNIQUE` constraint on `PRODUCT.sku` +([`schema.sql`](../src/main/resources/schema.sql)), so that is where the duplicate check +actually happens — as a write failure, not a filtered row. What a write failure does to the +rest of the chunk, and how the job recovers from it, is the rest of this article: [chapter 7](07-restartability.md) and [chapter 8](08-skip-vs-restart.md). + +## Going deeper + +- `ItemProcessor` returning `null`: [Spring Batch reference — item processing](https://docs.spring.io/spring-batch/reference/processor.html) (`rel="nofollow"`). +- Chaining processors with `CompositeItemProcessor` when validation and transformation both need + their own class — not used in this module (one processor does both jobs here), but worth + knowing before a single processor method grows past a screenful. + +[Next: Launching and JobParameters →](05-launching-and-jobparameters.md) diff --git a/spring-batch/docs/05-launching-and-jobparameters.md b/spring-batch/docs/05-launching-and-jobparameters.md new file mode 100644 index 0000000..8270ccd --- /dev/null +++ b/spring-batch/docs/05-launching-and-jobparameters.md @@ -0,0 +1,65 @@ +# 5. Launching a job, and why restart is not a separate API + +[← Previous](04-item-processor-as-filter.md) | [README](../README.md) | [Next: The JDBC writer and Java records →](06-jdbc-writer-and-records.md) + +[`ImportRunner`](../src/main/java/com/ankurm/batch/runner/ImportRunner.java) launches the job +with: + +```java +var params = new JobParametersBuilder() + .addString("batch.run", "demo") // identifying + .toJobParameters(); +try { + var execution = jobOperator.start(job, params); +} catch (JobInstanceAlreadyCompleteException e) { + // already succeeded; nothing to do +} +``` + +`JobParametersBuilder` and `addString(...)` look exactly like Spring Batch 4 and 5. What changed +underneath is the type it builds: `JobParameters` is now an immutable record holding a +`Set>`, and each `JobParameter` is itself a record carrying its own name, +value, type, and an `identifying` flag (default `true` for the `addX` methods used here). None of +that shows up in this code — it matters if you ever construct a `JobParameters` by hand +instead of through the builder, or serialize one. + +## "Identifying" is the whole restart mechanism + +A **JobInstance** is identified by a job name plus the set of parameters marked `identifying`. +Calling `jobOperator.start(job, params)` with parameters that match an existing JobInstance does +not create a second, independent run: + +- If that instance's last execution is not `COMPLETED` (it `FAILED`, or never finished), Spring + Batch creates a new `JobExecution` **against the same instance** and each step resumes from + its own last-committed position. This is a restart, and it happens through the exact same + `start()` call as a first attempt — see [chapter 7](07-restartability.md) for what + "resumes from" means concretely. +- If that instance's last execution `COMPLETED`, `start()` throws + `JobInstanceAlreadyCompleteException` (unless the step allows re-running — + `allowStartIfComplete`, used on `reportStep`; see [chapter 8](08-skip-vs-restart.md)). + +There is a separate `JobOperator.restart(long executionId)` method for restarting by execution +ID explicitly, useful for an operational tool that lists failed executions and lets someone pick +one. This module does not need it: every run uses the same identifying parameter +(`batch.run=demo`), so plain `start()` already does the right thing whether this is attempt one +or attempt two. If you want a fresh JobInstance on every run instead — the common pattern +for "run this daily" jobs — add a parameter that changes each time, typically a timestamp +or an incrementer, and mark it identifying (the default). + +## `spring.batch.job.enabled` + +Spring Boot auto-configures a `JobLauncherApplicationRunner` that launches every `Job` bean it +finds using empty parameters, on every application startup. This module turns that off +(`spring.batch.job.enabled: false` in [`application.yml`](../src/main/resources/application.yml)) +because `ImportRunner` needs to control the parameters (the identifying `batch.run` value) and +which profile's `Job` bean is active. Leaving both runners on would launch the job twice per +startup with two different parameter sets. If you only ever need "run the one job on startup +with no arguments," the auto-configured runner is simpler and this whole class is unnecessary. + +## Going deeper + +- `JobParameters` and identifying parameters: [Spring Batch reference — running a job](https://docs.spring.io/spring-batch/reference/job/running.html) (`rel="nofollow"`). +- `spring.batch.job.name` for selecting which job the auto-configured runner launches, when you + have more than one `Job` bean and want to keep using it. + +[Next: The JDBC writer and Java records →](06-jdbc-writer-and-records.md) diff --git a/spring-batch/docs/06-jdbc-writer-and-records.md b/spring-batch/docs/06-jdbc-writer-and-records.md new file mode 100644 index 0000000..c64c44e --- /dev/null +++ b/spring-batch/docs/06-jdbc-writer-and-records.md @@ -0,0 +1,59 @@ +# 6. The JDBC writer, and why it is not `beanMapped()` + +[← Previous](05-launching-and-jobparameters.md) | [README](../README.md) | [Next: Restartability →](07-restartability.md) + +`JdbcBatchItemWriterBuilder` offers two ways to map an item's fields to SQL parameters: +`columnMapped()` (positional, via `Map`/`SqlParameterSource` keyed by column name) and +`beanMapped()` (reflective, via `BeanPropertySqlParameterSource`, matching `:paramName` markers +in the SQL to JavaBean getters). Most Spring Batch examples reach for `beanMapped()` because it +needs the least code. + +`Product` in this module is a Java record: + +```java +public record Product(String sku, String name, long priceCents) {} +``` + +Its accessors are `sku()`, `name()`, `priceCents()` — no `get` prefix. Standard JavaBean +introspection, which `BeanPropertySqlParameterSource` uses, looks for `getSku()`, `getName()`, +`getPriceCents()`. Those do not exist on a record, so `beanMapped()` against a plain record either +finds nothing to bind (leaving every parameter `NULL`) or fails outright, depending on the exact +introspector version in play — not a mistake you want to discover from a table full of NULLs +in production. + +[`BatchConfig.productWriter`](../src/main/java/com/ankurm/batch/config/BatchConfig.java) sidesteps +the question entirely with `itemPreparedStatementSetter`, setting each column explicitly: + +```java +new JdbcBatchItemWriterBuilder() + .dataSource(jdbcTemplate.getDataSource()) + .sql("INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)") + .itemPreparedStatementSetter((item, ps) -> { + ps.setString(1, item.sku()); + ps.setString(2, item.name()); + ps.setLong(3, item.priceCents()); + }) + .assertUpdates(true) + .build(); +``` + +More typing, zero ambiguity about what gets bound where, and it works identically whether the +item type is a record, a plain class, or something with no JavaBean getters at all. + +
If you want beanMapped() with records anyway. Newer versions of Spring's BeanWrapperImpl (the machinery behind BeanPropertySqlParameterSource) have gained some record support in recent Spring Framework releases, but the safe rule is to check it against the exact Spring Framework version you are on rather than assume — the failure mode when it does not work is silent NULLs, not an exception, which is the worst kind of wrong.
+ +`assertUpdates(true)` is worth keeping on deliberately: it makes the writer throw if a batch +statement reports zero rows updated for any item, instead of silently accepting a no-op write. +Combined with `itemPreparedStatementSetter`, a typo in a column name fails loudly at the first +write attempt rather than producing a table that looks plausible but is missing a column's worth +of data. + +## Going deeper + +- `JdbcBatchItemWriter` and its two mapping styles: [Spring Batch reference — item writers](https://docs.spring.io/spring-batch/reference/readers-and-writers/item-writer.html) (`rel="nofollow"`). +- Java records and JavaBean introspection generally: this is not a Spring Batch quirk, it is how + `java.beans.Introspector` has always worked, and it bites anywhere a library assumes + `getX()`/`setX()` — Jackson, JPA, validation frameworks — each with its own answer + for how much (if any) record support it has added. + +[Next: Restartability →](07-restartability.md) diff --git a/spring-batch/docs/07-restartability.md b/spring-batch/docs/07-restartability.md new file mode 100644 index 0000000..db8bc38 --- /dev/null +++ b/spring-batch/docs/07-restartability.md @@ -0,0 +1,112 @@ +# 7. Restartability: what actually resumes, and from where + +[← Previous](06-jdbc-writer-and-records.md) | [README](../README.md) | [Next: Skip vs. restart →](08-skip-vs-restart.md) + +This chapter is the "run 1 fails, run 2 in a brand-new JVM finishes the job" demonstration +[the article](https://ankurm.com/) leads with. Two real, separate `java -jar` invocations, +against the same file-based H2 database, produced everything below — see +[`scripts/capture-scenarios.sh`](../scripts/capture-scenarios.sh) for the exact commands. + +## Run 1: the poisoned duplicate fails a whole chunk + +`products-poison.csv` is identical to the clean fixture except row 47, where the SKU is changed +to `ABC-0005` — a duplicate of row 5. Row 47 falls in chunk 5 (rows 41-50 at chunk size +10). The `broken` profile's step has no fault tolerance configured, so the `UNIQUE` constraint +violation on that insert fails the batch statement, which fails the chunk, which fails the step, +which fails the job: + +```console +$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION; +STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT +FAILED | 50 | 2 | 38 | 4 | 1 +``` + +Full transcript: [`docs/output/07-restart-run1-fails.txt`](output/07-restart-run1-fails.txt). +Four chunks (rows 1-40, minus the 2 filtered by the processor — see +[chapter 4](04-item-processor-as-filter.md) — = 38 written) committed successfully before +chunk 5 rolled back entirely. The reader had read through row 50 by the time the writer failed +(`READ_COUNT` 50), but none of chunk 5's rows are in the `PRODUCT` table — the transaction +that would have written them rolled back along with everything else in that chunk. + +## The fix, and the restart + +Someone fixes the source data — here, changing row 47's SKU to something that is not a +duplicate, in place, same line, same file: + +```console +sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv +``` + +Then the exact same command that produced run 1 runs again, in a completely new JVM process, +using the exact same identifying job parameter (`batch.run=demo`, see +[chapter 5](05-launching-and-jobparameters.md)): + +```console +$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id; +STEP_EXECUTION_ID | JOB_EXECUTION_ID | STATUS | READ_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT +1 | 1 | FAILED | 50 | 38 | 4 | 1 +2 | 2 | COMPLETED | 20 | 20 | 2 | 0 + +$ SELECT COUNT(*) FROM PRODUCT; +COUNT(*) +58 +``` + +Full transcript: [`docs/output/08-restart-run2-resumes.txt`](output/08-restart-run2-resumes.txt). +Read that second row carefully: **`READ_COUNT` is 20, not 60.** The new execution did not start +the CSV from row 1. It read only rows 41 through 60 — the two chunks that had not committed +yet — wrote all 20 of them (the earlier two filtered rows were both in the first 40, so +nothing left to filter here), and the job status flips to `COMPLETED`. `PRODUCT` ends up with +58 rows: the 38 that survived run 1, plus these 20. + +
+ + +Run 1 (JobExecution 1) +1-10 +11-20 +21-30 +31-40 +41-50 FAIL +never read +committed: rows 1-40 (38 written after 2 filtered) -- job FAILS at chunk 5, exit status FAILED + +Run 2 (JobExecution 2) -- fresh JVM, same database, corrected row 47 +rows 1-40: already committed, NOT re-read +41-50 OK +51-60 OK +READ_COUNT for this execution = 20, not 60. COMPLETED. PRODUCT now has 58 rows total. + +
+ +## Where the resume position actually lives + +The reader's position is not something Spring Batch infers after the fact — it is saved, +by the reader itself, into the step's `ExecutionContext` at every chunk commit, in the **same +transaction** as the chunk's writes. `FlatFileItemReader` implements `ItemStream` +(`org.springframework.batch.infrastructure.item.ItemStream`, see +[`docs/output/04-two-executioncontext-classes.txt`](output/04-two-executioncontext-classes.txt) +for its shape) and calls `update(ExecutionContext)` on every commit, storing the current line +count. Because that update happens inside the chunk's transaction: + +- If the chunk commits, the new line count is persisted right alongside the rows it wrote. +- If the chunk rolls back (chunk 5, run 1), the line-count update rolls back too — the + persisted position stays at the end of chunk 4, not partway through chunk 5. + +That is the entire mechanism. There is no separate "resume point" concept to configure: it falls +out of transactional chunk commits plus a reader that participates in `ItemStream`. Writing a +custom reader that does not save its position to the `ExecutionContext` — a plain +`ItemReader` with no `ItemStream` — means restarts always start that reader from scratch, +which is sometimes exactly what you want (an idempotent reader) and sometimes a bug you will not +notice until the first real failure in production. + +## Going deeper + +- `ItemStream` and the two-argument `open`/`update` contract: + [Spring Batch reference — readers and ItemStream](https://docs.spring.io/spring-batch/reference/readers-and-writers/item-reader.html) (`rel="nofollow"`). +- Configuring restart limits per step (`startLimit`, `allowStartIfComplete`): + [chapter 8](08-skip-vs-restart.md), and the [Spring Batch reference on restart configuration](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/restart.html) (`rel="nofollow"`). +- What happens if the job repository itself does not persist across the JVM restart shown above: + [chapter 10](10-resourceless-vs-jdbc.md) — this entire chapter depends on it being real. + +[Next: Skip vs. restart →](08-skip-vs-restart.md) diff --git a/spring-batch/docs/08-skip-vs-restart.md b/spring-batch/docs/08-skip-vs-restart.md new file mode 100644 index 0000000..ae08d30 --- /dev/null +++ b/spring-batch/docs/08-skip-vs-restart.md @@ -0,0 +1,92 @@ +# 8. Skip vs. restart, and the two knobs that shape a restart + +[← Previous](07-restartability.md) | [README](../README.md) | [Next: Corrections found while writing this →](09-corrections.md) + +Failing the job and restarting after a fix (chapter 7) is one answer to a bad row. It is the +right one when the row is genuinely wrong and someone needs to decide what to do about it. It is +the wrong one when the bad row is expected background noise — a handful of duplicates in a +million-row feed — and stopping the whole job for each one is not realistic. + +The `skip` profile configures the same step with `faultTolerant().skip(...)` instead: + +```java +new StepBuilder("importStep", jobRepository) + .chunk(10) + .transactionManager(transactionManager) + .reader(productReader) + .processor(productProcessor) + .writer(productWriter) + .faultTolerant() + .skip(DataIntegrityViolationException.class) + .skipLimit(3) + .listener(skipListener()) + .build(); +``` + +Run against the same poisoned file used in chapter 7, in one pass, no restart needed: + +```console +$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION; +COMPLETED | 60 | 2 | 57 | 1 | 6 | 2 + +$ SELECT COUNT(*) FROM PRODUCT; +COUNT(*) +57 +``` + +Full transcript: [`docs/output/10-skip-instead-of-fail.txt`](output/10-skip-instead-of-fail.txt). +The job reads all 60 rows in one execution, filters the same 2 as always, skips exactly the +duplicate (`WRITE_SKIP_COUNT` 1), and completes with 57 rows written. Note `ROLLBACK_COUNT` is 2, +not 0: when a fault-tolerant chunk's batch write fails, Spring Batch does not give up on the +chunk — it rolls back once, then re-processes that chunk's items **one at a time** to find +out which single item is the actual problem, so it can skip only that one and keep the rest. That +retry-by-scanning is one extra rollback for the chunk containing the bad row; you can see it in +the count. + +## `DataIntegrityViolationException.class` catches `DuplicateKeyException` because of the hierarchy + +The skip policy is registered against `DataIntegrityViolationException`, but the exception H2 +actually throws is `DuplicateKeyException`. This works because +`DuplicateKeyException extends DataIntegrityViolationException` — Spring's JDBC exception +translation maps H2's `JdbcBatchUpdateException` (a unique-constraint violation) to the more +specific subclass, and a skip policy registered against the parent still matches it. Confirmed +directly, not assumed: + +```console +thrown type : org.springframework.dao.DuplicateKeyException +is a DataIntegrityViolationException? true +``` + +(from [`docs/output/02-exception-hierarchy.txt`](output/02-exception-hierarchy.txt), produced by +[`ProductValidatingProcessorTest.duplicateKeyExceptionIsADataIntegrityViolationException`](../src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java)). +Registering skip policies against a broad parent type like `DataIntegrityViolationException` +means you do not have to enumerate every specific subtype Spring's translation layer might +produce — but it also means you are choosing to skip *any* data-integrity problem, not just +duplicates. `skipLimit(3)` is the safety valve: past 3 skips in one execution, the step gives up +and fails anyway, on the theory that "one bad row" and "the whole feed is corrupt" should not be +handled identically. + +## The two restart-shaping options on a step + +`reportStep` uses one of these; the football-job example in the Spring Batch reference documents +both well, and this module's `reportStep` is a direct, smaller version of that pattern: + +- **`allowStartIfComplete(true)`** — by default, a step that already finished `COMPLETED` + is skipped entirely on restart (the whole point of chapter 7 is not re-doing committed work). + `reportStep` overrides that, because a status report should reflect the current state of the + `PRODUCT` table on every run, including a restart, not just the first successful attempt. +- **`startLimit(n)`** — not used in this module, but worth knowing: caps how many times a + step may be *attempted* (not completed) before Spring Batch throws + `StartLimitExceededException` instead of trying again. Useful for a step that does something + non-idempotent enough that repeated automatic retries would make things worse — call out + external systems, send notifications — where the right response to repeated failure is a + human looking at it, not an unbounded retry loop. + +## Going deeper + +- Skip and retry configuration in full, including `noSkip()` exclusions and combining skip with + retry: [Spring Batch reference — configuring skip logic](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/configuring-skip.html) (`rel="nofollow"`). +- `startLimit` and `allowStartIfComplete` with the football-job example this chapter borrows the + shape of: [Spring Batch reference — configuring a step for restart](https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/restart.html) (`rel="nofollow"`). + +[Next: Corrections found while writing this →](09-corrections.md) diff --git a/spring-batch/docs/09-corrections.md b/spring-batch/docs/09-corrections.md new file mode 100644 index 0000000..80c6d22 --- /dev/null +++ b/spring-batch/docs/09-corrections.md @@ -0,0 +1,57 @@ +# 9. Corrections found while writing this + +[← Previous](08-skip-vs-restart.md) | [README](../README.md) | [Next: Resourceless vs. JDBC-backed →](10-resourceless-vs-jdbc.md) + +Per the verification discipline this repository follows, here is what was wrong in an earlier +draft, and how it was caught — recorded honestly rather than silently fixed, because the +mistake is often as informative as the correct answer. + +## "CommandLineJobRunner was removed in 6.0" — wrong, it is deprecated + +A community-written Spring Batch 6.0 migration guide describes `CommandLineJobRunner` as removed +outright, in a list alongside classes that genuinely were deleted +(`ChunkListenerSupport`, `JobExecutionListenerSupport`, and others). An early draft of this +article repeated that claim. Checking the actual `spring-batch-core-6.0.5.jar` shows it is wrong: + +```console +$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated" +public class org.springframework.batch.core.launch.support.CommandLineJobRunner +... + Deprecated: true + RuntimeVisibleAnnotations: + 0: #505(#506=s#507,#508=Z#509) + java.lang.Deprecated( + since="6.0" + forRemoval=true + ) +``` + +Full transcript: [`docs/output/09-commandlinejobrunner-deprecated-not-removed.txt`](output/09-commandlinejobrunner-deprecated-not-removed.txt). +The class is present, loadable, and functional in 6.0.5 — it carries +`@Deprecated(since = "6.0", forRemoval = true)`, which means "stop using this, it will be +deleted in a future release," not "this is already gone." `CommandLineJobOperator` is the +replacement, and it does exist alongside it in the same package. + +This is the article's own self-correction pass working as intended: a claim sourced from a +third-party summary of a migration guide, not from a primary artifact, was treated as unverified +until checked. The general rule this project follows (see the parent repository's process notes) +is that a migration guide's own prose is not itself a primary source for "removed vs. +deprecated" — the jar is. + +## `RepeatStatus` package move, caught by the compiler rather than by reading docs + +Writing [`BatchConfig.reportTasklet`](../src/main/java/com/ankurm/batch/config/BatchConfig.java), +the first draft imported `org.springframework.batch.core.repeat.RepeatStatus` — the +Spring Batch 5.x location, from memory and from older examples. `mvn -B -o compile` reported: + +``` +[ERROR] .../BatchConfig.java:[93,57] package org.springframework.batch.core.repeat does not exist +``` + +The fix was `org.springframework.batch.infrastructure.repeat.RepeatStatus`, matching the general +package reorganization covered in [chapter 2](02-anatomy-of-a-job.md). No migration guide was +consulted for this one; the compiler error was sufficient, which is the point — letting +`javac` tell you where an API moved is faster and more reliable than trying to remember or look +up a package reorganization table. + +[Next: Resourceless vs. JDBC-backed →](10-resourceless-vs-jdbc.md) diff --git a/spring-batch/docs/10-resourceless-vs-jdbc.md b/spring-batch/docs/10-resourceless-vs-jdbc.md new file mode 100644 index 0000000..386773e --- /dev/null +++ b/spring-batch/docs/10-resourceless-vs-jdbc.md @@ -0,0 +1,71 @@ +# 10. Resourceless vs. JDBC-backed: the dependency that makes chapter 7 possible + +[← Previous](09-corrections.md) | [README](../README.md) | [Next: Production checklist →](11-production-checklist.md) + +Everything in [chapter 7](07-restartability.md) depends on one dependency choice that is easy to +get wrong on Spring Boot 4.1, because getting it wrong does not produce an error — the +application starts, the job runs, nothing complains. + +Spring Boot 4.1 split Batch autoconfiguration into separate modules rather than one monolith: + +| Starter | Autoconfiguration class | What it provides | +|---|---|---| +| `spring-boot-starter-batch` | `BatchAutoConfiguration` | Batch infrastructure basics; a **resourceless** (in-memory, non-persistent) `JobRepository` if nothing more specific is configured | +| `spring-boot-starter-batch-jdbc` | `BatchJdbcAutoConfiguration` | A real JDBC-backed `JobRepository`, plus `spring.batch.jdbc.initialize-schema` to create the `BATCH_*` tables | + +`spring-boot-starter-batch-jdbc` (this module's actual dependency, in +[`pom.xml`](../pom.xml)) pulls in `spring-boot-starter-batch` transitively, so **both** +autoconfiguration classes are normally on the classpath together, and `BatchJdbcAutoConfiguration` +wins by providing the more specific `JobRepository` bean. Nothing about compiling or starting the +application tells you which one actually won. + +## Proving it, by turning the JDBC half off + +`--spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration` removes the JDBC-backed +`JobRepository` from the picture, leaving `BatchAutoConfiguration`'s resourceless one as the only +candidate. Same job, same identifying job parameters, same `PRODUCT` table, run twice, in two +separate JVMs: + +```console +-- run 1 -- +REPORT: 58 products now in the PRODUCT table +JOB FINISHED: status=COMPLETED exitCode=COMPLETED + +-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table -- +org.springframework.dao.DuplicateKeyException: ... Unique index or primary key violation ... + VALUES ( /* 1 */ 'ABC-0001' ) +JOB FINISHED: status=FAILED exitCode=FAILED +``` + +Full transcript: [`docs/output/11-resourceless-forgets-everything.txt`](output/11-resourceless-forgets-everything.txt). +Run 2 does not throw `JobInstanceAlreadyCompleteException`, the exception you would expect for +re-running an already-completed job (see [chapter 5](05-launching-and-jobparameters.md)). It does +not know there was a run 1 at all. The resourceless `JobRepository` lives only in that JVM's +heap; when the process exits, every `JobInstance`, `JobExecution`, and `StepExecution` it ever +recorded goes with it. Run 2 starts `productImportJob` as if for the first time, tries to insert +`ABC-0001` again, and collides with what run 1 already committed to the `PRODUCT` table — +a table that, unlike the job repository, *is* backed by the same file-based H2 database both +times. + +This is the practical failure mode: **the job repository's persistence and your own application +data's persistence are two separate decisions**, and it is entirely possible to get the second +one right (a real database, a real table) while getting the first one wrong (resourceless, +because `-jdbc` was left off a dependency list or excluded by a profile without realizing what it +carried). The symptom is not a startup error. It is "my restart didn't restart" or "why did this +job double-insert everything," discovered in whatever environment happens to restart the JVM +between runs — which, for anything running in a container orchestrator, is any redeploy, +crash-restart, or scale-down/scale-up cycle. + +
How to check which one you actually have. Query information_schema.tables (or your database's equivalent) for a table named BATCH_JOB_INSTANCE after your application starts. If it is not there, you have the resourceless job repository, whatever your dependency list looks like on paper — check for an exclusion in spring.autoconfigure.exclude, a conflicting JobRepository bean of your own, or simply a missing -jdbc/-mongodb starter.
+ +## Going deeper + +- `BatchProperties` and `BatchJdbcProperties`, including `spring.batch.job.enabled`, + `spring.batch.jdbc.initialize-schema`, and `spring.batch.jdbc.table-prefix`: the + `spring-configuration-metadata.json` inside `spring-boot-batch-4.1.1.jar` and + `spring-boot-batch-jdbc-4.1.1.jar` is the authoritative list, more current than any blog post + including this one. +- MongoDB-backed job repositories follow the identical pattern via + `spring-boot-starter-batch-data-mongodb`, not covered by this module. + +[Next: Production checklist →](11-production-checklist.md) diff --git a/spring-batch/docs/11-production-checklist.md b/spring-batch/docs/11-production-checklist.md new file mode 100644 index 0000000..0cdceb0 --- /dev/null +++ b/spring-batch/docs/11-production-checklist.md @@ -0,0 +1,47 @@ +# 11. Production checklist + +[← Previous](10-resourceless-vs-jdbc.md) | [README](../README.md) + +A short list, aimed at the gap between "this demo works" and "this is safe to point at a real +feed." Each item links back to the chapter that explains the why. + +- **Confirm the job repository is actually persistent** — query for `BATCH_JOB_INSTANCE` + after startup, don't assume from the dependency list. [Chapter 10](10-resourceless-vs-jdbc.md). +- **Pick chunk size from measurement, not habit.** Smaller costs more round trips; larger costs + more rework per failure. [Chapter 3](03-chunk-oriented-processing.md). +- **Decide skip vs. fail-and-restart per failure type, not per job.** A step can have a + `skipLimit` for genuinely expected bad rows and still fail hard past that limit — the two + are not mutually exclusive. [Chapter 8](08-skip-vs-restart.md). +- **Use `chunk(int).transactionManager(tx)`, not `chunk(int, tx)`,** unless you specifically need + the legacy `SimpleStepBuilder` for something the new model does not yet cover. + [Chapter 2](02-anatomy-of-a-job.md). +- **Do not use `beanMapped()` against a Java record** without checking your exact Spring + Framework version's record support; prefer `itemPreparedStatementSetter` when in doubt. + [Chapter 6](06-jdbc-writer-and-records.md). +- **Give every step you want to always re-run on restart `allowStartIfComplete(true)` + explicitly** — a reporting or notification step that silently gets skipped after a + restart is a surprising, hard-to-notice gap. [Chapter 8](08-skip-vs-restart.md). +- **A custom `ItemReader` that does not implement `ItemStream` restarts from scratch every time**, + which is fine for an idempotent reader and a bug for anything else — check this + deliberately rather than by finding out during an incident. [Chapter 7](07-restartability.md). +- **`spring.batch.job.enabled=false` if you launch jobs yourself** through `JobOperator`, so the + auto-configured runner does not also try. [Chapter 5](05-launching-and-jobparameters.md). +- **Delete or secure any diagnostic endpoint you add while building this out.** This module has + none, but the pattern (an endpoint dumping live `JobExplorer` state) is common enough to call + out: it is invaluable while developing and a liability left in production. + +## Should you even build a custom batch job for this? + +Sometimes the honest answer is no. If the volume is small enough to fit in a single request-response +cycle and does not need to survive a crash mid-way, a scheduled `@Component` method with its own +try/catch is less machinery than a full Job/Step/JobRepository setup, and easier for the next +person to read. Spring Batch earns its complexity when you actually need the things this article +demonstrates — durable progress tracking, transactional chunking, a real restart story +— not by default just because the word "batch" is in the requirements. + +## Further reading + +- [Spring Batch reference documentation](https://docs.spring.io/spring-batch/reference/) (`rel="nofollow"`) +- [What's new in Spring Batch 6](https://docs.spring.io/spring-batch/reference/whatsnew.html) (`rel="nofollow"`) +- [Spring Batch 6.0 Migration Guide](https://github.com/spring-projects/spring-batch/wiki/Spring-Batch-6.0-Migration-Guide) (`rel="nofollow"`, and see [chapter 9](09-corrections.md) for where this specific document was wrong) +- [Spring Boot 4.1.0 release announcement](https://spring.io/blog/2026/06/10/spring-boot-4/) (`rel="nofollow"`) diff --git a/spring-batch/docs/output/01-processor-filter.txt b/spring-batch/docs/output/01-processor-filter.txt new file mode 100644 index 0000000..769333a --- /dev/null +++ b/spring-batch/docs/output/01-processor-filter.txt @@ -0,0 +1,10 @@ +# The processor as a filter: null means skip, not fail + +input : Product[sku=ABC-0001, name=Widget 1, priceCents=1001] +result : Product[sku=ABC-0001, name=Widget 1, priceCents=1001] + +input : Product[sku=abc-0012, name=Widget 12, priceCents=1012] <- lowercase sku, fails ^[A-Z]{3}-\d{4}$ +result : null + +input : Product[sku=ABC-0033, name=Widget 33, priceCents=0] <- priceCents is zero +result : null diff --git a/spring-batch/docs/output/02-exception-hierarchy.txt b/spring-batch/docs/output/02-exception-hierarchy.txt new file mode 100644 index 0000000..23c41e6 --- /dev/null +++ b/spring-batch/docs/output/02-exception-hierarchy.txt @@ -0,0 +1,9 @@ +# Why skip(DataIntegrityViolationException.class) also catches the duplicate-key case + +thrown type : org.springframework.dao.DuplicateKeyException +is a DataIntegrityViolationException? true + +H2's JdbcBatchUpdateException on a UNIQUE-constraint violation is translated by +Spring's SQLExceptionSubclassTranslator into DuplicateKeyException, which extends +DataIntegrityViolationException. A skip policy configured against the parent class +catches the subclass too -- see docs/07-restartability.md and docs/08-skip-vs-restart.md. diff --git a/spring-batch/docs/output/03-stepbuilder-chunk-overloads.txt b/spring-batch/docs/output/03-stepbuilder-chunk-overloads.txt new file mode 100644 index 0000000..04a9948 --- /dev/null +++ b/spring-batch/docs/output/03-stepbuilder-chunk-overloads.txt @@ -0,0 +1,19 @@ +# javap org.springframework.batch.core.step.builder.StepBuilder (spring-batch-core 6.0.5) + +$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.step.builder.StepBuilder +Compiled from "StepBuilder.java" +public class org.springframework.batch.core.step.builder.StepBuilder extends org.springframework.batch.core.step.builder.StepBuilderHelper { + public org.springframework.batch.core.step.builder.StepBuilder(org.springframework.batch.core.repository.JobRepository); + public org.springframework.batch.core.step.builder.StepBuilder(java.lang.String, org.springframework.batch.core.repository.JobRepository); + public org.springframework.batch.core.step.builder.TaskletStepBuilder tasklet(org.springframework.batch.core.step.tasklet.Tasklet, org.springframework.transaction.PlatformTransactionManager); + public org.springframework.batch.core.step.builder.TaskletStepBuilder tasklet(org.springframework.batch.core.step.tasklet.Tasklet); + public org.springframework.batch.core.step.builder.SimpleStepBuilder chunk(int, org.springframework.transaction.PlatformTransactionManager); + public org.springframework.batch.core.step.builder.ChunkOrientedStepBuilder chunk(int); + public org.springframework.batch.core.step.builder.SimpleStepBuilder chunk(org.springframework.batch.infrastructure.repeat.CompletionPolicy, org.springframework.transaction.PlatformTransactionManager); + public org.springframework.batch.core.step.builder.PartitionStepBuilder partitioner(java.lang.String, org.springframework.batch.core.partition.Partitioner); + public org.springframework.batch.core.step.builder.PartitionStepBuilder partitioner(org.springframework.batch.core.step.Step); + public org.springframework.batch.core.step.builder.JobStepBuilder job(org.springframework.batch.core.job.Job); + public org.springframework.batch.core.step.builder.FlowStepBuilder flow(org.springframework.batch.core.job.flow.Flow); + protected org.springframework.batch.core.step.builder.StepBuilder self(); + protected org.springframework.batch.core.step.builder.StepBuilderHelper self(); +} diff --git a/spring-batch/docs/output/04-two-executioncontext-classes.txt b/spring-batch/docs/output/04-two-executioncontext-classes.txt new file mode 100644 index 0000000..001cf6f --- /dev/null +++ b/spring-batch/docs/output/04-two-executioncontext-classes.txt @@ -0,0 +1,48 @@ +# javap: two ExecutionContext classes in two different packages + +$ javap -cp spring-batch-infrastructure-6.0.5.jar org.springframework.batch.infrastructure.item.ExecutionContext +Compiled from "ExecutionContext.java" +public class org.springframework.batch.infrastructure.item.ExecutionContext implements java.io.Serializable { + public org.springframework.batch.infrastructure.item.ExecutionContext(); + public org.springframework.batch.infrastructure.item.ExecutionContext(java.util.Map); + public org.springframework.batch.infrastructure.item.ExecutionContext(org.springframework.batch.infrastructure.item.ExecutionContext); + public void putString(java.lang.String, java.lang.String); + public void putLong(java.lang.String, long); + public void putInt(java.lang.String, int); + public void putDouble(java.lang.String, double); + public void put(java.lang.String, java.lang.Object); + public boolean isDirty(); + public java.lang.String getString(java.lang.String); + public java.lang.String getString(java.lang.String, java.lang.String); + public long getLong(java.lang.String); + public long getLong(java.lang.String, long); + public int getInt(java.lang.String); + public int getInt(java.lang.String, int); + public double getDouble(java.lang.String); + public double getDouble(java.lang.String, double); + public java.lang.Object get(java.lang.String); + public V get(java.lang.String, java.lang.Class); + public V get(java.lang.String, java.lang.Class, V); + public boolean isEmpty(); + public void clearDirtyFlag(); + public java.util.Set> entrySet(); + public java.util.Map toMap(); + public boolean containsKey(java.lang.String); + public java.lang.Object remove(java.lang.String); + public boolean containsValue(java.lang.Object); + public boolean equals(java.lang.Object); + public int hashCode(); + public java.lang.String toString(); + public int size(); +} + +$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.repository.persistence.ExecutionContext +Compiled from "ExecutionContext.java" +public final class org.springframework.batch.core.repository.persistence.ExecutionContext extends java.lang.Record { + public org.springframework.batch.core.repository.persistence.ExecutionContext(java.util.Map, boolean); + public final java.lang.String toString(); + public final int hashCode(); + public final boolean equals(java.lang.Object); + public java.util.Map map(); + public boolean dirty(); +} diff --git a/spring-batch/docs/output/05-happy-path.txt b/spring-batch/docs/output/05-happy-path.txt new file mode 100644 index 0000000..92ad794 --- /dev/null +++ b/spring-batch/docs/output/05-happy-path.txt @@ -0,0 +1,19 @@ +# The happy path: 60 rows in, 58 products out + +$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \ + --import.file=file:scenario-data/clean-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db + +2026-09-13T06:23:05.509Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep] +2026-09-13T06:23:05.530Z WARN 5242 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$ +2026-09-13T06:23:05.543Z WARN 5242 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0 +2026-09-13T06:23:05.570Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 61ms +2026-09-13T06:23:05.590Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep] +REPORT: 58 products now in the PRODUCT table +2026-09-13T06:23:05.600Z INFO 5242 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 13ms +JOB FINISHED: status=COMPLETED exitCode=COMPLETED + +$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION; +READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT +60 | 2 | 58 | 6 +(1 row, 21 ms) diff --git a/spring-batch/docs/output/07-restart-run1-fails.txt b/spring-batch/docs/output/07-restart-run1-fails.txt new file mode 100644 index 0000000..4604c36 --- /dev/null +++ b/spring-batch/docs/output/07-restart-run1-fails.txt @@ -0,0 +1,23 @@ +# Run 1: the poisoned duplicate at row 47 fails the whole chunk + +$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \ + --import.file=file:scenario-data/restart-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db + +2026-09-13T06:23:09.122Z INFO 5302 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep] +2026-09-13T06:23:09.162Z WARN 5302 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$ +2026-09-13T06:23:09.176Z WARN 5302 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0 +org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement: +Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement: +2026-09-13T06:23:09.205Z INFO 5302 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 85ms +JOB FINISHED: status=FAILED exitCode=FAILED + +$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION; +STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT +FAILED | 50 | 2 | 38 | 4 | 1 +(1 row, 21 ms) + +$ SELECT COUNT(*) FROM PRODUCT; +COUNT(*) +38 +(1 row, 20 ms) diff --git a/spring-batch/docs/output/08-restart-run2-resumes.txt b/spring-batch/docs/output/08-restart-run2-resumes.txt new file mode 100644 index 0000000..8b09218 --- /dev/null +++ b/spring-batch/docs/output/08-restart-run2-resumes.txt @@ -0,0 +1,25 @@ +# Run 2: a brand-new JVM, the SAME job parameters, the corrected file + +sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change + +$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \ + --import.file=file:scenario-data/restart-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file + +2026-09-13T06:23:13.205Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep] +2026-09-13T06:23:13.232Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 28ms +2026-09-13T06:23:13.240Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep] +REPORT: 58 products now in the PRODUCT table +2026-09-13T06:23:13.246Z INFO 5386 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 6ms +JOB FINISHED: status=COMPLETED exitCode=COMPLETED + +$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id; +STEP_EXECUTION_ID | JOB_EXECUTION_ID | STATUS | READ_COUNT | WRITE_COUNT | COMMIT_COUNT | ROLLBACK_COUNT +1 | 1 | FAILED | 50 | 38 | 4 | 1 +2 | 2 | COMPLETED | 20 | 20 | 2 | 0 +(2 rows, 22 ms) + +$ SELECT COUNT(*) FROM PRODUCT; +COUNT(*) +58 +(1 row, 19 ms) diff --git a/spring-batch/docs/output/09-commandlinejobrunner-deprecated-not-removed.txt b/spring-batch/docs/output/09-commandlinejobrunner-deprecated-not-removed.txt new file mode 100644 index 0000000..28b834d --- /dev/null +++ b/spring-batch/docs/output/09-commandlinejobrunner-deprecated-not-removed.txt @@ -0,0 +1,23 @@ +# javap: CommandLineJobRunner is deprecated (forRemoval), not removed, in 6.0.5 + +$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated" +public class org.springframework.batch.core.launch.support.CommandLineJobRunner + minor version: 0 + major version: 61 + flags: (0x0021) ACC_PUBLIC, ACC_SUPER +-- + #503 = Utf8 Deprecated + #504 = Utf8 RuntimeVisibleAnnotations + #505 = Utf8 Ljava/lang/Deprecated; + #506 = Utf8 since + #507 = Utf8 6.0 + #508 = Utf8 forRemoval +-- + Deprecated: true + RuntimeVisibleAnnotations: + 0: #505(#506=s#507,#508=Z#509) + java.lang.Deprecated( + since="6.0" + forRemoval=true + ) +-- diff --git a/spring-batch/docs/output/10-skip-instead-of-fail.txt b/spring-batch/docs/output/10-skip-instead-of-fail.txt new file mode 100644 index 0000000..4f2258d --- /dev/null +++ b/spring-batch/docs/output/10-skip-instead-of-fail.txt @@ -0,0 +1,27 @@ +# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES + +$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \ + --import.file=file:scenario-data/skip-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db + +2026-09-13T06:23:17.111Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [importStep] +2026-09-13T06:23:17.139Z WARN 5469 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering abc-0012: does not match ^[A-Z]{3}-\d{4}$ +2026-09-13T06:23:17.150Z WARN 5469 --- [ main] c.a.b.p.ProductValidatingProcessor : filtering ABC-0033: priceCents must be positive, was 0 +Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement: +Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement: +SKIPPED on write: ABC-0005 (DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 5 */ 'ABC-0005' )"; SQL statement: +2026-09-13T06:23:17.235Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Step: [importStep] executed in 124ms +2026-09-13T06:23:17.247Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Executing step: [reportStep] +REPORT: 57 products now in the PRODUCT table +2026-09-13T06:23:17.256Z INFO 5469 --- [ main] o.s.batch.core.step.AbstractStep : Step: [reportStep] executed in 9ms +JOB FINISHED: status=COMPLETED exitCode=COMPLETED + +$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION; +STATUS | READ_COUNT | FILTER_COUNT | WRITE_COUNT | WRITE_SKIP_COUNT | COMMIT_COUNT | ROLLBACK_COUNT +COMPLETED | 60 | 2 | 57 | 1 | 6 | 2 +(1 row, 22 ms) + +$ SELECT COUNT(*) FROM PRODUCT; +COUNT(*) +57 +(1 row, 19 ms) diff --git a/spring-batch/docs/output/11-resourceless-forgets-everything.txt b/spring-batch/docs/output/11-resourceless-forgets-everything.txt new file mode 100644 index 0000000..d7accd4 --- /dev/null +++ b/spring-batch/docs/output/11-resourceless-forgets-everything.txt @@ -0,0 +1,21 @@ +# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory + +$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \ + --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \ + --import.file=file:scenario-data/resourceless-demo/input.csv \ + --spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db + +-- run 1 -- +REPORT: 58 products now in the PRODUCT table +JOB FINISHED: status=COMPLETED exitCode=COMPLETED + +-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table -- +org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 1 */ 'ABC-0001' )"; SQL statement: +Caused by: org.springframework.dao.DuplicateKeyException: PreparedStatementCallback; SQL [INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)]; Unique index or primary key violation: "PUBLIC.CONSTRAINT_18 INDEX PUBLIC.CONSTRAINT_INDEX_1 ON PUBLIC.PRODUCT(SKU NULLS FIRST) VALUES ( /* 1 */ 'ABC-0001' )"; SQL statement: +JOB FINISHED: status=FAILED exitCode=FAILED + +No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea +run 1 ever happened, so it tries the whole job again and collides with what run 1 already +wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration +used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead, +which ImportRunner treats as "nothing to do" -- see docs/07-restartability.md. diff --git a/spring-batch/pom.xml b/spring-batch/pom.xml new file mode 100644 index 0000000..f983da9 --- /dev/null +++ b/spring-batch/pom.xml @@ -0,0 +1,62 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + spring-batch + 1.0.0 + spring-batch + Spring Batch on Boot 4.1: jobs, steps, chunk processing and restartability + + + 25 + + + + + + org.springframework.boot + spring-boot-starter-batch-jdbc + + + + org.springframework.boot + spring-boot-starter-jdbc + + + + + com.h2database + h2 + runtime + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + diff --git a/spring-batch/scripts/capture-javap.sh b/spring-batch/scripts/capture-javap.sh new file mode 100755 index 0000000..6d089ae --- /dev/null +++ b/spring-batch/scripts/capture-javap.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# Captures real javap output pinning the Spring Batch 6.0 API facts this article relies on: +# the ChunkOrientedStepBuilder that chunk(int) now returns, the two different ExecutionContext +# classes, the package move of RepeatStatus, and CommandLineJobRunner's deprecation (NOT removal +# -- an earlier draft trusted a migration-guide summary that said "removed" and this transcript +# is what caught the error; see docs/09-corrections.md). +# +# ./scripts/capture-javap.sh +# +# Needs the spring-batch-core and spring-batch-infrastructure 6.0.5 jars in the local Maven repo +# (a normal `mvn -B -o package` in this module already pulls them in). +set -euo pipefail +cd "$(dirname "$0")/.." + +CORE=$(find "$HOME/.m2" -name 'spring-batch-core-6.0.5.jar' | head -1) +INFRA=$(find "$HOME/.m2" -name 'spring-batch-infrastructure-6.0.5.jar' | head -1) +: "${CORE:?spring-batch-core-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}" +: "${INFRA:?spring-batch-infrastructure-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}" + +WORK=$(mktemp -d) +mkdir -p "$WORK/core" "$WORK/infra" +unzip -o -q "$CORE" -d "$WORK/core" +unzip -o -q "$INFRA" -d "$WORK/infra" + +mkdir -p docs/output + +{ + echo "# javap org.springframework.batch.core.step.builder.StepBuilder (spring-batch-core 6.0.5)" + echo + echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.step.builder.StepBuilder' + javap -cp "$WORK/core" org.springframework.batch.core.step.builder.StepBuilder +} > docs/output/03-stepbuilder-chunk-overloads.txt + +{ + echo "# javap: two ExecutionContext classes in two different packages" + echo + echo '$ javap -cp spring-batch-infrastructure-6.0.5.jar org.springframework.batch.infrastructure.item.ExecutionContext' + javap -cp "$WORK/infra" org.springframework.batch.infrastructure.item.ExecutionContext + echo + echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.repository.persistence.ExecutionContext' + javap -cp "$WORK/core" org.springframework.batch.core.repository.persistence.ExecutionContext +} > docs/output/04-two-executioncontext-classes.txt + +{ + echo "# javap: CommandLineJobRunner is deprecated (forRemoval), not removed, in 6.0.5" + echo + echo '$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated"' + javap -verbose -cp "$WORK/core" org.springframework.batch.core.launch.support.CommandLineJobRunner \ + | grep -A3 '^public class\|Deprecated' | head -20 +} > docs/output/09-commandlinejobrunner-deprecated-not-removed.txt + +rm -rf "$WORK" +echo "wrote docs/output/03-stepbuilder-chunk-overloads.txt" +echo "wrote docs/output/04-two-executioncontext-classes.txt" +echo "wrote docs/output/09-commandlinejobrunner-deprecated-not-removed.txt" diff --git a/spring-batch/scripts/capture-scenarios.sh b/spring-batch/scripts/capture-scenarios.sh new file mode 100755 index 0000000..8895fe4 --- /dev/null +++ b/spring-batch/scripts/capture-scenarios.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# Runs the three end-to-end scenarios this article is built on and writes what actually +# happened -- console output plus the real BATCH_STEP_EXECUTION and PRODUCT rows -- to +# docs/output/. Nothing here is retyped; the numbers in the article come out of these files. +# +# ./scripts/capture-scenarios.sh +# +# Needs target/spring-batch-1.0.0.jar (run `mvn -B -o -DskipTests package` first) and a JDK 25. +# Each scenario gets its own file-based H2 database under scenario-data// so the three +# runs never interfere with each other. +set -euo pipefail +cd "$(dirname "$0")/.." + +JAR=target/spring-batch-1.0.0.jar +: "${JAR:?run mvn -B -o -DskipTests package first}" +H2_JAR=$(find "$HOME/.m2" -name 'h2-2.4.240.jar' | head -1) +mkdir -p docs/output +rm -rf scenario-data +mkdir -p scenario-data + +query() { + # $1 = db dir, $2 = sql + java -cp "$H2_JAR" org.h2.tools.Shell -url "jdbc:h2:file:${PWD}/$1/db" -user sa -password "" -sql "$2" 2>/dev/null \ + | grep -v '^$' +} + +run_scenario() { + # $1 = name, $2 = profile, $3 = source csv + local name="$1" profile="$2" src="$3" dir + dir="scenario-data/$1" + mkdir -p "$dir" + cp "$src" "$dir/input.csv" + java -jar "$JAR" \ + --spring.profiles.active="$profile" \ + --import.file="file:${dir}/input.csv" \ + --spring.datasource.url="jdbc:h2:file:${PWD}/${dir}/db" \ + 2>&1 | grep -E "REPORT:|JOB FINISHED|SKIPPED on write|DuplicateKeyException:|WARN.*ProductValidatingProcessor|Executing step|executed in" || true +} + +echo "== scenario: clean (05-happy-path.txt)" +{ + echo "# The happy path: 60 rows in, 58 products out" + echo + echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \' + echo ' --import.file=file:scenario-data/clean-demo/input.csv \' + echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db' + echo + run_scenario clean-demo clean src/main/resources/data/products.csv + echo + echo '$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;' + query scenario-data/clean-demo "SELECT READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';" +} > docs/output/05-happy-path.txt + +echo "== scenario: broken, run 1 (07-restart-run1-fails.txt)" +{ + echo "# Run 1: the poisoned duplicate at row 47 fails the whole chunk" + echo + echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \' + echo ' --import.file=file:scenario-data/restart-demo/input.csv \' + echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db' + echo + run_scenario restart-demo broken src/main/resources/data/products-poison.csv + echo + echo '$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;' + query scenario-data/restart-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';" + echo + echo '$ SELECT COUNT(*) FROM PRODUCT;' + query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;" +} > docs/output/07-restart-run1-fails.txt + +echo "== fixing the duplicate in place (same file, same line count) ==" +sed -i '48s/^ABC-0005,Widget 47,1047$/ABC-0999,Widget 47,1047/' scenario-data/restart-demo/input.csv +grep -n "Widget 47" scenario-data/restart-demo/input.csv + +echo "== scenario: broken, run 2 = restart in a fresh JVM (07-restart-run2-resumes.txt)" +{ + echo "# Run 2: a brand-new JVM, the SAME job parameters, the corrected file" + echo + echo "sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change" + echo + echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \' + echo ' --import.file=file:scenario-data/restart-demo/input.csv \' + echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file' + echo + java -jar "$JAR" \ + --spring.profiles.active=broken \ + --import.file="file:scenario-data/restart-demo/input.csv" \ + --spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/restart-demo/db" \ + 2>&1 | grep -E "REPORT:|JOB FINISHED|Executing step|executed in" || true + echo + echo '$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;' + query scenario-data/restart-demo "SELECT STEP_EXECUTION_ID, JOB_EXECUTION_ID, STATUS, READ_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep' ORDER BY STEP_EXECUTION_ID;" + echo + echo '$ SELECT COUNT(*) FROM PRODUCT;' + query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;" +} > docs/output/08-restart-run2-resumes.txt + +echo "== scenario: skip (10-skip-instead-of-fail.txt)" +{ + echo "# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES" + echo + echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \' + echo ' --import.file=file:scenario-data/skip-demo/input.csv \' + echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db' + echo + run_scenario skip-demo skip src/main/resources/data/products-poison.csv + echo + echo '$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;' + query scenario-data/skip-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, WRITE_SKIP_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';" + echo + echo '$ SELECT COUNT(*) FROM PRODUCT;' + query scenario-data/skip-demo "SELECT COUNT(*) FROM PRODUCT;" +} > docs/output/10-skip-instead-of-fail.txt + +echo "== scenario: resourceless (11-resourceless-forgets-everything.txt)" +rm -rf scenario-data/resourceless-demo +mkdir -p scenario-data/resourceless-demo +cp src/main/resources/data/products.csv scenario-data/resourceless-demo/input.csv +{ + echo "# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory" + echo + echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \' + echo ' --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \' + echo ' --import.file=file:scenario-data/resourceless-demo/input.csv \' + echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db' + echo + echo "-- run 1 --" + java -jar "$JAR" \ + --spring.profiles.active=clean \ + --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \ + --import.file="file:scenario-data/resourceless-demo/input.csv" \ + --spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \ + 2>&1 | grep -E "REPORT:|JOB FINISHED" || true + echo + echo "-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --" + java -jar "$JAR" \ + --spring.profiles.active=clean \ + --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \ + --import.file="file:scenario-data/resourceless-demo/input.csv" \ + --spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \ + 2>&1 | grep -E "REPORT:|JOB FINISHED|DuplicateKeyException:" | head -3 || true + echo + echo "No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea" + echo "run 1 ever happened, so it tries the whole job again and collides with what run 1 already" + echo "wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration" + echo "used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead," + echo "which ImportRunner treats as \"nothing to do\" -- see docs/07-restartability.md." +} > docs/output/11-resourceless-forgets-everything.txt + +echo +echo "docs/output:" +ls -1 docs/output diff --git a/spring-batch/scripts/run-all.sh b/spring-batch/scripts/run-all.sh new file mode 100755 index 0000000..ee593f6 --- /dev/null +++ b/spring-batch/scripts/run-all.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Regenerates every file under docs/output/. +# +# ./scripts/run-all.sh +# +# Needs a JDK 25 and Maven 3.9, with dependencies already resolved once online (mvn needs a +# networked run before -o works for spring-boot:run / surefire -- see companion-repo notes). +set -euo pipefail +cd "$(dirname "$0")/.." + +echo "== unit tests (transcripts 01-02)" +mvn -B -o test + +echo "== javap transcripts (03, 04, 09)" +./scripts/capture-javap.sh + +echo "== package" +mvn -B -o -DskipTests package + +echo "== end-to-end scenarios (05, 07, 08, 10, 11)" +./scripts/capture-scenarios.sh + +echo +echo "docs/output:" +ls -1 docs/output diff --git a/spring-batch/scripts/run-scenario.sh b/spring-batch/scripts/run-scenario.sh new file mode 100755 index 0000000..6769745 --- /dev/null +++ b/spring-batch/scripts/run-scenario.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Runs one scenario against its own external H2 file database and CSV drop file (both under +# scenario-data//, outside target/ so a rebuild never touches them -- see +# docs/07-restartability.md for why that matters: `mvn spring-boot:run` re-copies +# src/main/resources over target/classes before every run, which would silently undo an +# in-place "fix" to a classpath resource between two runs of the same scenario. +# +# ./scripts/run-scenario.sh +# +# Reuses the packaged jar (target/spring-batch-1.0.0.jar); run `mvn -B -o -DskipTests package` +# first if it is missing or stale. +set -euo pipefail +cd "$(dirname "$0")/.." + +NAME="${1:?usage: run-scenario.sh }" +PROFILE="${2:?usage: run-scenario.sh }" +SRC_CSV="${3:?usage: run-scenario.sh }" + +DIR="scenario-data/${NAME}" +mkdir -p "$DIR" +if [ ! -f "$DIR/input.csv" ]; then + cp "$SRC_CSV" "$DIR/input.csv" +fi + +java -jar target/spring-batch-1.0.0.jar \ + --spring.profiles.active="$PROFILE" \ + --import.file="file:${DIR}/input.csv" \ + --spring.datasource.url="jdbc:h2:file:${PWD}/${DIR}/db" diff --git a/spring-batch/src/main/java/com/ankurm/batch/BatchDemoApplication.java b/spring-batch/src/main/java/com/ankurm/batch/BatchDemoApplication.java new file mode 100644 index 0000000..e9c15bb --- /dev/null +++ b/spring-batch/src/main/java/com/ankurm/batch/BatchDemoApplication.java @@ -0,0 +1,11 @@ +package com.ankurm.batch; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class BatchDemoApplication { + public static void main(String[] args) { + SpringApplication.run(BatchDemoApplication.class, args); + } +} diff --git a/spring-batch/src/main/java/com/ankurm/batch/config/BatchConfig.java b/spring-batch/src/main/java/com/ankurm/batch/config/BatchConfig.java new file mode 100644 index 0000000..77d424e --- /dev/null +++ b/spring-batch/src/main/java/com/ankurm/batch/config/BatchConfig.java @@ -0,0 +1,189 @@ +package com.ankurm.batch.config; + +import com.ankurm.batch.domain.Product; +import com.ankurm.batch.processing.ProductValidatingProcessor; +import org.springframework.batch.core.job.Job; +import org.springframework.batch.core.job.builder.JobBuilder; +import org.springframework.batch.core.listener.SkipListener; +import org.springframework.batch.core.repository.JobRepository; +import org.springframework.batch.core.step.Step; +import org.springframework.batch.core.step.builder.StepBuilder; +import org.springframework.batch.core.step.tasklet.Tasklet; +import org.springframework.batch.infrastructure.item.ItemProcessor; +import org.springframework.batch.infrastructure.item.ItemWriter; +import org.springframework.batch.infrastructure.item.database.builder.JdbcBatchItemWriterBuilder; +import org.springframework.batch.infrastructure.item.file.builder.FlatFileItemReaderBuilder; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Profile; +import org.springframework.core.io.Resource; +import org.springframework.dao.DataIntegrityViolationException; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.transaction.PlatformTransactionManager; + +/** + * Wires the same "import products, then report a count" job three different ways, selected by + * Spring profile: + * + *
    + *
  • {@code clean} — the happy path, chapters 1-3
    • + *
  • {@code broken} — no fault tolerance, chapter 7 (restartability)
    • + *
  • {@code skip} — {@code faultTolerant().skip(...)}, chapter 8 (skip vs. restart)
    • + *
+ * + * All three read {@link #productReader}, run it through {@link ProductValidatingProcessor}, and + * write with {@link #productWriter}; only the step's fault-tolerance configuration and the input + * file differ. See {@code docs/02-anatomy-of-a-job.md} for why the job and step beans look like + * this rather than the {@code JobBuilderFactory} / {@code StepBuilderFactory} shape older + * tutorials still show (that pair was removed years before this article; it is not a Boot 4.1 + * surprise, just a dead end still copy-pasted in 2026). + */ +@Configuration +public class BatchConfig { + + @Value("${import.file:classpath:data/products.csv}") + private Resource inputFile; + + // ---- shared reader / processor / writer ----------------------------------------------- + + @Bean + public org.springframework.batch.infrastructure.item.file.FlatFileItemReader productReader() { + return new FlatFileItemReaderBuilder() + .name("productReader") + .resource(inputFile) + .linesToSkip(1) + .delimited().delimiter(",").names("sku", "name", "priceCents") + .fieldSetMapper(fs -> new Product(fs.readString("sku"), fs.readString("name"), fs.readLong("priceCents"))) + .build(); + } + + @Bean + public ItemProcessor productProcessor() { + return new ProductValidatingProcessor(); + } + + /** + * Deliberately NOT {@code beanMapped()}: {@code BeanPropertySqlParameterSource} looks for + * {@code getSku()}, {@code getName()}, {@code getPriceCents()} via standard JavaBean + * introspection, and a Java record's accessors are {@code sku()}, {@code name()}, + * {@code priceCents()} — no {@code get} prefix. {@code docs/06-jdbc-writer-and-records.md} + * has the transcript of {@code beanMapped()} silently writing every column as NULL against + * this exact record before this was caught. + */ + @Bean + public ItemWriter productWriter(JdbcTemplate jdbcTemplate) { + return new JdbcBatchItemWriterBuilder() + .dataSource(jdbcTemplate.getDataSource()) + .sql("INSERT INTO PRODUCT (sku, name, price_cents) VALUES (?, ?, ?)") + .itemPreparedStatementSetter((item, ps) -> { + ps.setString(1, item.sku()); + ps.setString(2, item.name()); + ps.setLong(3, item.priceCents()); + }) + .assertUpdates(true) + .build(); + } + + @Bean + public Tasklet reportTasklet(JdbcTemplate jdbcTemplate) { + return (contribution, chunkContext) -> { + int count = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM PRODUCT", Integer.class); + System.out.println("REPORT: " + count + " products now in the PRODUCT table"); + return org.springframework.batch.infrastructure.repeat.RepeatStatus.FINISHED; + }; + } + + // ---- clean profile: happy path, no fault tolerance needed ------------------------------ + + @Bean + @Profile("clean") + public Step importStepClean(JobRepository jobRepository, PlatformTransactionManager transactionManager, + org.springframework.batch.infrastructure.item.file.FlatFileItemReader productReader, + ItemProcessor productProcessor, ItemWriter productWriter) { + return new StepBuilder("importStep", jobRepository) + .chunk(10) + .transactionManager(transactionManager) + .reader(productReader) + .processor(productProcessor) + .writer(productWriter) + .build(); + } + + // ---- broken profile: no fault tolerance -> the whole chunk fails, job FAILS ------------ + + @Bean + @Profile("broken") + public Step importStepBroken(JobRepository jobRepository, PlatformTransactionManager transactionManager, + org.springframework.batch.infrastructure.item.file.FlatFileItemReader productReader, + ItemProcessor productProcessor, ItemWriter productWriter) { + return new StepBuilder("importStep", jobRepository) + .chunk(10) + .transactionManager(transactionManager) + .reader(productReader) + .processor(productProcessor) + .writer(productWriter) + .build(); + } + + // ---- skip profile: faultTolerant + skip -> the bad item is skipped, job COMPLETES ----- + + @Bean + @Profile("skip") + public Step importStepSkip(JobRepository jobRepository, PlatformTransactionManager transactionManager, + org.springframework.batch.infrastructure.item.file.FlatFileItemReader productReader, + ItemProcessor productProcessor, ItemWriter productWriter) { + return new StepBuilder("importStep", jobRepository) + .chunk(10) + .transactionManager(transactionManager) + .reader(productReader) + .processor(productProcessor) + .writer(productWriter) + .faultTolerant() + .skip(DataIntegrityViolationException.class) + .skipLimit(3) + .listener(skipListener()) + .build(); + } + + @Bean + public SkipListener skipListener() { + return new SkipListener<>() { + @Override + public void onSkipInWrite(Product item, Throwable t) { + System.out.println("SKIPPED on write: " + item.sku() + " (" + t.getClass().getSimpleName() + ": " + t.getMessage() + ")"); + } + }; + } + + // ---- reportStep: always runs, even if the step before it was already COMPLETED -------- + + @Bean + public Step reportStep(JobRepository jobRepository, PlatformTransactionManager transactionManager, + Tasklet reportTasklet) { + return new StepBuilder("reportStep", jobRepository) + .tasklet(reportTasklet, transactionManager) + .allowStartIfComplete(true) + .build(); + } + + // ---- one Job bean per profile, all built the same way ---------------------------------- + + @Bean + @Profile("clean") + public Job productImportJobClean(JobRepository jobRepository, Step importStepClean, Step reportStep) { + return new JobBuilder("productImportJob", jobRepository).start(importStepClean).next(reportStep).build(); + } + + @Bean + @Profile("broken") + public Job productImportJobBroken(JobRepository jobRepository, Step importStepBroken, Step reportStep) { + return new JobBuilder("productImportJob", jobRepository).start(importStepBroken).next(reportStep).build(); + } + + @Bean + @Profile("skip") + public Job productImportJobSkip(JobRepository jobRepository, Step importStepSkip, Step reportStep) { + return new JobBuilder("productImportJob", jobRepository).start(importStepSkip).next(reportStep).build(); + } +} diff --git a/spring-batch/src/main/java/com/ankurm/batch/domain/Product.java b/spring-batch/src/main/java/com/ankurm/batch/domain/Product.java new file mode 100644 index 0000000..0c21811 --- /dev/null +++ b/spring-batch/src/main/java/com/ankurm/batch/domain/Product.java @@ -0,0 +1,12 @@ +package com.ankurm.batch.domain; + +/** + * One validated row, ready to be written to the {@code PRODUCT} table. + * + *

See {@code docs/03-chunk-oriented-processing.md} for how instances of this record move + * through the reader → processor → writer pipeline in chunks, and + * {@code docs/06-jdbc-writer-and-records.md} for why the writer below does not use + * {@code beanMapped()} against this record. + */ +public record Product(String sku, String name, long priceCents) { +} diff --git a/spring-batch/src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java b/spring-batch/src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java new file mode 100644 index 0000000..38b70cc --- /dev/null +++ b/spring-batch/src/main/java/com/ankurm/batch/processing/ProductValidatingProcessor.java @@ -0,0 +1,35 @@ +package com.ankurm.batch.processing; + +import com.ankurm.batch.domain.Product; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.springframework.batch.infrastructure.item.ItemProcessor; + +import java.util.regex.Pattern; + +/** + * Chapter 4 ({@code docs/04-item-processor-as-filter.md}): a processor that returns + * {@code null} for a row it does not like. Spring Batch treats a {@code null} return as + * "filter this item" — it is counted separately from both reads and writes, and it + * never reaches the writer or the database. This is deliberately NOT how the poisoned + * duplicate SKU is handled: that one is format-valid and only fails at the database, which is + * the point of chapter 7. + */ +public class ProductValidatingProcessor implements ItemProcessor { + + private static final Log log = LogFactory.getLog(ProductValidatingProcessor.class); + private static final Pattern SKU_PATTERN = Pattern.compile("^[A-Z]{3}-\\d{4}$"); + + @Override + public Product process(Product item) { + if (!SKU_PATTERN.matcher(item.sku()).matches()) { + log.warn("filtering " + item.sku() + ": does not match " + SKU_PATTERN.pattern()); + return null; + } + if (item.priceCents() <= 0) { + log.warn("filtering " + item.sku() + ": priceCents must be positive, was " + item.priceCents()); + return null; + } + return item; + } +} diff --git a/spring-batch/src/main/java/com/ankurm/batch/runner/ImportRunner.java b/spring-batch/src/main/java/com/ankurm/batch/runner/ImportRunner.java new file mode 100644 index 0000000..908342d --- /dev/null +++ b/spring-batch/src/main/java/com/ankurm/batch/runner/ImportRunner.java @@ -0,0 +1,50 @@ +package com.ankurm.batch.runner; + +import org.springframework.batch.core.job.Job; +import org.springframework.batch.core.job.parameters.JobParametersBuilder; +import org.springframework.batch.core.launch.JobOperator; +import org.springframework.boot.ApplicationArguments; +import org.springframework.boot.ApplicationRunner; +import org.springframework.stereotype.Component; + +/** + * Chapter 5 ({@code docs/05-launching-and-jobparameters.md}) and chapter 7 + * ({@code docs/07-restartability.md}): calling {@link JobOperator#start} with the SAME + * identifying job parameter every time is, on its own, the restart mechanism. There is no + * separate "restart" button here: if a JobInstance with these identifying parameters already + * exists and its last execution did not complete, {@code start} runs a new JobExecution against + * that same instance and Spring Batch resumes each step from where its own ExecutionContext says + * it left off. If the last execution DID complete, {@code start} throws + * {@link org.springframework.batch.core.launch.JobInstanceAlreadyCompleteException}, which this + * runner treats as "nothing to do" rather than a failure. + * + *

{@code spring.batch.job.enabled} is left at its Boot default of {@code true}, but the + * auto-configured {@code JobLauncherApplicationRunner} would launch every Job bean in the + * context using default parameters, which collides with the explicit control this module wants + * to demonstrate -- see {@code application.yml}, where it is turned off. + */ +@Component +public class ImportRunner implements ApplicationRunner { + + private final JobOperator jobOperator; + private final Job job; + + public ImportRunner(JobOperator jobOperator, Job job) { + this.jobOperator = jobOperator; + this.job = job; + } + + @Override + public void run(ApplicationArguments args) throws Exception { + var params = new JobParametersBuilder() + .addString("batch.run", "demo") // identifying: same value = same JobInstance = restart target + .toJobParameters(); + try { + var execution = jobOperator.start(job, params); + System.out.println("JOB FINISHED: status=" + execution.getStatus() + + " exitCode=" + execution.getExitStatus().getExitCode()); + } catch (org.springframework.batch.core.launch.JobInstanceAlreadyCompleteException e) { + System.out.println("JOB ALREADY COMPLETE: " + e.getMessage()); + } + } +} diff --git a/spring-batch/src/main/resources/application.yml b/spring-batch/src/main/resources/application.yml new file mode 100644 index 0000000..3982df9 --- /dev/null +++ b/spring-batch/src/main/resources/application.yml @@ -0,0 +1,22 @@ +spring: + batch: + job: + # ImportRunner drives the job explicitly through JobOperator. The auto-configured + # JobLauncherApplicationRunner would ALSO launch every Job bean it finds, using + # parameterless defaults -- turning this off avoids running the job twice on every startup. + enabled: false + jdbc: + initialize-schema: always + datasource: + url: jdbc:h2:file:./data/batchdb;AUTO_SERVER=TRUE + username: sa + password: "" + driver-class-name: org.h2.Driver + sql: + init: + mode: always + schema-locations: classpath:schema.sql + +logging: + level: + org.springframework.batch: INFO diff --git a/spring-batch/src/main/resources/data/products-poison.csv b/spring-batch/src/main/resources/data/products-poison.csv new file mode 100644 index 0000000..e96e211 --- /dev/null +++ b/spring-batch/src/main/resources/data/products-poison.csv @@ -0,0 +1,61 @@ +sku,name,priceCents +ABC-0001,Widget 1,1001 +ABC-0002,Widget 2,1002 +ABC-0003,Widget 3,1003 +ABC-0004,Widget 4,1004 +ABC-0005,Widget 5,1005 +ABC-0006,Widget 6,1006 +ABC-0007,Widget 7,1007 +ABC-0008,Widget 8,1008 +ABC-0009,Widget 9,1009 +ABC-0010,Widget 10,1010 +ABC-0011,Widget 11,1011 +abc-0012,Widget 12,1012 +ABC-0013,Widget 13,1013 +ABC-0014,Widget 14,1014 +ABC-0015,Widget 15,1015 +ABC-0016,Widget 16,1016 +ABC-0017,Widget 17,1017 +ABC-0018,Widget 18,1018 +ABC-0019,Widget 19,1019 +ABC-0020,Widget 20,1020 +ABC-0021,Widget 21,1021 +ABC-0022,Widget 22,1022 +ABC-0023,Widget 23,1023 +ABC-0024,Widget 24,1024 +ABC-0025,Widget 25,1025 +ABC-0026,Widget 26,1026 +ABC-0027,Widget 27,1027 +ABC-0028,Widget 28,1028 +ABC-0029,Widget 29,1029 +ABC-0030,Widget 30,1030 +ABC-0031,Widget 31,1031 +ABC-0032,Widget 32,1032 +ABC-0033,Widget 33,0 +ABC-0034,Widget 34,1034 +ABC-0035,Widget 35,1035 +ABC-0036,Widget 36,1036 +ABC-0037,Widget 37,1037 +ABC-0038,Widget 38,1038 +ABC-0039,Widget 39,1039 +ABC-0040,Widget 40,1040 +ABC-0041,Widget 41,1041 +ABC-0042,Widget 42,1042 +ABC-0043,Widget 43,1043 +ABC-0044,Widget 44,1044 +ABC-0045,Widget 45,1045 +ABC-0046,Widget 46,1046 +ABC-0005,Widget 47,1047 +ABC-0048,Widget 48,1048 +ABC-0049,Widget 49,1049 +ABC-0050,Widget 50,1050 +ABC-0051,Widget 51,1051 +ABC-0052,Widget 52,1052 +ABC-0053,Widget 53,1053 +ABC-0054,Widget 54,1054 +ABC-0055,Widget 55,1055 +ABC-0056,Widget 56,1056 +ABC-0057,Widget 57,1057 +ABC-0058,Widget 58,1058 +ABC-0059,Widget 59,1059 +ABC-0060,Widget 60,1060 diff --git a/spring-batch/src/main/resources/data/products.csv b/spring-batch/src/main/resources/data/products.csv new file mode 100644 index 0000000..bdeb5bb --- /dev/null +++ b/spring-batch/src/main/resources/data/products.csv @@ -0,0 +1,61 @@ +sku,name,priceCents +ABC-0001,Widget 1,1001 +ABC-0002,Widget 2,1002 +ABC-0003,Widget 3,1003 +ABC-0004,Widget 4,1004 +ABC-0005,Widget 5,1005 +ABC-0006,Widget 6,1006 +ABC-0007,Widget 7,1007 +ABC-0008,Widget 8,1008 +ABC-0009,Widget 9,1009 +ABC-0010,Widget 10,1010 +ABC-0011,Widget 11,1011 +abc-0012,Widget 12,1012 +ABC-0013,Widget 13,1013 +ABC-0014,Widget 14,1014 +ABC-0015,Widget 15,1015 +ABC-0016,Widget 16,1016 +ABC-0017,Widget 17,1017 +ABC-0018,Widget 18,1018 +ABC-0019,Widget 19,1019 +ABC-0020,Widget 20,1020 +ABC-0021,Widget 21,1021 +ABC-0022,Widget 22,1022 +ABC-0023,Widget 23,1023 +ABC-0024,Widget 24,1024 +ABC-0025,Widget 25,1025 +ABC-0026,Widget 26,1026 +ABC-0027,Widget 27,1027 +ABC-0028,Widget 28,1028 +ABC-0029,Widget 29,1029 +ABC-0030,Widget 30,1030 +ABC-0031,Widget 31,1031 +ABC-0032,Widget 32,1032 +ABC-0033,Widget 33,0 +ABC-0034,Widget 34,1034 +ABC-0035,Widget 35,1035 +ABC-0036,Widget 36,1036 +ABC-0037,Widget 37,1037 +ABC-0038,Widget 38,1038 +ABC-0039,Widget 39,1039 +ABC-0040,Widget 40,1040 +ABC-0041,Widget 41,1041 +ABC-0042,Widget 42,1042 +ABC-0043,Widget 43,1043 +ABC-0044,Widget 44,1044 +ABC-0045,Widget 45,1045 +ABC-0046,Widget 46,1046 +ABC-0047,Widget 47,1047 +ABC-0048,Widget 48,1048 +ABC-0049,Widget 49,1049 +ABC-0050,Widget 50,1050 +ABC-0051,Widget 51,1051 +ABC-0052,Widget 52,1052 +ABC-0053,Widget 53,1053 +ABC-0054,Widget 54,1054 +ABC-0055,Widget 55,1055 +ABC-0056,Widget 56,1056 +ABC-0057,Widget 57,1057 +ABC-0058,Widget 58,1058 +ABC-0059,Widget 59,1059 +ABC-0060,Widget 60,1060 diff --git a/spring-batch/src/main/resources/schema.sql b/spring-batch/src/main/resources/schema.sql new file mode 100644 index 0000000..85bef86 --- /dev/null +++ b/spring-batch/src/main/resources/schema.sql @@ -0,0 +1,10 @@ +-- Application table, separate from the BATCH_* tables that BatchJdbcAutoConfiguration +-- creates via spring.batch.jdbc.initialize-schema. The UNIQUE constraint on sku is what turns +-- the poisoned duplicate row into a real DataIntegrityViolationException at chunk-commit time +-- rather than something the processor could have caught by looking at one row in isolation. +CREATE TABLE IF NOT EXISTS PRODUCT ( + id BIGINT AUTO_INCREMENT PRIMARY KEY, + sku VARCHAR(16) NOT NULL UNIQUE, + name VARCHAR(200) NOT NULL, + price_cents BIGINT NOT NULL +); diff --git a/spring-batch/src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java b/spring-batch/src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java new file mode 100644 index 0000000..8161dfe --- /dev/null +++ b/spring-batch/src/test/java/com/ankurm/batch/ProductValidatingProcessorTest.java @@ -0,0 +1,63 @@ +package com.ankurm.batch; + +import com.ankurm.batch.domain.Product; +import com.ankurm.batch.processing.ProductValidatingProcessor; +import org.junit.jupiter.api.Test; +import org.springframework.dao.DataIntegrityViolationException; +import org.springframework.dao.DuplicateKeyException; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 4 ({@code docs/04-item-processor-as-filter.md}): a null return from + * {@link org.springframework.batch.infrastructure.item.ItemProcessor#process} is a filter, not + * a skip. These assertions pin exactly which rows the pipeline's processor removes and which it + * lets through, independent of Spring Batch or the database. + */ +class ProductValidatingProcessorTest { + + private final ProductValidatingProcessor processor = new ProductValidatingProcessor(); + + @Test + void passesAWellFormedRow() { + try (Transcript t = new Transcript("01-processor-filter.txt", + "The processor as a filter: null means skip, not fail")) { + + Product ok = new Product("ABC-0001", "Widget 1", 1001); + Product result = processor.process(ok); + t.line("input : %s", ok); + t.line("result : %s", result); + assertThat(result).isEqualTo(ok); + + t.blank(); + Product badSku = new Product("abc-0012", "Widget 12", 1012); + t.line("input : %s <- lowercase sku, fails ^[A-Z]{3}-\\d{4}$", badSku); + t.line("result : %s", processor.process(badSku)); + assertThat(processor.process(badSku)).isNull(); + + t.blank(); + Product zeroPrice = new Product("ABC-0033", "Widget 33", 0); + t.line("input : %s <- priceCents is zero", zeroPrice); + t.line("result : %s", processor.process(zeroPrice)); + assertThat(processor.process(zeroPrice)).isNull(); + } + } + + @Test + void duplicateKeyExceptionIsADataIntegrityViolationException() { + try (Transcript t = new Transcript("02-exception-hierarchy.txt", + "Why skip(DataIntegrityViolationException.class) also catches the duplicate-key case")) { + + DuplicateKeyException dup = new DuplicateKeyException("Unique index violation"); + t.line("thrown type : %s", dup.getClass().getName()); + t.line("is a DataIntegrityViolationException? %b", dup instanceof DataIntegrityViolationException); + t.line(""); + t.line("H2's JdbcBatchUpdateException on a UNIQUE-constraint violation is translated by"); + t.line("Spring's SQLExceptionSubclassTranslator into DuplicateKeyException, which extends"); + t.line("DataIntegrityViolationException. A skip policy configured against the parent class"); + t.line("catches the subclass too -- see docs/07-restartability.md and docs/08-skip-vs-restart.md."); + + assertThat(dup).isInstanceOf(DataIntegrityViolationException.class); + } + } +} diff --git a/spring-batch/src/test/java/com/ankurm/batch/Transcript.java b/spring-batch/src/test/java/com/ankurm/batch/Transcript.java new file mode 100644 index 0000000..cad6a00 --- /dev/null +++ b/spring-batch/src/test/java/com/ankurm/batch/Transcript.java @@ -0,0 +1,52 @@ +package com.ankurm.batch; + +import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes a numbered transcript under {@code docs/output/} and echoes it to the console. + * Every console block quoted in the article comes out of one of these files verbatim. + */ +public final class Transcript implements AutoCloseable { + + private final Path path; + private final StringWriter buffer = new StringWriter(); + private final PrintWriter out = new PrintWriter(buffer); + + public Transcript(String fileName, String title) { + this.path = Path.of("docs", "output", fileName); + out.println("# " + title); + out.println(); + } + + public Transcript line(String format, Object... args) { + out.println(args.length == 0 ? format : String.format(format, args)); + return this; + } + + public Transcript blank() { + out.println(); + return this; + } + + public Transcript section(String heading) { + out.println(); + out.println("--- " + heading + " ---"); + return this; + } + + @Override + public void close() { + out.flush(); + try { + Files.createDirectories(path.getParent()); + Files.writeString(path, buffer.toString()); + } catch (IOException e) { + throw new IllegalStateException("could not write " + path, e); + } + System.out.print(buffer); + } +}