Add spring-batch: jobs, steps, chunk processing and restartability on Boot 4.1
Companion code for "Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability". A productImportJob configured three ways by profile against a poisoned CSV row, run as real java -jar processes (not just JUnit) so the restart story is genuine: a chunk fails and rolls back, the process exits, a brand-new JVM against the same file-based H2 database resumes at the exact next unread row (READ_COUNT 20, not 60) and completes. Findings the build pins: - StepBuilder.chunk(int, PlatformTransactionManager) still compiles in Batch 6.0.5 but returns the legacy SimpleStepBuilder; chunk(int) returns the new ChunkOrientedStepBuilder, and only the latter is used here. - Two different ExecutionContext classes now exist in two different packages (infrastructure.item vs core.repository.persistence) with different shapes. - spring-boot-starter-batch alone gives a resourceless JobRepository that forgets every JobInstance the moment the JVM exits; spring-boot-starter- batch-jdbc is what makes the restart demo possible at all, demonstrated by excluding BatchJdbcAutoConfiguration and watching a "restart" collide with the previous run's own data instead of resuming it. - A migration-guide summary claiming CommandLineJobRunner was removed in 6.0 is wrong -- javap against the real jar shows @Deprecated(forRemoval=true), not removed. - RepeatStatus moved from core.repeat to infrastructure.repeat, caught by the compiler rather than by reading docs. 11 documentation chapters, 10 captured transcripts (unit tests, javap output, and real two-JVM scenario runs), all regenerated by scripts/run-all.sh. Fixed after push: three dead docs.spring.io links in the doc chapters (readersAndWriters/* and chunk-oriented-processing/*.html paths moved when Spring Batch 6 reorganized its reference docs; corrected to the current readers-and-writers/*, processor.html and chunk-oriented-processing.html paths, verified 200 via curl before committing). Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019DXsJ1zpikbA1MQJN6RqFA
This commit is contained in:
Claude
Executable
+152
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs the three end-to-end scenarios this article is built on and writes what actually
|
||||
# happened -- console output plus the real BATCH_STEP_EXECUTION and PRODUCT rows -- to
|
||||
# docs/output/. Nothing here is retyped; the numbers in the article come out of these files.
|
||||
#
|
||||
# ./scripts/capture-scenarios.sh
|
||||
#
|
||||
# Needs target/spring-batch-1.0.0.jar (run `mvn -B -o -DskipTests package` first) and a JDK 25.
|
||||
# Each scenario gets its own file-based H2 database under scenario-data/<name>/ so the three
|
||||
# runs never interfere with each other.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
JAR=target/spring-batch-1.0.0.jar
|
||||
: "${JAR:?run mvn -B -o -DskipTests package first}"
|
||||
H2_JAR=$(find "$HOME/.m2" -name 'h2-2.4.240.jar' | head -1)
|
||||
mkdir -p docs/output
|
||||
rm -rf scenario-data
|
||||
mkdir -p scenario-data
|
||||
|
||||
query() {
|
||||
# $1 = db dir, $2 = sql
|
||||
java -cp "$H2_JAR" org.h2.tools.Shell -url "jdbc:h2:file:${PWD}/$1/db" -user sa -password "" -sql "$2" 2>/dev/null \
|
||||
| grep -v '^$'
|
||||
}
|
||||
|
||||
run_scenario() {
|
||||
# $1 = name, $2 = profile, $3 = source csv
|
||||
local name="$1" profile="$2" src="$3" dir
|
||||
dir="scenario-data/$1"
|
||||
mkdir -p "$dir"
|
||||
cp "$src" "$dir/input.csv"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active="$profile" \
|
||||
--import.file="file:${dir}/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/${dir}/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|SKIPPED on write|DuplicateKeyException:|WARN.*ProductValidatingProcessor|Executing step|executed in" || true
|
||||
}
|
||||
|
||||
echo "== scenario: clean (05-happy-path.txt)"
|
||||
{
|
||||
echo "# The happy path: 60 rows in, 58 products out"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --import.file=file:scenario-data/clean-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db'
|
||||
echo
|
||||
run_scenario clean-demo clean src/main/resources/data/products.csv
|
||||
echo
|
||||
echo '$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/clean-demo "SELECT READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
} > docs/output/05-happy-path.txt
|
||||
|
||||
echo "== scenario: broken, run 1 (07-restart-run1-fails.txt)"
|
||||
{
|
||||
echo "# Run 1: the poisoned duplicate at row 47 fails the whole chunk"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db'
|
||||
echo
|
||||
run_scenario restart-demo broken src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/restart-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/07-restart-run1-fails.txt
|
||||
|
||||
echo "== fixing the duplicate in place (same file, same line count) =="
|
||||
sed -i '48s/^ABC-0005,Widget 47,1047$/ABC-0999,Widget 47,1047/' scenario-data/restart-demo/input.csv
|
||||
grep -n "Widget 47" scenario-data/restart-demo/input.csv
|
||||
|
||||
echo "== scenario: broken, run 2 = restart in a fresh JVM (07-restart-run2-resumes.txt)"
|
||||
{
|
||||
echo "# Run 2: a brand-new JVM, the SAME job parameters, the corrected file"
|
||||
echo
|
||||
echo "sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file'
|
||||
echo
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=broken \
|
||||
--import.file="file:scenario-data/restart-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/restart-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|Executing step|executed in" || true
|
||||
echo
|
||||
echo '$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;'
|
||||
query scenario-data/restart-demo "SELECT STEP_EXECUTION_ID, JOB_EXECUTION_ID, STATUS, READ_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep' ORDER BY STEP_EXECUTION_ID;"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/08-restart-run2-resumes.txt
|
||||
|
||||
echo "== scenario: skip (10-skip-instead-of-fail.txt)"
|
||||
{
|
||||
echo "# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \'
|
||||
echo ' --import.file=file:scenario-data/skip-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db'
|
||||
echo
|
||||
run_scenario skip-demo skip src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/skip-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, WRITE_SKIP_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/skip-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/10-skip-instead-of-fail.txt
|
||||
|
||||
echo "== scenario: resourceless (11-resourceless-forgets-everything.txt)"
|
||||
rm -rf scenario-data/resourceless-demo
|
||||
mkdir -p scenario-data/resourceless-demo
|
||||
cp src/main/resources/data/products.csv scenario-data/resourceless-demo/input.csv
|
||||
{
|
||||
echo "# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \'
|
||||
echo ' --import.file=file:scenario-data/resourceless-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db'
|
||||
echo
|
||||
echo "-- run 1 --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED" || true
|
||||
echo
|
||||
echo "-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|DuplicateKeyException:" | head -3 || true
|
||||
echo
|
||||
echo "No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea"
|
||||
echo "run 1 ever happened, so it tries the whole job again and collides with what run 1 already"
|
||||
echo "wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration"
|
||||
echo "used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead,"
|
||||
echo "which ImportRunner treats as \"nothing to do\" -- see docs/07-restartability.md."
|
||||
} > docs/output/11-resourceless-forgets-everything.txt
|
||||
|
||||
echo
|
||||
echo "docs/output:"
|
||||
ls -1 docs/output
|
||||
Reference in New Issue
Block a user