Add spring-batch: jobs, steps, chunk processing and restartability on Boot 4.1
Companion code for "Spring Batch on Boot 4.1: Jobs, Steps, Chunk Processing and Restartability". A productImportJob configured three ways by profile against a poisoned CSV row, run as real java -jar processes (not just JUnit) so the restart story is genuine: a chunk fails and rolls back, the process exits, a brand-new JVM against the same file-based H2 database resumes at the exact next unread row (READ_COUNT 20, not 60) and completes. Findings the build pins: - StepBuilder.chunk(int, PlatformTransactionManager) still compiles in Batch 6.0.5 but returns the legacy SimpleStepBuilder; chunk(int) returns the new ChunkOrientedStepBuilder, and only the latter is used here. - Two different ExecutionContext classes now exist in two different packages (infrastructure.item vs core.repository.persistence) with different shapes. - spring-boot-starter-batch alone gives a resourceless JobRepository that forgets every JobInstance the moment the JVM exits; spring-boot-starter- batch-jdbc is what makes the restart demo possible at all, demonstrated by excluding BatchJdbcAutoConfiguration and watching a "restart" collide with the previous run's own data instead of resuming it. - A migration-guide summary claiming CommandLineJobRunner was removed in 6.0 is wrong -- javap against the real jar shows @Deprecated(forRemoval=true), not removed. - RepeatStatus moved from core.repeat to infrastructure.repeat, caught by the compiler rather than by reading docs. 11 documentation chapters, 10 captured transcripts (unit tests, javap output, and real two-JVM scenario runs), all regenerated by scripts/run-all.sh. Fixed after push: three dead docs.spring.io links in the doc chapters (readersAndWriters/* and chunk-oriented-processing/*.html paths moved when Spring Batch 6 reorganized its reference docs; corrected to the current readers-and-writers/*, processor.html and chunk-oriented-processing.html paths, verified 200 via curl before committing). Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019DXsJ1zpikbA1MQJN6RqFA
This commit is contained in:
Claude
Executable
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Captures real javap output pinning the Spring Batch 6.0 API facts this article relies on:
|
||||
# the ChunkOrientedStepBuilder that chunk(int) now returns, the two different ExecutionContext
|
||||
# classes, the package move of RepeatStatus, and CommandLineJobRunner's deprecation (NOT removal
|
||||
# -- an earlier draft trusted a migration-guide summary that said "removed" and this transcript
|
||||
# is what caught the error; see docs/09-corrections.md).
|
||||
#
|
||||
# ./scripts/capture-javap.sh
|
||||
#
|
||||
# Needs the spring-batch-core and spring-batch-infrastructure 6.0.5 jars in the local Maven repo
|
||||
# (a normal `mvn -B -o package` in this module already pulls them in).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
CORE=$(find "$HOME/.m2" -name 'spring-batch-core-6.0.5.jar' | head -1)
|
||||
INFRA=$(find "$HOME/.m2" -name 'spring-batch-infrastructure-6.0.5.jar' | head -1)
|
||||
: "${CORE:?spring-batch-core-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}"
|
||||
: "${INFRA:?spring-batch-infrastructure-6.0.5.jar not found in ~/.m2 -- run mvn -B -o package first}"
|
||||
|
||||
WORK=$(mktemp -d)
|
||||
mkdir -p "$WORK/core" "$WORK/infra"
|
||||
unzip -o -q "$CORE" -d "$WORK/core"
|
||||
unzip -o -q "$INFRA" -d "$WORK/infra"
|
||||
|
||||
mkdir -p docs/output
|
||||
|
||||
{
|
||||
echo "# javap org.springframework.batch.core.step.builder.StepBuilder (spring-batch-core 6.0.5)"
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.step.builder.StepBuilder'
|
||||
javap -cp "$WORK/core" org.springframework.batch.core.step.builder.StepBuilder
|
||||
} > docs/output/03-stepbuilder-chunk-overloads.txt
|
||||
|
||||
{
|
||||
echo "# javap: two ExecutionContext classes in two different packages"
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-infrastructure-6.0.5.jar org.springframework.batch.infrastructure.item.ExecutionContext'
|
||||
javap -cp "$WORK/infra" org.springframework.batch.infrastructure.item.ExecutionContext
|
||||
echo
|
||||
echo '$ javap -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.repository.persistence.ExecutionContext'
|
||||
javap -cp "$WORK/core" org.springframework.batch.core.repository.persistence.ExecutionContext
|
||||
} > docs/output/04-two-executioncontext-classes.txt
|
||||
|
||||
{
|
||||
echo "# javap: CommandLineJobRunner is deprecated (forRemoval), not removed, in 6.0.5"
|
||||
echo
|
||||
echo '$ javap -verbose -cp spring-batch-core-6.0.5.jar org.springframework.batch.core.launch.support.CommandLineJobRunner | grep -A3 "^public class\|Deprecated"'
|
||||
javap -verbose -cp "$WORK/core" org.springframework.batch.core.launch.support.CommandLineJobRunner \
|
||||
| grep -A3 '^public class\|Deprecated' | head -20
|
||||
} > docs/output/09-commandlinejobrunner-deprecated-not-removed.txt
|
||||
|
||||
rm -rf "$WORK"
|
||||
echo "wrote docs/output/03-stepbuilder-chunk-overloads.txt"
|
||||
echo "wrote docs/output/04-two-executioncontext-classes.txt"
|
||||
echo "wrote docs/output/09-commandlinejobrunner-deprecated-not-removed.txt"
|
||||
Executable
+152
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs the three end-to-end scenarios this article is built on and writes what actually
|
||||
# happened -- console output plus the real BATCH_STEP_EXECUTION and PRODUCT rows -- to
|
||||
# docs/output/. Nothing here is retyped; the numbers in the article come out of these files.
|
||||
#
|
||||
# ./scripts/capture-scenarios.sh
|
||||
#
|
||||
# Needs target/spring-batch-1.0.0.jar (run `mvn -B -o -DskipTests package` first) and a JDK 25.
|
||||
# Each scenario gets its own file-based H2 database under scenario-data/<name>/ so the three
|
||||
# runs never interfere with each other.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
JAR=target/spring-batch-1.0.0.jar
|
||||
: "${JAR:?run mvn -B -o -DskipTests package first}"
|
||||
H2_JAR=$(find "$HOME/.m2" -name 'h2-2.4.240.jar' | head -1)
|
||||
mkdir -p docs/output
|
||||
rm -rf scenario-data
|
||||
mkdir -p scenario-data
|
||||
|
||||
query() {
|
||||
# $1 = db dir, $2 = sql
|
||||
java -cp "$H2_JAR" org.h2.tools.Shell -url "jdbc:h2:file:${PWD}/$1/db" -user sa -password "" -sql "$2" 2>/dev/null \
|
||||
| grep -v '^$'
|
||||
}
|
||||
|
||||
run_scenario() {
|
||||
# $1 = name, $2 = profile, $3 = source csv
|
||||
local name="$1" profile="$2" src="$3" dir
|
||||
dir="scenario-data/$1"
|
||||
mkdir -p "$dir"
|
||||
cp "$src" "$dir/input.csv"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active="$profile" \
|
||||
--import.file="file:${dir}/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/${dir}/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|SKIPPED on write|DuplicateKeyException:|WARN.*ProductValidatingProcessor|Executing step|executed in" || true
|
||||
}
|
||||
|
||||
echo "== scenario: clean (05-happy-path.txt)"
|
||||
{
|
||||
echo "# The happy path: 60 rows in, 58 products out"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --import.file=file:scenario-data/clean-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/clean-demo/db'
|
||||
echo
|
||||
run_scenario clean-demo clean src/main/resources/data/products.csv
|
||||
echo
|
||||
echo '$ SELECT read_count, filter_count, write_count, commit_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/clean-demo "SELECT READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
} > docs/output/05-happy-path.txt
|
||||
|
||||
echo "== scenario: broken, run 1 (07-restart-run1-fails.txt)"
|
||||
{
|
||||
echo "# Run 1: the poisoned duplicate at row 47 fails the whole chunk"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db'
|
||||
echo
|
||||
run_scenario restart-demo broken src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/restart-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/07-restart-run1-fails.txt
|
||||
|
||||
echo "== fixing the duplicate in place (same file, same line count) =="
|
||||
sed -i '48s/^ABC-0005,Widget 47,1047$/ABC-0999,Widget 47,1047/' scenario-data/restart-demo/input.csv
|
||||
grep -n "Widget 47" scenario-data/restart-demo/input.csv
|
||||
|
||||
echo "== scenario: broken, run 2 = restart in a fresh JVM (07-restart-run2-resumes.txt)"
|
||||
{
|
||||
echo "# Run 2: a brand-new JVM, the SAME job parameters, the corrected file"
|
||||
echo
|
||||
echo "sed -i '48s/ABC-0005/ABC-0999/' scenario-data/restart-demo/input.csv # the only change"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=broken \'
|
||||
echo ' --import.file=file:scenario-data/restart-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/restart-demo/db # same db file'
|
||||
echo
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=broken \
|
||||
--import.file="file:scenario-data/restart-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/restart-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|Executing step|executed in" || true
|
||||
echo
|
||||
echo '$ SELECT step_execution_id, job_execution_id, status, read_count, write_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION ORDER BY step_execution_id;'
|
||||
query scenario-data/restart-demo "SELECT STEP_EXECUTION_ID, JOB_EXECUTION_ID, STATUS, READ_COUNT, WRITE_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep' ORDER BY STEP_EXECUTION_ID;"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/restart-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/08-restart-run2-resumes.txt
|
||||
|
||||
echo "== scenario: skip (10-skip-instead-of-fail.txt)"
|
||||
{
|
||||
echo "# faultTolerant().skip(DataIntegrityViolationException.class): one bad row, job still COMPLETES"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=skip \'
|
||||
echo ' --import.file=file:scenario-data/skip-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/skip-demo/db'
|
||||
echo
|
||||
run_scenario skip-demo skip src/main/resources/data/products-poison.csv
|
||||
echo
|
||||
echo '$ SELECT status, read_count, filter_count, write_count, write_skip_count, commit_count, rollback_count FROM BATCH_STEP_EXECUTION;'
|
||||
query scenario-data/skip-demo "SELECT STATUS, READ_COUNT, FILTER_COUNT, WRITE_COUNT, WRITE_SKIP_COUNT, COMMIT_COUNT, ROLLBACK_COUNT FROM BATCH_STEP_EXECUTION WHERE STEP_NAME='importStep';"
|
||||
echo
|
||||
echo '$ SELECT COUNT(*) FROM PRODUCT;'
|
||||
query scenario-data/skip-demo "SELECT COUNT(*) FROM PRODUCT;"
|
||||
} > docs/output/10-skip-instead-of-fail.txt
|
||||
|
||||
echo "== scenario: resourceless (11-resourceless-forgets-everything.txt)"
|
||||
rm -rf scenario-data/resourceless-demo
|
||||
mkdir -p scenario-data/resourceless-demo
|
||||
cp src/main/resources/data/products.csv scenario-data/resourceless-demo/input.csv
|
||||
{
|
||||
echo "# --spring.autoconfigure.exclude=...BatchJdbcAutoConfiguration: same job, same params, no memory"
|
||||
echo
|
||||
echo '$ java -jar target/spring-batch-1.0.0.jar --spring.profiles.active=clean \'
|
||||
echo ' --spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \'
|
||||
echo ' --import.file=file:scenario-data/resourceless-demo/input.csv \'
|
||||
echo ' --spring.datasource.url=jdbc:h2:file:.../scenario-data/resourceless-demo/db'
|
||||
echo
|
||||
echo "-- run 1 --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED" || true
|
||||
echo
|
||||
echo "-- run 2: a second, completely fresh JVM, same jar, same job parameters, same PRODUCT table --"
|
||||
java -jar "$JAR" \
|
||||
--spring.profiles.active=clean \
|
||||
--spring.autoconfigure.exclude=org.springframework.boot.batch.jdbc.autoconfigure.BatchJdbcAutoConfiguration \
|
||||
--import.file="file:scenario-data/resourceless-demo/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/scenario-data/resourceless-demo/db" \
|
||||
2>&1 | grep -E "REPORT:|JOB FINISHED|DuplicateKeyException:" | head -3 || true
|
||||
echo
|
||||
echo "No JobInstanceAlreadyCompleteException on run 2 -- the resourceless JobRepository has no idea"
|
||||
echo "run 1 ever happened, so it tries the whole job again and collides with what run 1 already"
|
||||
echo "wrote to the PRODUCT table. With spring-boot-starter-batch-jdbc (the default configuration"
|
||||
echo "used everywhere else in this module) run 2 throws JobInstanceAlreadyCompleteException instead,"
|
||||
echo "which ImportRunner treats as \"nothing to do\" -- see docs/07-restartability.md."
|
||||
} > docs/output/11-resourceless-forgets-everything.txt
|
||||
|
||||
echo
|
||||
echo "docs/output:"
|
||||
ls -1 docs/output
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates every file under docs/output/.
|
||||
#
|
||||
# ./scripts/run-all.sh
|
||||
#
|
||||
# Needs a JDK 25 and Maven 3.9, with dependencies already resolved once online (mvn needs a
|
||||
# networked run before -o works for spring-boot:run / surefire -- see companion-repo notes).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo "== unit tests (transcripts 01-02)"
|
||||
mvn -B -o test
|
||||
|
||||
echo "== javap transcripts (03, 04, 09)"
|
||||
./scripts/capture-javap.sh
|
||||
|
||||
echo "== package"
|
||||
mvn -B -o -DskipTests package
|
||||
|
||||
echo "== end-to-end scenarios (05, 07, 08, 10, 11)"
|
||||
./scripts/capture-scenarios.sh
|
||||
|
||||
echo
|
||||
echo "docs/output:"
|
||||
ls -1 docs/output
|
||||
Executable
+28
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs one scenario against its own external H2 file database and CSV drop file (both under
|
||||
# scenario-data/<name>/, outside target/ so a rebuild never touches them -- see
|
||||
# docs/07-restartability.md for why that matters: `mvn spring-boot:run` re-copies
|
||||
# src/main/resources over target/classes before every run, which would silently undo an
|
||||
# in-place "fix" to a classpath resource between two runs of the same scenario.
|
||||
#
|
||||
# ./scripts/run-scenario.sh <name> <profile> <source-csv>
|
||||
#
|
||||
# Reuses the packaged jar (target/spring-batch-1.0.0.jar); run `mvn -B -o -DskipTests package`
|
||||
# first if it is missing or stale.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
NAME="${1:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
PROFILE="${2:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
SRC_CSV="${3:?usage: run-scenario.sh <name> <profile> <source-csv>}"
|
||||
|
||||
DIR="scenario-data/${NAME}"
|
||||
mkdir -p "$DIR"
|
||||
if [ ! -f "$DIR/input.csv" ]; then
|
||||
cp "$SRC_CSV" "$DIR/input.csv"
|
||||
fi
|
||||
|
||||
java -jar target/spring-batch-1.0.0.jar \
|
||||
--spring.profiles.active="$PROFILE" \
|
||||
--import.file="file:${DIR}/input.csv" \
|
||||
--spring.datasource.url="jdbc:h2:file:${PWD}/${DIR}/db"
|
||||
Reference in New Issue
Block a user