83 lines
5.4 KiB
Markdown
83 lines
5.4 KiB
Markdown
# 7. The failure that does not look like a failure: rejected partitions
|
|
|
|
[← Previous](06-why-cpu-bound-not-io-bound.md) | [README](../README.md) | [Next: Restart reruns only the failed partition →](08-restart-reruns-only-the-failed-partition.md)
|
|
|
|
`TaskExecutorPartitionHandler` submits every partition's worker step to its `TaskExecutor` up
|
|
front, then waits for all of them. What happens if the executor cannot accept all of those
|
|
submissions?
|
|
|
|
## Reproducing it
|
|
|
|
The `reject` Spring profile swaps in a deliberately undersized `ThreadPoolTaskExecutor`:
|
|
`corePoolSize`/`maxPoolSize` both 1, `queueCapacity` 0, and
|
|
`java.util.concurrent.ThreadPoolExecutor.AbortPolicy` as the rejection handler (Spring's own
|
|
default, `CallerRunsPolicy`, would instead silently run the "extra" partitions on the submitting
|
|
thread one at a time — serializing them rather than rejecting them, which hides this problem
|
|
instead of surfacing it). Four shard files, pool size one:
|
|
|
|
```console
|
|
$ java -jar target/spring-batch-partitioning-1.0.0.jar --spring.profiles.active=reject \
|
|
--partition.shards-dir=./data/shards-small --partition.grid-size=4 --partition.reject.pool-size=1
|
|
```
|
|
|
|
Only one partition ever logs "Executing step." The job fails with
|
|
`JobExecutionException: Partition handler returned an unsuccessful step` — a message that
|
|
never mentions rejection, threads, or pools. Full transcript:
|
|
[`docs/output/06-rejected-partitions-stuck.txt`](output/06-rejected-partitions-stuck.txt).
|
|
|
|
## What actually happened, found by querying the job repository directly
|
|
|
|
Nothing in the console log says `TaskRejectedException` anywhere.
|
|
`TaskExecutorPartitionHandler.doHandle` catches whatever the executor throws on submission and
|
|
folds it into the corresponding `StepExecution`'s failure exceptions rather than logging it, so
|
|
the only way to see it is to look at the repository's own tables:
|
|
|
|
```console
|
|
$ java -cp h2-2.4.240.jar org.h2.tools.Shell -url jdbc:h2:file:./data/rejecttest -user sa -password "" \
|
|
-sql "SELECT STEP_EXECUTION_ID, STEP_NAME, STATUS, EXIT_CODE FROM BATCH_STEP_EXECUTION ORDER BY STEP_EXECUTION_ID;"
|
|
|
|
STEP_EXECUTION_ID | STEP_NAME | STATUS | EXIT_CODE
|
|
1 | ordersManagerStep | FAILED | FAILED
|
|
2 | ordersWorkerStep:partition3 | STARTING | EXECUTING
|
|
3 | ordersWorkerStep:partition2 | STARTING | EXECUTING
|
|
4 | ordersWorkerStep:partition1 | COMPLETED | COMPLETED
|
|
5 | ordersWorkerStep:partition0 | STARTING | EXECUTING
|
|
```
|
|
|
|
The manager step and the `JobExecution` both reach a clean `FAILED`. The three rejected worker
|
|
`StepExecution`s do not. They are permanently parked at `STARTING`/`EXECUTING` — not failed,
|
|
not running, just stuck — because nothing ever calls back into the job repository to close
|
|
them out; the executor rejected them before Spring Batch's own bookkeeping around that
|
|
`StepExecution` ever started.
|
|
|
|
<figure>
|
|
<svg viewBox="0 0 740 220" role="img" aria-label="Four partitions submitted to a pool of size one with a zero-capacity queue and AbortPolicy: partition1 runs and completes, the other three are rejected on submission and their StepExecutions are left at STARTING forever.">
|
|
<style>.h{font:600 12px sans-serif;fill:#1a1a1a}.c{font:11px sans-serif;fill:#4b5563}.m{font:11px monospace;fill:#1a1a1a}</style>
|
|
<rect x="20" y="20" width="160" height="50" rx="4" fill="#e7f4ea" stroke="#4a9d63"/><text x="35" y="42" class="h">partition1</text><text x="35" y="58" class="c">runs, COMPLETED</text>
|
|
<rect x="200" y="20" width="160" height="50" rx="4" fill="#f7d9d3" stroke="#c56a54"/><text x="215" y="42" class="h">partition0</text><text x="215" y="58" class="c">rejected on submit</text>
|
|
<rect x="380" y="20" width="160" height="50" rx="4" fill="#f7d9d3" stroke="#c56a54"/><text x="395" y="42" class="h">partition2</text><text x="395" y="58" class="c">rejected on submit</text>
|
|
<rect x="560" y="20" width="160" height="50" rx="4" fill="#f7d9d3" stroke="#c56a54"/><text x="575" y="42" class="h">partition3</text><text x="575" y="58" class="c">rejected on submit</text>
|
|
<text x="20" y="105" class="c">StepExecution status for the three rejected partitions:</text>
|
|
<rect x="20" y="120" width="700" height="34" rx="4" fill="#fdeccf" stroke="#c9973f"/>
|
|
<text x="34" y="142" class="m">STARTING / EXECUTING -- forever. No callback ever closes them out.</text>
|
|
<text x="20" y="185" class="c">The manager step and JobExecution both reach FAILED cleanly. These three do not.</text>
|
|
</svg>
|
|
</figure>
|
|
|
|
## Why this is worse than an ordinary failure
|
|
|
|
An ordinary failed step is exactly what restart exists for. This is not that — see
|
|
[chapter 8](08-restart-reruns-only-the-failed-partition.md) for what happens when you try to
|
|
restart a job in this state, and [chapter 9](09-jobexecutionalreadyrunning-and-recover.md) for the
|
|
Spring Batch 6.0 feature that exists specifically to get out of it.
|
|
|
|
## Going deeper
|
|
|
|
- `ThreadPoolTaskExecutor`'s rejection handler options (`AbortPolicy`, `CallerRunsPolicy`,
|
|
`DiscardPolicy`, `DiscardOldestPolicy`) and their very different failure characters:
|
|
[`ThreadPoolExecutor` Javadoc](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/concurrent/ThreadPoolTaskExecutor.html) (`rel="nofollow"`).
|
|
- Sizing a pool correctly relative to gridSize in the first place:
|
|
[chapter 3](03-what-gridsize-actually-controls.md).
|
|
|
|
[Next: Restart reruns only the failed partition →](08-restart-reruns-only-the-failed-partition.md)
|