Ankur Mhatre
eaa8171966
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
57 lines
2.9 KiB
Markdown
57 lines
2.9 KiB
Markdown
prev: [The lock](02-the-lock.md) · [README](../README.md) · next: [Clock skew](04-clock-skew.md)
|
|
|
|
# 3. One scheduler thread
|
|
|
|
`spring.task.scheduling.pool.size` defaults to **1**. Every `@Scheduled` method in the
|
|
application shares that one thread.
|
|
|
|
The expected consequence is that a slow job starves the others. What actually happens is more
|
|
interesting, and considerably worse.
|
|
|
|
Three scheduled methods — one that sleeps 1800 ms on a 2000 ms schedule, one on a 200 ms
|
|
schedule, one that throws — running for eight seconds. First with the default single thread
|
|
([output](output/scheduler-pool-1.txt)), then with four
|
|
([output](output/scheduler-pool-4.txt)):
|
|
|
|
| | pool.size=1 | pool.size=4 |
|
|
|---|---|---|
|
|
| `fast()` executions | 40 | 41 |
|
|
| longest gap between two `fast()` executions | **1995 ms** | 201 ms |
|
|
| `fast()` executions starting within 20 ms of the previous one | **35** | 0 |
|
|
| distinct scheduler threads | 1 | 4 |
|
|
|
|
The execution *count* is the same. A `fixedRate` schedule does not skip a tick it could not run:
|
|
the missed executions accumulate and are then fired back to back the moment the thread is free.
|
|
Thirty-five of the forty executions arrived in a burst.
|
|
|
|
So the metric everyone has — "the job ran 40 times, as expected" — is green, while the job's
|
|
actual behaviour is two seconds of silence followed by thirty-five invocations in a few
|
|
milliseconds. If that job calls a rate-limited API, or opens a database connection each time, the
|
|
burst is the incident.
|
|
|
|
## What to set
|
|
|
|
- **`spring.task.scheduling.pool.size`**: at least the number of `@Scheduled` methods that can
|
|
overlap. It costs a handful of mostly idle threads.
|
|
- **`fixedDelay` instead of `fixedRate`** where "every N seconds" really means "N seconds after
|
|
the last one finished". `fixedDelay` schedules the next run only after the current one
|
|
completes, so it cannot accumulate a backlog to burst through.
|
|
- **`spring.threads.virtual.enabled=true`** replaces the pool with a `SimpleAsyncTaskScheduler`
|
|
over virtual threads, which removes the shared-thread problem entirely. It also removes the
|
|
bound: `spring.task.scheduling.pool.size` is one of the properties Boot's own metadata marks
|
|
as having no effect when virtual threads are on.
|
|
|
|
## An exception does not stop the schedule
|
|
|
|
`throwing()` threw on all 27 of its executions and kept its schedule. Spring wraps a scheduled
|
|
method in `TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER`, so the exception is logged by
|
|
`o.s.s.s.TaskUtils$LoggingErrorHandler` as `Unexpected error occurred in scheduled task` and
|
|
discarded. A raw `ScheduledExecutorService` would have cancelled the task after the first
|
|
failure — which is where the folklore comes from, and it does not apply to `@Scheduled`.
|
|
|
|
The flip side is that a job which has been failing since the last deploy produces nothing but a
|
|
recurring `ERROR` line. Supply your own `SchedulingConfigurer` with an error handler if you want
|
|
a metric.
|
|
|
|
next: [Clock skew](04-clock-skew.md)
|