Ankur Mhatre
eaa8171966
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
2.9 KiB
prev: The lock · README · next: Clock skew
3. One scheduler thread
spring.task.scheduling.pool.size defaults to 1. Every @Scheduled method in the
application shares that one thread.
The expected consequence is that a slow job starves the others. What actually happens is more interesting, and considerably worse.
Three scheduled methods — one that sleeps 1800 ms on a 2000 ms schedule, one on a 200 ms schedule, one that throws — running for eight seconds. First with the default single thread (output), then with four (output):
| pool.size=1 | pool.size=4 | |
|---|---|---|
fast() executions |
40 | 41 |
longest gap between two fast() executions |
1995 ms | 201 ms |
fast() executions starting within 20 ms of the previous one |
35 | 0 |
| distinct scheduler threads | 1 | 4 |
The execution count is the same. A fixedRate schedule does not skip a tick it could not run:
the missed executions accumulate and are then fired back to back the moment the thread is free.
Thirty-five of the forty executions arrived in a burst.
So the metric everyone has — "the job ran 40 times, as expected" — is green, while the job's actual behaviour is two seconds of silence followed by thirty-five invocations in a few milliseconds. If that job calls a rate-limited API, or opens a database connection each time, the burst is the incident.
What to set
spring.task.scheduling.pool.size: at least the number of@Scheduledmethods that can overlap. It costs a handful of mostly idle threads.fixedDelayinstead offixedRatewhere "every N seconds" really means "N seconds after the last one finished".fixedDelayschedules the next run only after the current one completes, so it cannot accumulate a backlog to burst through.spring.threads.virtual.enabled=truereplaces the pool with aSimpleAsyncTaskSchedulerover virtual threads, which removes the shared-thread problem entirely. It also removes the bound:spring.task.scheduling.pool.sizeis one of the properties Boot's own metadata marks as having no effect when virtual threads are on.
An exception does not stop the schedule
throwing() threw on all 27 of its executions and kept its schedule. Spring wraps a scheduled
method in TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER, so the exception is logged by
o.s.s.s.TaskUtils$LoggingErrorHandler as Unexpected error occurred in scheduled task and
discarded. A raw ScheduledExecutorService would have cancelled the task after the first
failure — which is where the folklore comes from, and it does not apply to @Scheduled.
The flip side is that a job which has been failing since the last deploy produces nothing but a
recurring ERROR line. Supply your own SchedulingConfigurer with an error handler if you want
a metric.
next: Clock skew