asmhatre/spring-async-demo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the scheduling module

Browse Source 
Three replicas of one application against one PostgreSQL database, proving
duplicate @Scheduled execution and then removing it with ShedLock: 24
executions where 8 were due, then 7 for 7 ticks.

Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and
warns about nothing; spring.task.scheduling.pool.size=1 does not starve a
fixedRate job but delays it and fires 35 of 40 executions in a burst; and a
node whose clock is 40 seconds fast takes a live lock unless the provider
uses usingDbTime().
This commit is contained in:
Ankur Mhatre
2026-09-01 23:28:06 +05:30
parent 243cccd4ca
commit 6af6e4a3f1
31 changed files with 1373 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
scheduling/docs/01-three-replicas-three-executions.md Normal file
View File

@@ -0,0 +1,75 @@
[README](../README.md) · next: [The lock](02-the-lock.md)
# 1. Three replicas, three executions
`@Scheduled` is a per-JVM timer. It has no idea that other JVMs exist. Scale a deployment to
three replicas and every `@Scheduled` method in the application runs three times per tick, on
three machines, at almost the same instant.
That is the whole problem, and it is worth seeing rather than reading. `scripts/three-replicas.sh`
starts three copies of this application against one PostgreSQL database, lets the 3-second
schedule tick for 30 seconds, and counts.
From [`docs/output/three-replicas-unlocked.txt`](output/three-replicas-unlocked.txt):
```
-- executions per replica --
instance_id | count
-------------+-------
replica-1 | 8
replica-2 | 8
replica-3 | 8
-- pairs of executions that overlapped --
overlapping_pairs
-------------------
8
first | first_at | second | second_at
-----------+--------------+-----------+--------------
replica-1 | 22:48:54.564 | replica-3 | 22:48:54.752
replica-1 | 22:48:57.551 | replica-3 | 22:48:57.736
replica-1 | 22:49:00.551 | replica-3 | 22:49:00.736
```
Twenty-four executions where eight were due, and eight of them started while another replica was
still inside the method — 188 milliseconds apart in the first pair, against a job that only takes
200 milliseconds. Anything in that method which
is not idempotent is now a support ticket: a second invoice email, a double refund, two rows
where the unique constraint you did not add would have saved you.
## The part that makes it hard to spot
`ReportJob` is annotated `@SchedulerLock` in this run. Look at the end of the same transcript:
```
-- the lock row --
name | locked_by | locked_at | lock_until
------+-----------+-----------+------------
(0 rows)
```
The annotation is present, the lock table exists, the application started cleanly, and nothing
was ever locked. `@SchedulerLock` on its own is inert: it needs `@EnableSchedulerLock` to install
the interceptor and a `LockProvider` bean to have somewhere to record the lock. In this module
both live on `LockConfiguration`, which is `@Profile("locked")`.
There is no warning for the missing half. That is the single most important sentence in this
module — an application with the annotation and without the plumbing behaves exactly like an
application with no locking at all, and it looks locked in code review.
## Reproducing it
```bash
scripts/postgres.sh start # a throwaway PostgreSQL 14, no Docker, no root
scripts/three-replicas.sh unlocked
scripts/three-replicas.sh locked
```
or, with Docker:
```bash
docker compose up --build # PROFILE=locked docker compose up --build
```
next: [The lock](02-the-lock.md)

90
scheduling/docs/02-the-lock.md Normal file
View File

@@ -0,0 +1,90 @@
prev: [Three replicas, three executions](01-three-replicas-three-executions.md) · [README](../README.md) · next: [One scheduler thread](03-one-scheduler-thread.md)
# 2. The lock
Activate `LockConfiguration` and the same 30-second run produces this
([`docs/output/three-replicas-locked.txt`](output/three-replicas-locked.txt)):
```
-- executions per replica --
instance_id | count
-------------+-------
replica-2 | 1
replica-3 | 6
-- pairs of executions that overlapped --
overlapping_pairs
-------------------
0
-- the lock row --
name | locked_by | locked_at | lock_until
---------------+-----------+--------------+--------------
nightlyReport | unknown | 17:19:46.921 | 17:19:47.921
```
Seven executions for seven ticks, no overlap. The mechanism is one row in one table and a
conditional `UPDATE`: acquire means "set `lock_until` to a future time **where** `lock_until` is
already in the past", and only one replica's update can win because the row is locked for the
duration of that statement.
Three things in that transcript deserve attention.
## ShedLock does not distribute work
Six of the seven executions are `replica-3`. The lock is not a queue and there is no round-robin:
whichever replica's timer fires first each tick takes the lock, and on a stable cluster that is
overwhelmingly the same replica — here `replica-2` won the first tick and then never won
another. This is correct behaviour for "run exactly once" and
completely wrong if what you actually wanted was "spread the work across the cluster". For that
you want a work queue, not a lock.
## `locked_by` is not identity
It reads `unknown` above. ShedLock fills that column with the host name, and the host name of the
container that produced this transcript does not resolve, so it falls back. It is a diagnostic
field, never read back for correctness — but it does mean that on Kubernetes you often get a
column full of pod hashes or of `unknown`, and it is worth setting something meaningful if you
plan to use it during an incident.
## The timestamps are in a different time zone from your own
`locked_at` reads `17:19:46` while the execution recorded at the same moment in the same file
reads `22:49:46`. That is not a
bug; it is `usingDbTime()` doing its job. With it, the timestamps are written by PostgreSQL in
UTC, whereas `started_at` is written by the application in the JVM's zone. Anyone eyeballing the
`shedlock` table next to the application's own tables will see a five-and-a-half-hour discrepancy
and think the lock is stale. It is not.
## The two durations
```java
@SchedulerLock(name = "nightlyReport", lockAtMostFor = "PT20S", lockAtLeastFor = "PT1S")
```
**`lockAtMostFor`** is the answer to "the holder was `kill -9`'d; how long before somebody else
may run this?" It must be longer than the longest the job could possibly take, because when it
expires the lock is available whether or not the job finished. Set it to five minutes for a job
that normally takes ten seconds, and accept a five-minute gap after a crash — that is the trade,
and there is no setting that avoids it.
**`lockAtLeastFor`** keeps the lock held for a minimum period after a fast job finishes. It exists
for clock skew and for schedules where two replicas' timers fire within milliseconds of each
other: without it, a job that completes in 20 ms releases the lock in time for the next replica's
tick to pick it up, and you are back to two executions.
`@EnableSchedulerLock(defaultLockAtMostFor = ...)` has no default value in the annotation, so it
must be supplied. That is deliberate: there is no safe guess.
## `interceptMode`
ShedLock 7 defaults to `PROXY_METHOD`. The `PROXY_SCHEDULER` mode that most tutorials still show
is deprecated, and its own Javadoc says it "requires a reflection hack to work well with Spring
6.2". Leave the default alone.
Because `PROXY_METHOD` is a Spring AOP proxy, everything from
[the `@Async` article](https://ankurm.com/spring-boot-4-async-executors-virtual-threads/) applies
here too: a `@SchedulerLock` method called from inside its own class is not locked, and a `final`
method is not locked.
next: [One scheduler thread](03-one-scheduler-thread.md)

56
scheduling/docs/03-one-scheduler-thread.md Normal file
View File

@@ -0,0 +1,56 @@
prev: [The lock](02-the-lock.md) · [README](../README.md) · next: [Clock skew](04-clock-skew.md)
# 3. One scheduler thread
`spring.task.scheduling.pool.size` defaults to **1**. Every `@Scheduled` method in the
application shares that one thread.
The expected consequence is that a slow job starves the others. What actually happens is more
interesting, and considerably worse.
Three scheduled methods — one that sleeps 1800 ms on a 2000 ms schedule, one on a 200 ms
schedule, one that throws — running for eight seconds. First with the default single thread
([output](output/scheduler-pool-1.txt)), then with four
([output](output/scheduler-pool-4.txt)):
| | pool.size=1 | pool.size=4 |
|---|---|---|
| `fast()` executions | 40 | 41 |
| longest gap between two `fast()` executions | **1995 ms** | 201 ms |
| `fast()` executions starting within 20 ms of the previous one | **35** | 0 |
| distinct scheduler threads | 1 | 4 |
The execution *count* is the same. A `fixedRate` schedule does not skip a tick it could not run:
the missed executions accumulate and are then fired back to back the moment the thread is free.
Thirty-five of the forty executions arrived in a burst.
So the metric everyone has — "the job ran 40 times, as expected" — is green, while the job's
actual behaviour is two seconds of silence followed by thirty-five invocations in a few
milliseconds. If that job calls a rate-limited API, or opens a database connection each time, the
burst is the incident.
## What to set
- **`spring.task.scheduling.pool.size`**: at least the number of `@Scheduled` methods that can
overlap. It costs a handful of mostly idle threads.
- **`fixedDelay` instead of `fixedRate`** where "every N seconds" really means "N seconds after
the last one finished". `fixedDelay` schedules the next run only after the current one
completes, so it cannot accumulate a backlog to burst through.
- **`spring.threads.virtual.enabled=true`** replaces the pool with a `SimpleAsyncTaskScheduler`
over virtual threads, which removes the shared-thread problem entirely. It also removes the
bound: `spring.task.scheduling.pool.size` is one of the properties Boot's own metadata marks
as having no effect when virtual threads are on.
## An exception does not stop the schedule
`throwing()` threw on all 27 of its executions and kept its schedule. Spring wraps a scheduled
method in `TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER`, so the exception is logged by
`o.s.s.s.TaskUtils$LoggingErrorHandler` as `Unexpected error occurred in scheduled task` and
discarded. A raw `ScheduledExecutorService` would have cancelled the task after the first
failure — which is where the folklore comes from, and it does not apply to `@Scheduled`.
The flip side is that a job which has been failing since the last deploy produces nothing but a
recurring `ERROR` line. Supply your own `SchedulingConfigurer` with an error handler if you want
a metric.
next: [Clock skew](04-clock-skew.md)

56
scheduling/docs/04-clock-skew.md Normal file
View File

@@ -0,0 +1,56 @@
prev: [One scheduler thread](03-one-scheduler-thread.md) · [README](../README.md) · next: [When not to use a lock](05-when-not-to-use-a-lock.md)
# 4. Clock skew, and what `usingDbTime()` is for
```java
JdbcTemplateLockProvider.Configuration.builder()
.withJdbcTemplate(new JdbcTemplate(dataSource))
.usingDbTime()
.build();
```
Without `usingDbTime()`, each replica writes `lock_until` from its own clock and compares
`lock_until` against its own clock. The lock is then only as good as the agreement between three
machines about what time it is.
Three replicas racing is not a proof, because a race can go either way. `ClockSkewTest` drives
the `LockProvider` directly instead, so the two acquisitions are ordered by the test and the only
variable is what the second caller believes the time to be
([`docs/output/clock-skew.txt`](output/clock-skew.txt)):
```
lockAtMostFor = 30s, both callers ask for the same lock name.
JdbcTemplateLockProvider WITHOUT usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : ACQUIRED -- two holders at the same time
JdbcTemplateLockProvider WITH usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : refused
```
A node whose clock is ahead by more than `lockAtMostFor` considers every live lock expired. It
takes the lock while somebody else is holding it, and the whole mechanism silently stops working
— for that node only, which is why it presents as "it usually runs once".
The skew is simulated with ShedLock's own `ClockProvider.setClock(...)`, which is what the
provider reads for "now" on the non-database path. That is also the cheapest way to test this on
your own code: no NTP fiddling and no root.
`usingDbTime()` moves both the write and the comparison into a SQL statement, so there is exactly
one clock in the system. It is supported on PostgreSQL, MySQL, MariaDB, MS SQL, Oracle, DB2, HSQL
and H2.
**Use it.** Forty seconds of skew is not exotic — a VM resuming from a snapshot, a container on a
host with a broken NTP client, or a laptop that just woke up will all do it, and cloud instances
drift more than people expect.
## What it does not fix
`usingDbTime()` makes expiry decisions consistent. It does not shorten the window created by a
crash: if a holder dies, the lock still stays held until `lockAtMostFor` elapses. Nor does it help
if your `lockAtMostFor` is shorter than the job — in that case the lock expires legitimately, a
second replica starts, and both run. Size `lockAtMostFor` against the worst case, not the average.
next: [When not to use a lock](05-when-not-to-use-a-lock.md)

46
scheduling/docs/05-when-not-to-use-a-lock.md Normal file
View File

@@ -0,0 +1,46 @@
prev: [Clock skew](04-clock-skew.md) · [README](../README.md)
# 5. When not to use a lock
ShedLock is small, it has one dependency on your database, and it solves the stated problem. It
is still worth asking whether the problem should exist.
**Make the job idempotent instead.** A lock is a way of avoiding a second execution. Idempotence
is a way of not caring about one. If the job's work can be keyed — "mark invoices dated
2026-09-01 as sent, where they are not already marked" — then three replicas running it produce
one outcome and you have removed a distributed-systems dependency rather than adding one. This is
almost always the better engineering, and it is almost never what the article you searched for
suggests.
**A lock is not a guarantee of exactly-once.** It is a guarantee of at-most-one-per-lock-window,
which is different. If the holder dies halfway, the job did not complete and nothing retries it —
ShedLock has no notion of failure, only of expiry. Pairing it with a job table that records
completion is what makes "exactly once" true, and at that point the job table is doing most of
the work.
**Consider the scheduler you already have.** A Kubernetes `CronJob` runs one pod per schedule and
needs no lock, no library and no table; it costs you a pod start per run and the schedule lives
outside the application. Quartz in clustered mode owns misfire policy, persistence and recovery,
at the price of eleven tables and a great deal of configuration. ShedLock sits between them: it
adds a lock to the scheduler you already have, and deliberately adds nothing else.
**A lock provider needs its store to be consistent.** The JDBC provider is safe because a
conditional `UPDATE` on one row is atomic in every relational database. The Redis provider is
safe on a single instance and, as its own documentation notes, is subject to the well-known
argument about locks over Redis replication. If your store is eventually consistent, your lock
is too.
## A short checklist for the version you ship
- `@EnableSchedulerLock` present, and a `LockProvider` bean present — without both, the
annotation does nothing ([chapter 1](01-three-replicas-three-executions.md))
- `usingDbTime()` on ([chapter 4](04-clock-skew.md))
- `lockAtMostFor` longer than the worst-case run time, and you have accepted the gap it implies
after a crash
- `lockAtLeastFor` non-zero for any job shorter than the interval between two replicas' timers
- `spring.task.scheduling.pool.size` raised above 1 ([chapter 3](03-one-scheduler-thread.md))
- the lock names are unique per job — two jobs sharing a name share a lock, and one of them stops
running
- the method carrying `@SchedulerLock` is called from outside its own class, and is not `final`
[README](../README.md)

17
scheduling/docs/output/clock-skew.txt Normal file
View File

@@ -0,0 +1,17 @@
== One lock, two holders: what usingDbTime() prevents ==
lockAtMostFor = 30s, both callers ask for the same lock name.
JdbcTemplateLockProvider WITHOUT usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : ACQUIRED -- two holders at the same time
JdbcTemplateLockProvider WITH usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : refused
Without usingDbTime() the expiry comparison happens against the calling
JVM's clock, so a node that is ahead by more than lockAtMostFor considers
every live lock expired. With it, both the write and the comparison happen
in the database, and there is only one clock in the system.

12
scheduling/docs/output/scheduler-pool-1.txt Normal file
View File

@@ -0,0 +1,12 @@
== spring.task.scheduling.pool.size=1 (the default) ==
over 8 seconds, three @Scheduled methods on one application
slow() fixedRate 2000 ms, sleeps 1800 ms : 5 executions
fast() fixedRate 200 ms : 40 executions
throwing() fixedRate 300 ms, always throws : 27 executions
distinct scheduler threads : 1 [scheduling-1]
longest gap between two fast() executions : 1995 ms (200 ms was the schedule)
fast() executions that started within 20 ms
of the previous one (the catch-up burst) : 35

18
scheduling/docs/output/scheduler-pool-4.txt Normal file
View File

@@ -0,0 +1,18 @@
== spring.task.scheduling.pool.size=4 ==
over 8 seconds, three @Scheduled methods on one application
slow() fixedRate 2000 ms, sleeps 1800 ms : 5 executions
fast() fixedRate 200 ms : 41 executions
throwing() fixedRate 300 ms, always throws : 27 executions
distinct scheduler threads : 4 [scheduling-1, scheduling-2, scheduling-3, scheduling-4]
longest gap between two fast() executions : 200 ms (200 ms was the schedule)
fast() executions that started within 20 ms
of the previous one (the catch-up burst) : 0
throwing() kept its schedule after every failure. Spring wraps a scheduled
method in TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER, so each exception is
logged by o.s.s.s.TaskUtils$LoggingErrorHandler as "Unexpected error
occurred in scheduled task" and then discarded. A raw
ScheduledExecutorService would have cancelled the task at the first one.

8
scheduling/docs/output/tests.txt Normal file
View File

@@ -0,0 +1,8 @@
[INFO] Running com.ankurm.scheduling.SingleSchedulerThreadTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.10 s -- in com.ankurm.scheduling.SingleSchedulerThreadTest
[INFO] Running com.ankurm.scheduling.LargerSchedulerPoolTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.422 s -- in com.ankurm.scheduling.LargerSchedulerPoolTest
[INFO] Running com.ankurm.scheduling.ClockSkewTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.964 s -- in com.ankurm.scheduling.ClockSkewTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

37
scheduling/docs/output/three-replicas-locked.txt Normal file
View File

@@ -0,0 +1,37 @@
profile=locked replicas=3 run-for=30s job rate=3s job duration=PT0.2S replica-3 clock skew=0s
-- every execution, in order --
instance_id | started_at
-------------+--------------
replica-2 | 22:49:29.233
replica-3 | 22:49:31.931
replica-3 | 22:49:34.925
replica-3 | 22:49:37.934
replica-3 | 22:49:40.934
replica-3 | 22:49:43.925
replica-3 | 22:49:46.934
(7 rows)
-- executions per replica --
instance_id | count
-------------+-------
replica-2 | 1
replica-3 | 6
(2 rows)
-- pairs of executions that overlapped (a second replica started while the first was still working) --
overlapping_pairs
-------------------
0
(1 row)
first | first_at | second | second_at
-------+----------+--------+-----------
(0 rows)
-- the lock row --
name | locked_by | locked_at | lock_until
---------------+-----------+--------------+--------------
nightlyReport | unknown | 17:19:46.921 | 17:19:47.921
(1 row)

60
scheduling/docs/output/three-replicas-unlocked.txt Normal file
View File

@@ -0,0 +1,60 @@
profile=unlocked replicas=3 run-for=30s job rate=3s job duration=PT0.2S replica-3 clock skew=0s
-- every execution, in order --
instance_id | started_at
-------------+--------------
replica-1 | 22:48:54.564
replica-3 | 22:48:54.752
replica-2 | 22:48:55.052
replica-1 | 22:48:57.551
replica-3 | 22:48:57.736
replica-2 | 22:48:58.030
replica-1 | 22:49:00.551
replica-3 | 22:49:00.736
replica-2 | 22:49:01.030
replica-1 | 22:49:03.551
replica-3 | 22:49:03.736
replica-2 | 22:49:04.030
replica-1 | 22:49:06.551
replica-3 | 22:49:06.736
replica-2 | 22:49:07.030
replica-1 | 22:49:09.551
replica-3 | 22:49:09.736
replica-2 | 22:49:10.030
replica-1 | 22:49:12.551
replica-3 | 22:49:12.736
replica-2 | 22:49:13.030
replica-1 | 22:49:15.551
replica-3 | 22:49:15.736
replica-2 | 22:49:16.030
(24 rows)
-- executions per replica --
instance_id | count
-------------+-------
replica-1 | 8
replica-2 | 8
replica-3 | 8
(3 rows)
-- pairs of executions that overlapped (a second replica started while the first was still working) --
overlapping_pairs
-------------------
8
(1 row)
first | first_at | second | second_at
-----------+--------------+-----------+--------------
replica-1 | 22:48:54.564 | replica-3 | 22:48:54.752
replica-1 | 22:48:57.551 | replica-3 | 22:48:57.736
replica-1 | 22:49:00.551 | replica-3 | 22:49:00.736
replica-1 | 22:49:03.551 | replica-3 | 22:49:03.736
replica-1 | 22:49:06.551 | replica-3 | 22:49:06.736
replica-1 | 22:49:09.551 | replica-3 | 22:49:09.736
(6 rows)
-- the lock row --
name | locked_by | locked_at | lock_until
------+-----------+-----------+------------
(0 rows)