Ankur Mhatre
6af6e4a3f1
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
2.7 KiB
1. Three replicas, three executions
@Scheduled is a per-JVM timer. It has no idea that other JVMs exist. Scale a deployment to
three replicas and every @Scheduled method in the application runs three times per tick, on
three machines, at almost the same instant.
That is the whole problem, and it is worth seeing rather than reading. scripts/three-replicas.sh
starts three copies of this application against one PostgreSQL database, lets the 3-second
schedule tick for 30 seconds, and counts.
From docs/output/three-replicas-unlocked.txt:
-- executions per replica --
instance_id | count
-------------+-------
replica-1 | 8
replica-2 | 8
replica-3 | 8
-- pairs of executions that overlapped --
overlapping_pairs
-------------------
8
first | first_at | second | second_at
-----------+--------------+-----------+--------------
replica-1 | 22:48:54.564 | replica-3 | 22:48:54.752
replica-1 | 22:48:57.551 | replica-3 | 22:48:57.736
replica-1 | 22:49:00.551 | replica-3 | 22:49:00.736
Twenty-four executions where eight were due, and eight of them started while another replica was still inside the method — 188 milliseconds apart in the first pair, against a job that only takes 200 milliseconds. Anything in that method which is not idempotent is now a support ticket: a second invoice email, a double refund, two rows where the unique constraint you did not add would have saved you.
The part that makes it hard to spot
ReportJob is annotated @SchedulerLock in this run. Look at the end of the same transcript:
-- the lock row --
name | locked_by | locked_at | lock_until
------+-----------+-----------+------------
(0 rows)
The annotation is present, the lock table exists, the application started cleanly, and nothing
was ever locked. @SchedulerLock on its own is inert: it needs @EnableSchedulerLock to install
the interceptor and a LockProvider bean to have somewhere to record the lock. In this module
both live on LockConfiguration, which is @Profile("locked").
There is no warning for the missing half. That is the single most important sentence in this module — an application with the annotation and without the plumbing behaves exactly like an application with no locking at all, and it looks locked in code review.
Reproducing it
scripts/postgres.sh start # a throwaway PostgreSQL 14, no Docker, no root
scripts/three-replicas.sh unlocked
scripts/three-replicas.sh locked
or, with Docker:
docker compose up --build # PROFILE=locked docker compose up --build
next: The lock