Ankur Mhatre
6af6e4a3f1
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
4.1 KiB
prev: Three replicas, three executions · README · next: One scheduler thread
2. The lock
Activate LockConfiguration and the same 30-second run produces this
(docs/output/three-replicas-locked.txt):
-- executions per replica --
instance_id | count
-------------+-------
replica-2 | 1
replica-3 | 6
-- pairs of executions that overlapped --
overlapping_pairs
-------------------
0
-- the lock row --
name | locked_by | locked_at | lock_until
---------------+-----------+--------------+--------------
nightlyReport | unknown | 17:19:46.921 | 17:19:47.921
Seven executions for seven ticks, no overlap. The mechanism is one row in one table and a
conditional UPDATE: acquire means "set lock_until to a future time where lock_until is
already in the past", and only one replica's update can win because the row is locked for the
duration of that statement.
Three things in that transcript deserve attention.
ShedLock does not distribute work
Six of the seven executions are replica-3. The lock is not a queue and there is no round-robin:
whichever replica's timer fires first each tick takes the lock, and on a stable cluster that is
overwhelmingly the same replica — here replica-2 won the first tick and then never won
another. This is correct behaviour for "run exactly once" and
completely wrong if what you actually wanted was "spread the work across the cluster". For that
you want a work queue, not a lock.
locked_by is not identity
It reads unknown above. ShedLock fills that column with the host name, and the host name of the
container that produced this transcript does not resolve, so it falls back. It is a diagnostic
field, never read back for correctness — but it does mean that on Kubernetes you often get a
column full of pod hashes or of unknown, and it is worth setting something meaningful if you
plan to use it during an incident.
The timestamps are in a different time zone from your own
locked_at reads 17:19:46 while the execution recorded at the same moment in the same file
reads 22:49:46. That is not a
bug; it is usingDbTime() doing its job. With it, the timestamps are written by PostgreSQL in
UTC, whereas started_at is written by the application in the JVM's zone. Anyone eyeballing the
shedlock table next to the application's own tables will see a five-and-a-half-hour discrepancy
and think the lock is stale. It is not.
The two durations
@SchedulerLock(name = "nightlyReport", lockAtMostFor = "PT20S", lockAtLeastFor = "PT1S")
lockAtMostFor is the answer to "the holder was kill -9'd; how long before somebody else
may run this?" It must be longer than the longest the job could possibly take, because when it
expires the lock is available whether or not the job finished. Set it to five minutes for a job
that normally takes ten seconds, and accept a five-minute gap after a crash — that is the trade,
and there is no setting that avoids it.
lockAtLeastFor keeps the lock held for a minimum period after a fast job finishes. It exists
for clock skew and for schedules where two replicas' timers fire within milliseconds of each
other: without it, a job that completes in 20 ms releases the lock in time for the next replica's
tick to pick it up, and you are back to two executions.
@EnableSchedulerLock(defaultLockAtMostFor = ...) has no default value in the annotation, so it
must be supplied. That is deliberate: there is no safe guess.
interceptMode
ShedLock 7 defaults to PROXY_METHOD. The PROXY_SCHEDULER mode that most tutorials still show
is deprecated, and its own Javadoc says it "requires a reflection hack to work well with Spring
6.2". Leave the default alone.
Because PROXY_METHOD is a Spring AOP proxy, everything from
the @Async article applies
here too: a @SchedulerLock method called from inside its own class is not locked, and a final
method is not locked.
next: One scheduler thread