Ankur Mhatre
6af6e4a3f1
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
91 lines
4.1 KiB
Markdown
91 lines
4.1 KiB
Markdown
prev: [Three replicas, three executions](01-three-replicas-three-executions.md) · [README](../README.md) · next: [One scheduler thread](03-one-scheduler-thread.md)
|
|
|
|
# 2. The lock
|
|
|
|
Activate `LockConfiguration` and the same 30-second run produces this
|
|
([`docs/output/three-replicas-locked.txt`](output/three-replicas-locked.txt)):
|
|
|
|
```
|
|
-- executions per replica --
|
|
instance_id | count
|
|
-------------+-------
|
|
replica-2 | 1
|
|
replica-3 | 6
|
|
|
|
-- pairs of executions that overlapped --
|
|
overlapping_pairs
|
|
-------------------
|
|
0
|
|
|
|
-- the lock row --
|
|
name | locked_by | locked_at | lock_until
|
|
---------------+-----------+--------------+--------------
|
|
nightlyReport | unknown | 17:19:46.921 | 17:19:47.921
|
|
```
|
|
|
|
Seven executions for seven ticks, no overlap. The mechanism is one row in one table and a
|
|
conditional `UPDATE`: acquire means "set `lock_until` to a future time **where** `lock_until` is
|
|
already in the past", and only one replica's update can win because the row is locked for the
|
|
duration of that statement.
|
|
|
|
Three things in that transcript deserve attention.
|
|
|
|
## ShedLock does not distribute work
|
|
|
|
Six of the seven executions are `replica-3`. The lock is not a queue and there is no round-robin:
|
|
whichever replica's timer fires first each tick takes the lock, and on a stable cluster that is
|
|
overwhelmingly the same replica — here `replica-2` won the first tick and then never won
|
|
another. This is correct behaviour for "run exactly once" and
|
|
completely wrong if what you actually wanted was "spread the work across the cluster". For that
|
|
you want a work queue, not a lock.
|
|
|
|
## `locked_by` is not identity
|
|
|
|
It reads `unknown` above. ShedLock fills that column with the host name, and the host name of the
|
|
container that produced this transcript does not resolve, so it falls back. It is a diagnostic
|
|
field, never read back for correctness — but it does mean that on Kubernetes you often get a
|
|
column full of pod hashes or of `unknown`, and it is worth setting something meaningful if you
|
|
plan to use it during an incident.
|
|
|
|
## The timestamps are in a different time zone from your own
|
|
|
|
`locked_at` reads `17:19:46` while the execution recorded at the same moment in the same file
|
|
reads `22:49:46`. That is not a
|
|
bug; it is `usingDbTime()` doing its job. With it, the timestamps are written by PostgreSQL in
|
|
UTC, whereas `started_at` is written by the application in the JVM's zone. Anyone eyeballing the
|
|
`shedlock` table next to the application's own tables will see a five-and-a-half-hour discrepancy
|
|
and think the lock is stale. It is not.
|
|
|
|
## The two durations
|
|
|
|
```java
|
|
@SchedulerLock(name = "nightlyReport", lockAtMostFor = "PT20S", lockAtLeastFor = "PT1S")
|
|
```
|
|
|
|
**`lockAtMostFor`** is the answer to "the holder was `kill -9`'d; how long before somebody else
|
|
may run this?" It must be longer than the longest the job could possibly take, because when it
|
|
expires the lock is available whether or not the job finished. Set it to five minutes for a job
|
|
that normally takes ten seconds, and accept a five-minute gap after a crash — that is the trade,
|
|
and there is no setting that avoids it.
|
|
|
|
**`lockAtLeastFor`** keeps the lock held for a minimum period after a fast job finishes. It exists
|
|
for clock skew and for schedules where two replicas' timers fire within milliseconds of each
|
|
other: without it, a job that completes in 20 ms releases the lock in time for the next replica's
|
|
tick to pick it up, and you are back to two executions.
|
|
|
|
`@EnableSchedulerLock(defaultLockAtMostFor = ...)` has no default value in the annotation, so it
|
|
must be supplied. That is deliberate: there is no safe guess.
|
|
|
|
## `interceptMode`
|
|
|
|
ShedLock 7 defaults to `PROXY_METHOD`. The `PROXY_SCHEDULER` mode that most tutorials still show
|
|
is deprecated, and its own Javadoc says it "requires a reflection hack to work well with Spring
|
|
6.2". Leave the default alone.
|
|
|
|
Because `PROXY_METHOD` is a Spring AOP proxy, everything from
|
|
[the `@Async` article](https://ankurm.com/spring-boot-4-async-executors-virtual-threads/) applies
|
|
here too: a `@SchedulerLock` method called from inside its own class is not locked, and a `final`
|
|
method is not locked.
|
|
|
|
next: [One scheduler thread](03-one-scheduler-thread.md)
|