Ankur Mhatre
eaa8171966
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
2.6 KiB
prev: One scheduler thread · README · next: When not to use a lock
4. Clock skew, and what usingDbTime() is for
JdbcTemplateLockProvider.Configuration.builder()
.withJdbcTemplate(new JdbcTemplate(dataSource))
.usingDbTime()
.build();
Without usingDbTime(), each replica writes lock_until from its own clock and compares
lock_until against its own clock. The lock is then only as good as the agreement between three
machines about what time it is.
Three replicas racing is not a proof, because a race can go either way. ClockSkewTest drives
the LockProvider directly instead, so the two acquisitions are ordered by the test and the only
variable is what the second caller believes the time to be
(docs/output/clock-skew.txt):
lockAtMostFor = 30s, both callers ask for the same lock name.
JdbcTemplateLockProvider WITHOUT usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : ACQUIRED -- two holders at the same time
JdbcTemplateLockProvider WITH usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : refused
A node whose clock is ahead by more than lockAtMostFor considers every live lock expired. It
takes the lock while somebody else is holding it, and the whole mechanism silently stops working
— for that node only, which is why it presents as "it usually runs once".
The skew is simulated with ShedLock's own ClockProvider.setClock(...), which is what the
provider reads for "now" on the non-database path. That is also the cheapest way to test this on
your own code: no NTP fiddling and no root.
usingDbTime() moves both the write and the comparison into a SQL statement, so there is exactly
one clock in the system. It is supported on PostgreSQL, MySQL, MariaDB, MS SQL, Oracle, DB2, HSQL
and H2.
Use it. Forty seconds of skew is not exotic — a VM resuming from a snapshot, a container on a host with a broken NTP client, or a laptop that just woke up will all do it, and cloud instances drift more than people expect.
What it does not fix
usingDbTime() makes expiry decisions consistent. It does not shorten the window created by a
crash: if a holder dies, the lock still stays held until lockAtMostFor elapses. Nor does it help
if your lockAtMostFor is shorter than the job — in that case the lock expires legitimately, a
second replica starts, and both run. Size lockAtMostFor against the worst case, not the average.
next: When not to use a lock