@Scheduled, ShedLock and Distributed Cron: Scheduling That Survives Three Replicas
Scale a Spring Boot deployment to three replicas and every @Scheduled method runs three times per tick. Measured on Boot 4.1.1 and PostgreSQL: 24 executions where 8 were due, a @SchedulerLock annotation that silently does nothing without @EnableSchedulerLock, a single scheduler thread that fires 35 of 40 executions in a burst rather than skipping them, and a node whose clock is 40 seconds fast taking a lock somebody else is holding.
@Scheduled is a per-JVM timer. It has no idea that other JVMs exist.
That is the whole problem, and it does not appear in development, where there is one instance, or in staging, where there is usually one instance. It appears the first time somebody scales a deployment to three replicas, and it appears as duplicate work rather than as an error: two invoice emails, two refunds, two rows where the unique constraint you did not add would have saved you.
The fix is well known and takes about fifteen lines. Most of what follows is about the three ways those fifteen lines can be present and still not work.
If you want…
Read
the failure, reproduced, and the smallest thing that fixes it
Part 1
why the annotation is there and the lock table is empty
Part 2
scheduler threads, clock skew, and whether you want a lock at all
Part 3
Versions. Everything below was run on JDK 25.0.4.1+1 (Temurin), Spring Boot 4.1.1 (published to Maven Central on 20 August 2026), Spring Framework 7.0.9, ShedLock 7.9.0 and PostgreSQL 14.24. ShedLock is not managed by spring-boot-dependencies, so its version is pinned by hand; 7.9.0 comes from maven-metadata.xml on Maven Central. The companion project is asmhatre/spring-async-demo, module scheduling/ — three replicas, one database, and a docker-compose.yml if you would rather not unpack PostgreSQL by hand.
Part 1 — Three replicas, three executions
The companion project starts three copies of one application against one PostgreSQL database, lets a three-second schedule tick for thirty seconds, and records every execution in a table with the instance that ran it. Counting rows is the measurement.
Twenty-four executions where eight were due. Eight of them started while another replica was still inside the method — 188 milliseconds apart in the first pair, against a job that takes 200 milliseconds. If the job is not idempotent, that is not a race you will win with a database transaction; the three replicas are doing genuinely independent work at genuinely the same time.
The fifteen lines
ShedLock adds a lock to the scheduler you already have, and deliberately adds nothing else. Two dependencies, one configuration class, one annotation, one table.
create table if not exists shedlock (
name varchar(64) not null,
lock_until timestamp not null,
locked_at timestamp not null,
locked_by varchar(255) not null,
primary key (name)
);
The mechanism is a conditional UPDATE: acquire means “set lock_until to a future time wherelock_until is already in the past”. Only one replica’s update can win, because the row is locked for the duration of that statement. There is no consensus protocol and nothing to operate; the database you already have does the work.
Part 2 — The annotation that does nothing
Here is the transcript from the failing run again, this time the last part of it:
-- the lock row --
name | locked_by | locked_at | lock_until
------+-----------+-----------+------------
(0 rows)
The job in that run is annotated @SchedulerLock. The dependency is on the classpath, the table exists, the application started cleanly, and nothing was ever locked.
@SchedulerLock on its own is inert. It needs @EnableSchedulerLock to install the interceptor and a LockProvider bean to have somewhere to record the lock. In the companion project both live on one @Configuration class which is @Profile("locked"), so the entire difference between twenty-four executions and seven is whether that class is active. The annotation on the method is identical in both runs.
There is no warning for the missing half. An application with the annotation and without the plumbing behaves exactly like an application with no locking at all, starts without complaint at any log level, and passes code review — because what a reviewer sees is the annotation. If you take one thing from this article: after wiring ShedLock, look at the shedlock table and confirm there is a row in it. That query is the only proof.
What the locked run looks like
-- executions per replica --
instance_id | count
-------------+-------
replica-2 | 1
replica-3 | 6
-- pairs of executions that overlapped --
overlapping_pairs : 0
-- the lock row --
name | locked_by | locked_at | lock_until
---------------+-----------+--------------+--------------
nightlyReport | unknown | 17:19:46.921 | 17:19:47.921
Seven executions for seven ticks, no overlap. Three details in that output are worth more than the headline.
ShedLock does not distribute work. Six of the seven executions are replica-3; replica-2 won the first tick and never won another. The lock is not a queue and there is no round-robin: whichever replica’s timer fires first each tick takes it, and on a stable cluster that is overwhelmingly the same replica. This is correct behaviour for “run exactly once” and completely wrong if what you wanted was “spread the work”. For that you want a work queue, not a lock.
locked_by is not identity. It reads unknown because ShedLock fills that column with the host name and the container that produced this transcript has none that resolves. It is a diagnostic field and is never read back for correctness — but on Kubernetes you will get a column full of pod hashes or of unknown, which is worth fixing before you need it during an incident.
The timestamps are in a different time zone from your own.locked_at reads 17:19:46 while the execution recorded at the same instant reads 22:49:46. That is usingDbTime() working as intended: the lock timestamps are written by PostgreSQL in UTC, while the application writes its own rows in the JVM’s zone. Anyone eyeballing the shedlock table next to the application’s tables will see a five-and-a-half-hour discrepancy and conclude the lock is stale. It is not.
The two durations
lockAtMostFor answers “the holder was kill -9‘d; how long before somebody else may run this?” It must be longer than the longest the job could possibly take, because when it expires the lock is available whether or not the job finished. Set it to five minutes for a job that normally takes ten seconds and accept a five-minute gap after a crash. That is the trade, and no setting avoids it.
lockAtLeastFor keeps the lock held for a minimum period after a fast job finishes. Without it, a job that completes in 20 ms releases the lock in time for the next replica’s tick — which may be milliseconds later — to pick it up, and you are back to two executions.
@EnableSchedulerLock(defaultLockAtMostFor = ...) has no default value in the annotation, so the compiler makes you supply it. That is deliberate: there is no safe guess.
Leave interceptMode alone. ShedLock 7 defaults to PROXY_METHOD. The PROXY_SCHEDULER mode that most tutorials still show is deprecated, and its own Javadoc says it “requires a reflection hack to work well with Spring 6.2”.
Because PROXY_METHOD is a Spring AOP proxy, every proxy limitation applies here too: a @SchedulerLock method called from inside its own class is not locked, and a final method is not locked. Both fail the same silent way as @Async — see the companion article on @Async, where the same trap is measured by thread name.
Part 3 — The two things that break it after it works
One scheduler thread
spring.task.scheduling.pool.size defaults to 1. Every @Scheduled method in the application shares that thread.
The expected consequence is that a slow job starves the others. What actually happens is more interesting and considerably worse. Three scheduled methods — one sleeping 1800 ms on a 2000 ms schedule, one on a 200 ms schedule, one that throws — running for eight seconds:
pool.size=1 (default)
pool.size=4
executions of the 200 ms job
40
41
longest gap between two of them
1995 ms
201 ms
executions starting within 20 ms of the previous one
35
0
distinct scheduler threads
1
4
The execution count is the same. A fixedRate schedule does not skip a tick it could not run: the missed executions accumulate and are fired back to back the moment the thread is free. Thirty-five of the forty arrived in a burst.
So the metric everyone has — “the job ran 40 times, as expected” — is green, while the behaviour is two seconds of silence followed by thirty-five invocations a few milliseconds apart. If that job calls a rate-limited API, the burst is the incident.
Use fixedDelay when you mean it.fixedRate means “start every N seconds” and accumulates a backlog it will later burst through. fixedDelay means “N seconds after the last one finished” and cannot accumulate anything. Most jobs described as “every five minutes” actually want fixedDelay. Raise spring.task.scheduling.pool.size as well — it costs a handful of mostly idle threads — or set spring.threads.virtual.enabled=true, which replaces the pool with a SimpleAsyncTaskScheduler over virtual threads and removes the shared-thread problem along with the bound.
While we are here: the job that threw on all 27 of its executions kept its schedule. Spring wraps a scheduled method in TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER, so each exception is logged by o.s.s.s.TaskUtils$LoggingErrorHandler as Unexpected error occurred in scheduled task and then discarded. A raw ScheduledExecutorService would have cancelled the task after the first failure, which is where the folklore comes from and it does not apply here. The flip side: a job that has been failing since the last deploy produces nothing but a recurring ERROR line.
Clock skew, and what usingDbTime() is for
Without usingDbTime(), each replica writes lock_until from its own clock and compares lock_until against its own clock. The lock is then only as good as three machines’ agreement about what time it is.
Three replicas racing cannot prove this, because a race can go either way. Driving the LockProvider directly can: two acquisitions, ordered by the test, with the only variable being what the second caller believes the time to be.
lockAtMostFor = 30s, both callers ask for the same lock name.
JdbcTemplateLockProvider WITHOUT usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : ACQUIRED -- two holders at the same time
JdbcTemplateLockProvider WITH usingDbTime()
node with a correct clock : acquired
node with a clock 40s fast : refused
A node whose clock is ahead by more than lockAtMostFor considers every live lock expired. It takes the lock while somebody else is holding it, and the whole mechanism silently stops working — for that node only, which is why it presents as “it usually runs once”.
Forty seconds of skew is not exotic. A VM resuming from a snapshot, a container on a host with a broken NTP client, or a laptop that has just woken up will all do it. usingDbTime() moves both the write and the comparison into the SQL statement, so there is exactly one clock in the system. It is supported on PostgreSQL, MySQL, MariaDB, MS SQL, Oracle, DB2, HSQL and H2. Use it.
The skew in that test is simulated with ShedLock’s own ClockProvider.setClock(...), which is what the provider reads for “now” on the non-database path. That is also the cheapest way to test this against your own code: no NTP fiddling and no root.
The long tail
usingDbTime() does not shorten the window a crash creates: the lock still stays held until lockAtMostFor elapses — chapter 4
Two jobs sharing a lock name share a lock, and one of them quietly stops running — chapter 5
The Redis lock provider is safe on a single instance and inherits the well-known argument about locks over Redis replication; if your store is eventually consistent, so is your lock — chapter 5
ShedLock has no notion of failure, only of expiry: if the holder dies halfway, nothing retries the job — chapter 5
A throwaway PostgreSQL for reproducing all of this, with no Docker and no root — scripts/postgres.sh
Before you ship it
@EnableSchedulerLock present and a LockProvider bean present — verify by looking at the table, not the code
usingDbTime() on
lockAtMostFor longer than the worst-case run time, and the gap it implies after a crash accepted
lockAtLeastFor non-zero for any job shorter than the interval between two replicas’ timers
spring.task.scheduling.pool.size raised above 1, and fixedDelay used where you meant it
lock names unique per job
the @SchedulerLock method called from outside its own class, and not final
Should you add the lock at all? Often the better answer is to make the job idempotent. A lock avoids a second execution; idempotence means not caring about one. If the work can be keyed — “mark invoices dated 2026-09-01 as sent, where they are not already marked” — then three replicas produce one outcome and you have removed a distributed-systems dependency instead of adding one.
And note what a lock is not: it guarantees at-most-one-per-lock-window, not exactly-once. If the holder dies halfway the job did not complete and nothing retries it. Pair it with a table that records completion and you have exactly-once — at which point the table is doing most of the work. A Kubernetes CronJob needs no lock, no library and no table at all; Quartz in clustered mode owns misfire policy and recovery, at the price of eleven tables. ShedLock sits between them on purpose.
Further reading
The companion project — three replicas, one database, five documentation chapters and every transcript above
No Comments yet!