Ankur Mhatre
eaa8171966
Three replicas of one application against one PostgreSQL database, proving duplicate @Scheduled execution and then removing it with ShedLock: 24 executions where 8 were due, then 7 for 7 ticks. Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and warns about nothing; spring.task.scheduling.pool.size=1 does not starve a fixedRate job but delays it and fires 35 of 40 executions in a burst; and a node whose clock is 40 seconds fast takes a live lock unless the provider uses usingDbTime().
76 lines
2.7 KiB
Markdown
76 lines
2.7 KiB
Markdown
[README](../README.md) · next: [The lock](02-the-lock.md)
|
|
|
|
# 1. Three replicas, three executions
|
|
|
|
`@Scheduled` is a per-JVM timer. It has no idea that other JVMs exist. Scale a deployment to
|
|
three replicas and every `@Scheduled` method in the application runs three times per tick, on
|
|
three machines, at almost the same instant.
|
|
|
|
That is the whole problem, and it is worth seeing rather than reading. `scripts/three-replicas.sh`
|
|
starts three copies of this application against one PostgreSQL database, lets the 3-second
|
|
schedule tick for 30 seconds, and counts.
|
|
|
|
From [`docs/output/three-replicas-unlocked.txt`](output/three-replicas-unlocked.txt):
|
|
|
|
```
|
|
-- executions per replica --
|
|
instance_id | count
|
|
-------------+-------
|
|
replica-1 | 8
|
|
replica-2 | 8
|
|
replica-3 | 8
|
|
|
|
-- pairs of executions that overlapped --
|
|
overlapping_pairs
|
|
-------------------
|
|
8
|
|
|
|
first | first_at | second | second_at
|
|
-----------+--------------+-----------+--------------
|
|
replica-1 | 22:48:54.564 | replica-3 | 22:48:54.752
|
|
replica-1 | 22:48:57.551 | replica-3 | 22:48:57.736
|
|
replica-1 | 22:49:00.551 | replica-3 | 22:49:00.736
|
|
```
|
|
|
|
Twenty-four executions where eight were due, and eight of them started while another replica was
|
|
still inside the method — 188 milliseconds apart in the first pair, against a job that only takes
|
|
200 milliseconds. Anything in that method which
|
|
is not idempotent is now a support ticket: a second invoice email, a double refund, two rows
|
|
where the unique constraint you did not add would have saved you.
|
|
|
|
## The part that makes it hard to spot
|
|
|
|
`ReportJob` is annotated `@SchedulerLock` in this run. Look at the end of the same transcript:
|
|
|
|
```
|
|
-- the lock row --
|
|
name | locked_by | locked_at | lock_until
|
|
------+-----------+-----------+------------
|
|
(0 rows)
|
|
```
|
|
|
|
The annotation is present, the lock table exists, the application started cleanly, and nothing
|
|
was ever locked. `@SchedulerLock` on its own is inert: it needs `@EnableSchedulerLock` to install
|
|
the interceptor and a `LockProvider` bean to have somewhere to record the lock. In this module
|
|
both live on `LockConfiguration`, which is `@Profile("locked")`.
|
|
|
|
There is no warning for the missing half. That is the single most important sentence in this
|
|
module — an application with the annotation and without the plumbing behaves exactly like an
|
|
application with no locking at all, and it looks locked in code review.
|
|
|
|
## Reproducing it
|
|
|
|
```bash
|
|
scripts/postgres.sh start # a throwaway PostgreSQL 14, no Docker, no root
|
|
scripts/three-replicas.sh unlocked
|
|
scripts/three-replicas.sh locked
|
|
```
|
|
|
|
or, with Docker:
|
|
|
|
```bash
|
|
docker compose up --build # PROFILE=locked docker compose up --build
|
|
```
|
|
|
|
next: [The lock](02-the-lock.md)
|