Add the scheduling module

Three replicas of one application against one PostgreSQL database, proving
duplicate @Scheduled execution and then removing it with ShedLock: 24
executions where 8 were due, then 7 for 7 ticks.

Also measured: @SchedulerLock without @EnableSchedulerLock does nothing and
warns about nothing; spring.task.scheduling.pool.size=1 does not starve a
fixedRate job but delays it and fires 35 of 40 executions in a burst; and a
node whose clock is 40 seconds fast takes a live lock unless the provider
uses usingDbTime().

scheduling/docs/03-one-scheduler-thread.md

@@ -0,0 +1,56 @@
+prev: [The lock](02-the-lock.md) · [README](../README.md) · next: [Clock skew](04-clock-skew.md)
+
+# 3. One scheduler thread
+
+`spring.task.scheduling.pool.size` defaults to **1**. Every `@Scheduled` method in the
+application shares that one thread.
+
+The expected consequence is that a slow job starves the others. What actually happens is more
+interesting, and considerably worse.
+
+Three scheduled methods — one that sleeps 1800 ms on a 2000 ms schedule, one on a 200 ms
+schedule, one that throws — running for eight seconds. First with the default single thread
+([output](output/scheduler-pool-1.txt)), then with four
+([output](output/scheduler-pool-4.txt)):
+
+| | pool.size=1 | pool.size=4 |
+|---|---|---|
+| `fast()` executions | 40 | 41 |
+| longest gap between two `fast()` executions | **1995 ms** | 201 ms |
+| `fast()` executions starting within 20 ms of the previous one | **35** | 0 |
+| distinct scheduler threads | 1 | 4 |
+
+The execution *count* is the same. A `fixedRate` schedule does not skip a tick it could not run:
+the missed executions accumulate and are then fired back to back the moment the thread is free.
+Thirty-five of the forty executions arrived in a burst.
+
+So the metric everyone has — "the job ran 40 times, as expected" — is green, while the job's
+actual behaviour is two seconds of silence followed by thirty-five invocations in a few
+milliseconds. If that job calls a rate-limited API, or opens a database connection each time, the
+burst is the incident.
+
+## What to set
+
+- **`spring.task.scheduling.pool.size`**: at least the number of `@Scheduled` methods that can
+  overlap. It costs a handful of mostly idle threads.
+- **`fixedDelay` instead of `fixedRate`** where "every N seconds" really means "N seconds after
+  the last one finished". `fixedDelay` schedules the next run only after the current one
+  completes, so it cannot accumulate a backlog to burst through.
+- **`spring.threads.virtual.enabled=true`** replaces the pool with a `SimpleAsyncTaskScheduler`
+  over virtual threads, which removes the shared-thread problem entirely. It also removes the
+  bound: `spring.task.scheduling.pool.size` is one of the properties Boot's own metadata marks
+  as having no effect when virtual threads are on.
+
+## An exception does not stop the schedule
+
+`throwing()` threw on all 27 of its executions and kept its schedule. Spring wraps a scheduled
+method in `TaskUtils.LOG_AND_SUPPRESS_ERROR_HANDLER`, so the exception is logged by
+`o.s.s.s.TaskUtils$LoggingErrorHandler` as `Unexpected error occurred in scheduled task` and
+discarded. A raw `ScheduledExecutorService` would have cancelled the task after the first
+failure — which is where the folklore comes from, and it does not apply to `@Scheduled`.
+
+The flip side is that a job which has been failing since the last deploy produces nothing but a
+recurring `ERROR` line. Supply your own `SchedulingConfigurer` with an error handler if you want
+a metric.
+
+next: [Clock skew](04-clock-skew.md)