asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
78 lines
3.4 KiB
Markdown
78 lines
3.4 KiB
Markdown
# 13. Graceful shutdown vs. kill -9
|
|
|
|
[← 12. The AUTO_SERVER trap](12-the-auto-server-trap.md) · [Next: 14. The DDL lock window →](14-the-ddl-lock-window.md)
|
|
|
|
`application.yml` sets one line that does nothing by itself:
|
|
|
|
```yaml
|
|
server:
|
|
shutdown: graceful
|
|
```
|
|
|
|
`server.shutdown: graceful` only changes behavior on `SIGTERM` — it stops accepting
|
|
new connections but lets in-flight requests finish before the process exits. An
|
|
earlier version of
|
|
[`stop-instance.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/stop-instance.sh)
|
|
used `kill -9`, which bypasses graceful shutdown entirely — the process disappears
|
|
mid-request, and every request in flight at that instant surfaces in the load
|
|
generator as a raw `ConnectException` or `IOException`. Switching to `SIGTERM`
|
|
(`kill -15`), with a bounded wait for a clean exit and `SIGKILL` only as a fallback,
|
|
is the first half of the fix:
|
|
|
|
```bash
|
|
kill -15 "$PID"
|
|
for i in $(seq 1 40); do
|
|
kill -0 "$PID" 2>/dev/null || break
|
|
sleep 0.25
|
|
done
|
|
if kill -0 "$PID" 2>/dev/null; then
|
|
kill -9 "$PID"
|
|
fi
|
|
```
|
|
|
|
That alone wasn't enough. `server.shutdown: graceful` starts refusing new connections
|
|
the instant `SIGTERM` arrives — but the load generator's health checker polls every
|
|
300ms, and the pool didn't yet know to stop routing traffic there. The gap between
|
|
"the process just stopped accepting connections" and "the load balancer's health
|
|
check has noticed and rerouted" is exactly where `ConnectException` bursts kept
|
|
showing up, even with `SIGTERM` in place.
|
|
|
|
The second half of the fix is a way to say "stop sending me traffic" *before* the
|
|
process is touched at all:
|
|
[`DrainController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/DrainController.java):
|
|
|
|
```java
|
|
@PostMapping("/admin/drain")
|
|
public String drain() {
|
|
AvailabilityChangeEvent.publish(events, this, ReadinessState.REFUSING_TRAFFIC);
|
|
return "draining";
|
|
}
|
|
```
|
|
|
|
Publishing `ReadinessState.REFUSING_TRAFFIC` flips `/actuator/health`'s readiness
|
|
group immediately — this is the same event a Kubernetes-style `preStop` hook
|
|
publishes before the container is sent `SIGTERM`. `stop-instance.sh` calls it, sleeps,
|
|
*then* sends `SIGTERM`:
|
|
|
|
```bash
|
|
curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
|
|
sleep 1.5
|
|
kill -15 "$PID"
|
|
```
|
|
|
|
That 1.5-second pause is deliberate slack for the health checker's 300ms poll
|
|
interval — enough for at least a couple of checks to land and pull this instance out
|
|
of rotation before it's asked to stop at all. Together, drain-then-SIGTERM is what
|
|
took the rolling-restart portion of the article's live run to zero
|
|
`ConnectException`/`IOException` errors — the six errors that remain in the final
|
|
summary are a completely different, database-level cause, covered next.
|
|
|
|
## Going deeper
|
|
|
|
- Spring's own `ReadinessState` and the Kubernetes probe pattern it mirrors:
|
|
[Spring Boot reference docs, Application Availability](https://docs.spring.io/spring-boot/reference/actuator/application-availability.html) (nofollow).
|
|
- `/admin/drain` is a diagnostic-grade endpoint with no auth — see
|
|
[chapter 15](15-production-checklist.md) for what to do with it before shipping.
|
|
|
|
[← 12. The AUTO_SERVER trap](12-the-auto-server-trap.md) · [Next: 14. The DDL lock window →](14-the-ddl-lock-window.md)
|