asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
3.4 KiB
13. Graceful shutdown vs. kill -9
← 12. The AUTO_SERVER trap · Next: 14. The DDL lock window →
application.yml sets one line that does nothing by itself:
server:
shutdown: graceful
server.shutdown: graceful only changes behavior on SIGTERM — it stops accepting
new connections but lets in-flight requests finish before the process exits. An
earlier version of
stop-instance.sh
used kill -9, which bypasses graceful shutdown entirely — the process disappears
mid-request, and every request in flight at that instant surfaces in the load
generator as a raw ConnectException or IOException. Switching to SIGTERM
(kill -15), with a bounded wait for a clean exit and SIGKILL only as a fallback,
is the first half of the fix:
kill -15 "$PID"
for i in $(seq 1 40); do
kill -0 "$PID" 2>/dev/null || break
sleep 0.25
done
if kill -0 "$PID" 2>/dev/null; then
kill -9 "$PID"
fi
That alone wasn't enough. server.shutdown: graceful starts refusing new connections
the instant SIGTERM arrives — but the load generator's health checker polls every
300ms, and the pool didn't yet know to stop routing traffic there. The gap between
"the process just stopped accepting connections" and "the load balancer's health
check has noticed and rerouted" is exactly where ConnectException bursts kept
showing up, even with SIGTERM in place.
The second half of the fix is a way to say "stop sending me traffic" before the
process is touched at all:
DrainController:
@PostMapping("/admin/drain")
public String drain() {
AvailabilityChangeEvent.publish(events, this, ReadinessState.REFUSING_TRAFFIC);
return "draining";
}
Publishing ReadinessState.REFUSING_TRAFFIC flips /actuator/health's readiness
group immediately — this is the same event a Kubernetes-style preStop hook
publishes before the container is sent SIGTERM. stop-instance.sh calls it, sleeps,
then sends SIGTERM:
curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
sleep 1.5
kill -15 "$PID"
That 1.5-second pause is deliberate slack for the health checker's 300ms poll
interval — enough for at least a couple of checks to land and pull this instance out
of rotation before it's asked to stop at all. Together, drain-then-SIGTERM is what
took the rolling-restart portion of the article's live run to zero
ConnectException/IOException errors — the six errors that remain in the final
summary are a completely different, database-level cause, covered next.
Going deeper
- Spring's own
ReadinessStateand the Kubernetes probe pattern it mirrors: Spring Boot reference docs, Application Availability (nofollow). /admin/drainis a diagnostic-grade endpoint with no auth — see chapter 15 for what to do with it before shipping.
← 12. The AUTO_SERVER trap · Next: 14. The DDL lock window →