Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2026-09-16 19:22:48 +00:00
parent 22c30d4a5b
commit e478eafda3
60 changed files with 3561 additions and 0 deletions
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Stops one replica by the PID file start-instance.sh wrote for it - never by matching
+# the process name or command line, which risks matching the wrong process (including
+# this very script's own shell). Usage: stop-instance.sh <port>
+#
+# Sends SIGTERM, not SIGKILL. "server.shutdown: graceful" in application.yml only
+# does anything on SIGTERM: it stops accepting new connections but lets in-flight
+# requests finish first. An earlier version used `kill -9` here, which bypasses that
+# entirely, and requests that were in flight the instant the process vanished showed
+# up in the load generator's summary as ConnectException/IOException - a real
+# artifact of skipping the drain step, not a defect in the migration itself. See
+# docs/13-graceful-shutdown-vs-kill-9.md.
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$DIR/env.sh"
+
+PORT="$1"
+PID_FILE="$EC_PID_DIR/$PORT.pid"
+if [ -f "$PID_FILE" ]; then
+    PID="$(cat "$PID_FILE")"
+    if kill -0 "$PID" 2>/dev/null; then
+        # Deregister BEFORE terminating: tell the load balancer to stop sending new
+        # traffic here, then give its health check a couple of poll cycles to notice,
+        # THEN stop the process. Skipping this drain window and going straight to
+        # SIGTERM is what produced the ConnectException bursts in an earlier run.
+        curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
+        sleep 1.5
+        kill -15 "$PID"
+        for i in $(seq 1 40); do
+            kill -0 "$PID" 2>/dev/null || break
+            sleep 0.25
+        done
+        if kill -0 "$PID" 2>/dev/null; then
+            echo "port $PORT (pid $PID) did not exit gracefully in 10s, sending SIGKILL" >&2
+            kill -9 "$PID"
+        fi
+        echo "stopped port $PORT (pid $PID)"
+    fi
+    rm -f "$PID_FILE"
+fi
+wait_down "$PORT" || echo "warning: port $PORT still answering after stop" >&2