asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
42 lines
1.9 KiB
Bash
Executable File
42 lines
1.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Stops one replica by the PID file start-instance.sh wrote for it - never by matching
|
|
# the process name or command line, which risks matching the wrong process (including
|
|
# this very script's own shell). Usage: stop-instance.sh <port>
|
|
#
|
|
# Sends SIGTERM, not SIGKILL. "server.shutdown: graceful" in application.yml only
|
|
# does anything on SIGTERM: it stops accepting new connections but lets in-flight
|
|
# requests finish first. An earlier version used `kill -9` here, which bypasses that
|
|
# entirely, and requests that were in flight the instant the process vanished showed
|
|
# up in the load generator's summary as ConnectException/IOException - a real
|
|
# artifact of skipping the drain step, not a defect in the migration itself. See
|
|
# docs/13-graceful-shutdown-vs-kill-9.md.
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
source "$DIR/env.sh"
|
|
|
|
PORT="$1"
|
|
PID_FILE="$EC_PID_DIR/$PORT.pid"
|
|
if [ -f "$PID_FILE" ]; then
|
|
PID="$(cat "$PID_FILE")"
|
|
if kill -0 "$PID" 2>/dev/null; then
|
|
# Deregister BEFORE terminating: tell the load balancer to stop sending new
|
|
# traffic here, then give its health check a couple of poll cycles to notice,
|
|
# THEN stop the process. Skipping this drain window and going straight to
|
|
# SIGTERM is what produced the ConnectException bursts in an earlier run.
|
|
curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
|
|
sleep 1.5
|
|
kill -15 "$PID"
|
|
for i in $(seq 1 40); do
|
|
kill -0 "$PID" 2>/dev/null || break
|
|
sleep 0.25
|
|
done
|
|
if kill -0 "$PID" 2>/dev/null; then
|
|
echo "port $PORT (pid $PID) did not exit gracefully in 10s, sending SIGKILL" >&2
|
|
kill -9 "$PID"
|
|
fi
|
|
echo "stopped port $PORT (pid $PID)"
|
|
fi
|
|
rm -f "$PID_FILE"
|
|
fi
|
|
wait_down "$PORT" || echo "warning: port $PORT still answering after stop" >&2
|