asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
3.5 KiB
12. The AUTO_SERVER trap
← 11. The load generator · Next: 13. Graceful shutdown vs. kill -9 →
This module's first working draft pointed both replicas at the same H2 file with
AUTO_SERVER=TRUE — the mode most H2 tutorials show for "let two JVMs share one
embedded database file without a separate server process":
jdbc:h2:file:./data/expand-contract;AUTO_SERVER=TRUE
It works, right up until a rolling deploy restarts the particular replica that
happened to open the file first. AUTO_SERVER makes the first connecting process the
de facto database server for every other connection — internally, later connections
become SessionRemote clients of that first process, not independent embedded
sessions. Kill that one process — which a rolling deploy does, routinely, by design —
and every other replica's connection to the "embedded" database breaks with it. In
this module's own early runs, that showed up as Table "CUSTOMERS" not found on the
surviving replica, immediately after the first replica restarted, for a table that
had existed the entire time. It was single-point-of-failure architecture disguised as
an embedded database, and it was the single biggest source of load-generator errors
before it was found and fixed — cutting the error rate by roughly 90% on its own.
The fix, in
start-db-server.sh,
is to stop pretending the database is embedded at all and run it as what it actually
needs to be: its own standalone process that neither replica owns and neither
replica's lifecycle affects.
nohup java -cp "$(cat "$CP_FILE")" org.h2.tools.Server \
-tcp -tcpPort "$EC_DB_TCP_PORT" -baseDir "$EC_DB_BASE_DIR" -ifNotExists \
> "$EC_LOG_DIR/db-server.log" 2>&1 < /dev/null &
Both replicas — and
MigrationCli
— connect to it the same way, as ordinary TCP clients:
url: jdbc:h2:tcp://localhost:${EC_DB_TCP_PORT:9092}/${EC_DB_NAME:expand-contract}
scripts/run-all.sh starts this server first, before either replica, and stops it
last, after both — the one process in the entire sequence that is never restarted,
because it's standing in for what a real production database always is: a process
that outlives every deploy of every application that talks to it.
One flag worth calling out because it looks interchangeable and isn't:
-tcpDaemon marks the server thread as a daemon thread, which is for embedding an H2
server inside another long-running JVM that manages its own lifecycle — for a
standalone, always-on server process like this one, it made the process exit
immediately in a manual test, because there was no non-daemon thread left to keep the
JVM alive. Leave it off for a server meant to run on its own.
Going deeper
- H2's own documentation on
automatic mixed mode
(nofollow) describes
AUTO_SERVERfor the single-application-process, multiple-connections case it's actually designed for — not for two independent application processes that need to survive each other's restarts.
← 11. The load generator · Next: 13. Graceful shutdown vs. kill -9 →