Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run
as an actual rolling deploy across two live replicas, with a load generator sending
continuous HTTP traffic through all four deploys (99.98% success, every residual
error traced to a root cause rather than left unexplained). Findings include a real
NOT NULL constraint trap in the expand migration, a backfill-window bug in the read
switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling
restart, the drain-before-SIGTERM fix needed to close a health-check gap during
graceful shutdown, and H2 silently discarding a concurrently committed INSERT during
an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an
H2-specific behavior rather than a property of the technique itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA

db-migrations-expand-contract/docs/12-the-auto-server-trap.md

@@ -0,0 +1,65 @@
+# 12. The AUTO_SERVER trap
+
+[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)
+
+This module's first working draft pointed both replicas at the same H2 file with
+`AUTO_SERVER=TRUE` — the mode most H2 tutorials show for "let two JVMs share one
+embedded database file without a separate server process":
+
+```
+jdbc:h2:file:./data/expand-contract;AUTO_SERVER=TRUE
+```
+
+It works, right up until a rolling deploy restarts the *particular* replica that
+happened to open the file first. `AUTO_SERVER` makes the first connecting process the
+de facto database server for every other connection — internally, later connections
+become `SessionRemote` clients of that first process, not independent embedded
+sessions. Kill that one process — which a rolling deploy does, routinely, by design —
+and every *other* replica's connection to the "embedded" database breaks with it. In
+this module's own early runs, that showed up as `Table "CUSTOMERS" not found` on the
+surviving replica, immediately after the first replica restarted, for a table that
+had existed the entire time. It was single-point-of-failure architecture disguised as
+an embedded database, and it was the single biggest source of load-generator errors
+before it was found and fixed — cutting the error rate by roughly 90% on its own.
+
+The fix, in
+[`start-db-server.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/start-db-server.sh),
+is to stop pretending the database is embedded at all and run it as what it actually
+needs to be: its own standalone process that neither replica owns and neither
+replica's lifecycle affects.
+
+```bash
+nohup java -cp "$(cat "$CP_FILE")" org.h2.tools.Server \
+    -tcp -tcpPort "$EC_DB_TCP_PORT" -baseDir "$EC_DB_BASE_DIR" -ifNotExists \
+    > "$EC_LOG_DIR/db-server.log" 2>&1 < /dev/null &
+```
+
+Both replicas — and
+[`MigrationCli`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/migration/MigrationCli.java)
+— connect to it the same way, as ordinary TCP clients:
+
+```yaml
+url: jdbc:h2:tcp://localhost:${EC_DB_TCP_PORT:9092}/${EC_DB_NAME:expand-contract}
+```
+
+`scripts/run-all.sh` starts this server first, before either replica, and stops it
+last, after both — the one process in the entire sequence that is never restarted,
+because it's standing in for what a real production database always is: a process
+that outlives every deploy of every application that talks to it.
+
+One flag worth calling out because it looks interchangeable and isn't:
+`-tcpDaemon` marks the server thread as a daemon thread, which is for embedding an H2
+server *inside* another long-running JVM that manages its own lifecycle — for a
+standalone, always-on server process like this one, it made the process exit
+immediately in a manual test, because there was no non-daemon thread left to keep the
+JVM alive. Leave it off for a server meant to run on its own.
+
+## Going deeper
+
+- H2's own documentation on
+  [automatic mixed mode](https://www.h2database.com/html/features.html#auto_mixed_mode)
+  (nofollow) describes `AUTO_SERVER` for the single-application-process,
+  multiple-connections case it's actually designed for — not for two independent
+  application processes that need to survive each other's restarts.
+
+[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)