asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
66 lines
3.5 KiB
Markdown
66 lines
3.5 KiB
Markdown
# 12. The AUTO_SERVER trap
|
|
|
|
[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)
|
|
|
|
This module's first working draft pointed both replicas at the same H2 file with
|
|
`AUTO_SERVER=TRUE` — the mode most H2 tutorials show for "let two JVMs share one
|
|
embedded database file without a separate server process":
|
|
|
|
```
|
|
jdbc:h2:file:./data/expand-contract;AUTO_SERVER=TRUE
|
|
```
|
|
|
|
It works, right up until a rolling deploy restarts the *particular* replica that
|
|
happened to open the file first. `AUTO_SERVER` makes the first connecting process the
|
|
de facto database server for every other connection — internally, later connections
|
|
become `SessionRemote` clients of that first process, not independent embedded
|
|
sessions. Kill that one process — which a rolling deploy does, routinely, by design —
|
|
and every *other* replica's connection to the "embedded" database breaks with it. In
|
|
this module's own early runs, that showed up as `Table "CUSTOMERS" not found` on the
|
|
surviving replica, immediately after the first replica restarted, for a table that
|
|
had existed the entire time. It was single-point-of-failure architecture disguised as
|
|
an embedded database, and it was the single biggest source of load-generator errors
|
|
before it was found and fixed — cutting the error rate by roughly 90% on its own.
|
|
|
|
The fix, in
|
|
[`start-db-server.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/start-db-server.sh),
|
|
is to stop pretending the database is embedded at all and run it as what it actually
|
|
needs to be: its own standalone process that neither replica owns and neither
|
|
replica's lifecycle affects.
|
|
|
|
```bash
|
|
nohup java -cp "$(cat "$CP_FILE")" org.h2.tools.Server \
|
|
-tcp -tcpPort "$EC_DB_TCP_PORT" -baseDir "$EC_DB_BASE_DIR" -ifNotExists \
|
|
> "$EC_LOG_DIR/db-server.log" 2>&1 < /dev/null &
|
|
```
|
|
|
|
Both replicas — and
|
|
[`MigrationCli`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/migration/MigrationCli.java)
|
|
— connect to it the same way, as ordinary TCP clients:
|
|
|
|
```yaml
|
|
url: jdbc:h2:tcp://localhost:${EC_DB_TCP_PORT:9092}/${EC_DB_NAME:expand-contract}
|
|
```
|
|
|
|
`scripts/run-all.sh` starts this server first, before either replica, and stops it
|
|
last, after both — the one process in the entire sequence that is never restarted,
|
|
because it's standing in for what a real production database always is: a process
|
|
that outlives every deploy of every application that talks to it.
|
|
|
|
One flag worth calling out because it looks interchangeable and isn't:
|
|
`-tcpDaemon` marks the server thread as a daemon thread, which is for embedding an H2
|
|
server *inside* another long-running JVM that manages its own lifecycle — for a
|
|
standalone, always-on server process like this one, it made the process exit
|
|
immediately in a manual test, because there was no non-daemon thread left to keep the
|
|
JVM alive. Leave it off for a server meant to run on its own.
|
|
|
|
## Going deeper
|
|
|
|
- H2's own documentation on
|
|
[automatic mixed mode](https://www.h2database.com/html/features.html#auto_mixed_mode)
|
|
(nofollow) describes `AUTO_SERVER` for the single-application-process,
|
|
multiple-connections case it's actually designed for — not for two independent
|
|
application processes that need to survive each other's restarts.
|
|
|
|
[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)
|