Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,36 @@
|
||||
=========================================================================
|
||||
Deploy 1 (EXPAND): additive column + backfill, Stage 1 code untouched
|
||||
=========================================================================
|
||||
captured: 2026-09-16T19:20:56.676919361Z
|
||||
|
||||
|
||||
-- schema before Deploy 1 --
|
||||
COLUMN_NAME
|
||||
-----------
|
||||
ID
|
||||
NAME
|
||||
EMAIL
|
||||
CREATED_AT
|
||||
(4 rows)
|
||||
|
||||
-- schema after Deploy 1 (email_address added) --
|
||||
COLUMN_NAME
|
||||
-------------
|
||||
ID
|
||||
NAME
|
||||
EMAIL
|
||||
CREATED_AT
|
||||
EMAIL_ADDRESS
|
||||
(5 rows)
|
||||
|
||||
-- Ada's row was backfilled by the migration itself --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
-------------+------------------+-----------------
|
||||
Ada Lovelace | [email protected] | [email protected]
|
||||
(1 row)
|
||||
|
||||
-- Stage 1's original INSERT still works, unmodified, after the migration --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
-------------+--------------------+--------------
|
||||
Grace Hopper | [email protected] | NULL
|
||||
(1 row)
|
||||
@@ -0,0 +1,17 @@
|
||||
==================================================================
|
||||
Deploy 2 (MIGRATE WRITES): Stage 2 writes land in both columns
|
||||
==================================================================
|
||||
captured: 2026-09-16T19:21:04.333919632Z
|
||||
|
||||
|
||||
-- after create() --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-----------------------+----------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
(1 row)
|
||||
|
||||
-- after updateEmail() - the old value is gone from BOTH columns, not just one --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-------------------------+------------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
(1 row)
|
||||
@@ -0,0 +1,14 @@
|
||||
==========================================================================
|
||||
The NOT NULL trap: expand without relaxing the old column's constraint
|
||||
==========================================================================
|
||||
captured: 2026-09-16T19:21:04.662650854Z
|
||||
|
||||
|
||||
-- Stage 4 create() against the NAIVE migration (no DROP NOT NULL) --
|
||||
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO customers(name, email_address) VALUES (?, ?)]; NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
root cause: org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
|
||||
-- Stage 4 create() against the SHIPPED V2 migration (DROP NOT NULL included) --
|
||||
Customer[id=1, name=On Time, [email protected]]
|
||||
@@ -0,0 +1,14 @@
|
||||
==============================================================================
|
||||
The backfill window: a Stage 1 write after Deploy 1, read by a naive Stage 3
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:56.276837493Z
|
||||
|
||||
|
||||
-- the row a lingering Stage 1 instance just wrote --
|
||||
[{NAME=Katherine Johnson, [email protected], EMAIL_ADDRESS=null}]
|
||||
|
||||
-- a NAIVE Stage 3 read (email_address alone) - the bug --
|
||||
naive Stage 3 email column value: null
|
||||
|
||||
-- the SHIPPED Stage 3 read (CustomerService, COALESCE) - the fix --
|
||||
CustomerService (stage 3) result: Optional[Customer[id=1, name=Katherine Johnson, [email protected]]]
|
||||
@@ -0,0 +1,23 @@
|
||||
====================================================================
|
||||
Cross-stage consistency during each of the three rolling deploys
|
||||
====================================================================
|
||||
captured: 2026-09-16T19:21:05.241547756Z
|
||||
|
||||
|
||||
-- Stage 1 writes, Stage 2 reads --
|
||||
Customer[id=1, name=Radia Perlman, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 1 reads --
|
||||
Customer[id=2, name=Barbara Liskov, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 3 reads --
|
||||
Customer[id=3, name=Shafi Goldwasser, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 2 reads --
|
||||
Customer[id=4, name=Frances Allen, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 4 reads --
|
||||
Customer[id=5, name=Adele Goldberg, [email protected]]
|
||||
|
||||
-- Stage 4 writes, Stage 3 reads --
|
||||
Customer[id=6, name=Karen Sparck Jones, [email protected]]
|
||||
@@ -0,0 +1,11 @@
|
||||
==============================================================================
|
||||
Deploy 4 (CONTRACT): Stage 4 after the drop, and what breaks if you drop too soon
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:57.618780340Z
|
||||
|
||||
|
||||
-- Stage 4 read, after V3 dropped the email column --
|
||||
Customer[id=1, name=Annie Easley, [email protected]]
|
||||
|
||||
-- Stage 4 create + read, entirely after the drop --
|
||||
Customer[id=2, name=Mary Allen Wilkes, [email protected]]
|
||||
@@ -0,0 +1,9 @@
|
||||
==============================================================================
|
||||
What a lingering Stage 1 instance sees if the drop runs before it is retired
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:58.517481426Z
|
||||
|
||||
Stage 1 create() after V3 dropped "email": org.springframework.jdbc.BadSqlGrammarException
|
||||
message: PreparedStatementCallback; bad SQL grammar [INSERT INTO customers(name, email) VALUES (?, ?)]
|
||||
root cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "EMAIL" not found; SQL statement:
|
||||
INSERT INTO customers(name, email) VALUES (?, ?) [42122-240]
|
||||
@@ -0,0 +1,103 @@
|
||||
=====================================================================
|
||||
Zero-downtime expand-contract: live 4-deploy sequence
|
||||
=====================================================================
|
||||
19:00:53 captured: 2026-09-16T19:00:53Z
|
||||
19:00:53 Starting the database as its own standalone process (not owned by either replica)
|
||||
H2 TCP server up on port 9092 (pid 10861), baseDir /tmp/ec-demo/db
|
||||
19:00:53 Deploy 0: create schema (V1), start two Stage 1 replicas
|
||||
=== Before migrate (target=1) ===
|
||||
00:30:54.634 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:54.702 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:30:54.750 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
|
||||
1 create customer PENDING
|
||||
2 add email address column ABOVE_TARGET
|
||||
3 drop email column ABOVE_TARGET
|
||||
00:30:54.850 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:54.863 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:30:54.874 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
|
||||
00:30:54.877 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.007s)
|
||||
00:30:54.888 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Creating Schema History table "PUBLIC"."flyway_schema_history" ...
|
||||
00:30:54.940 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": << Empty Schema >>
|
||||
00:30:54.956 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "1 - create customer"
|
||||
00:30:55.018 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v1 (execution time 00:00.014s)
|
||||
=== After migrate ===
|
||||
00:30:55.055 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:55.063 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column ABOVE_TARGET
|
||||
3 drop email column ABOVE_TARGET
|
||||
Migrations executed: 1, target schema version: 1, success: true
|
||||
started stage 1 on port 8081 (pid 10946)
|
||||
port 8081 healthy
|
||||
started stage 1 on port 8082 (pid 11011)
|
||||
port 8082 healthy
|
||||
19:01:02 load generator running against both replicas
|
||||
19:01:12 baseline soak complete (10s, both replicas on Stage 1)
|
||||
19:01:12 Deploy 1 (EXPAND): migrating to V2 live - zero app restarts
|
||||
=== Before migrate (target=2) ===
|
||||
00:31:13.484 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:13.607 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column PENDING
|
||||
3 drop email column ABOVE_TARGET
|
||||
00:31:13.782 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:13.801 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:31:13.817 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.008s)
|
||||
00:31:13.842 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": 1
|
||||
00:31:13.903 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "2 - add email address column"
|
||||
00:31:14.097 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v2 (execution time 00:00.118s)
|
||||
=== After migrate ===
|
||||
00:31:14.136 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:14.174 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column ABOVE_TARGET
|
||||
Migrations executed: 1, target schema version: 2, success: true
|
||||
19:01:19 Deploy 2 (MIGRATE WRITES): rolling restart to Stage 2, replica A first
|
||||
stopped port 8081 (pid 10946)
|
||||
started stage 2 on port 8081 (pid 11226)
|
||||
port 8081 healthy
|
||||
19:01:29 Deploy 2: replica B
|
||||
stopped port 8082 (pid 11011)
|
||||
started stage 2 on port 8082 (pid 11338)
|
||||
port 8082 healthy
|
||||
19:01:45 Deploy 3 (MIGRATE READS): rolling restart to Stage 3, replica A first
|
||||
stopped port 8081 (pid 11226)
|
||||
started stage 3 on port 8081 (pid 11456)
|
||||
port 8081 healthy
|
||||
19:01:55 Deploy 3: replica B
|
||||
stopped port 8082 (pid 11338)
|
||||
started stage 3 on port 8082 (pid 11567)
|
||||
port 8082 healthy
|
||||
19:02:11 Deploy 4a (CONTRACT code): rolling restart to Stage 4, replica A first
|
||||
stopped port 8081 (pid 11456)
|
||||
started stage 4 on port 8081 (pid 11679)
|
||||
port 8081 healthy
|
||||
19:02:20 Deploy 4a: replica B
|
||||
stopped port 8082 (pid 11567)
|
||||
started stage 4 on port 8082 (pid 11784)
|
||||
port 8082 healthy
|
||||
19:02:35 Deploy 4b (CONTRACT schema): migrating to V3 live - drops "email", zero app restarts
|
||||
=== Before migrate (target=latest) ===
|
||||
00:32:36.510 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.574 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column PENDING
|
||||
00:32:36.669 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.680 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:32:36.694 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.008s)
|
||||
00:32:36.707 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": 2
|
||||
00:32:36.726 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "3 - drop email column"
|
||||
00:32:36.795 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v3 (execution time 00:00.041s)
|
||||
=== After migrate ===
|
||||
00:32:36.811 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.829 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column SUCCESS
|
||||
Migrations executed: 1, target schema version: 3, success: true
|
||||
19:02:46 final soak complete
|
||||
19:02:46 waiting for the load generator to finish its run...
|
||||
19:03:04 load generator finished
|
||||
19:03:11 === deploy sequence complete ===
|
||||
@@ -0,0 +1,21 @@
|
||||
Load generator summary
|
||||
=======================
|
||||
Total requests: 30911
|
||||
Successful: 30905
|
||||
Errors: 6
|
||||
|
||||
By phase:
|
||||
04a-stage4-soak ok=2130 error=0
|
||||
04b-contract-migration ok=336 error=0
|
||||
04a-deploy-stage4-rollout ok=4147 error=0
|
||||
03-stage3-soak ok=2116 error=0
|
||||
03-deploy-stage3-rollout ok=4645 error=0
|
||||
00-baseline-soak ok=1893 error=0
|
||||
05-final-soak ok=7118 error=4
|
||||
- read-http-404 2
|
||||
- update-http-404 2
|
||||
02-stage2-soak ok=2068 error=0
|
||||
01-expand-migration ok=1773 error=1
|
||||
- read-http-404 1
|
||||
02-deploy-stage2-rollout ok=4679 error=1
|
||||
- update-http-404 1
|
||||
@@ -0,0 +1,18 @@
|
||||
|
||||
-- after Deploy 0 (Stage 1 / Stage 1) --
|
||||
{"appStage":1,"columns":["ID","NAME","EMAIL","CREATED_AT"],"rowCount":0,"rowsWithEmail":0}
|
||||
|
||||
-- after Deploy 1 (schema expanded, both replicas still Stage 1) --
|
||||
{"appStage":1,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":1212,"rowsWithEmail":1212,"rowsWithEmailAddress":1184}
|
||||
|
||||
-- after Deploy 2 (both replicas Stage 2, dual-write live) --
|
||||
{"appStage":2,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":4218,"rowsWithEmail":4218,"rowsWithEmailAddress":2561}
|
||||
|
||||
-- after Deploy 3 (both replicas Stage 3, reading email_address) --
|
||||
{"appStage":3,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":7547,"rowsWithEmail":7547,"rowsWithEmailAddress":6115}
|
||||
|
||||
-- after Deploy 4a (both replicas Stage 4, email column still present but unused) --
|
||||
{"appStage":4,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":10642,"rowsWithEmail":9653,"rowsWithEmailAddress":9212}
|
||||
|
||||
-- after Deploy 4b (email column dropped) --
|
||||
{"appStage":4,"columns":["ID","NAME","CREATED_AT","EMAIL_ADDRESS"],"rowCount":11806,"rowsWithEmailAddress":10374}
|
||||
@@ -0,0 +1,14 @@
|
||||
==============================================================================
|
||||
The failure the retry cannot catch: a committed INSERT that ALTER TABLE loses silently
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:58.574465432Z
|
||||
|
||||
attempts needed to reproduce the race: 2 of 20
|
||||
customer creates that returned a generated id with no error: 50
|
||||
customer creates that got the already-documented, already-fixed DDL-collision error: 1
|
||||
of the ids that came back with no error, missing from the table once V2 finished: 8
|
||||
example missing ids: [41, 42, 43, 44, 45]
|
||||
|
||||
This is why the retry in CustomerService cannot be the whole fix: these inserts
|
||||
never threw anything to retry. The row was committed, then discarded when the
|
||||
ADD COLUMN rebuild swapped in a new table that had already been scanned.
|
||||
Reference in New Issue
Block a user