Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,78 @@
|
||||
# 8. The rolling-window proof
|
||||
|
||||
[← 7. The backfill window bug](07-the-backfill-window-bug.md) · [Next: 9. The contract migration →](09-the-contract-migration.md)
|
||||
|
||||
Every rolling deploy in this sequence — Deploy 2, Deploy 3, Deploy 4a — has a window
|
||||
where two adjacent stages are serving real traffic against the same database at the
|
||||
same time. [`MixedStageRollingWindowTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java)
|
||||
is the property this whole article rests on, checked directly: it builds two
|
||||
`CustomerService` instances on adjacent stages sharing one database, and cross-checks
|
||||
every write/read direction across all three rollouts.
|
||||
|
||||
```
|
||||
-- Stage 1 writes, Stage 2 reads --
|
||||
Customer[id=1, name=Radia Perlman, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 1 reads --
|
||||
Customer[id=2, name=Barbara Liskov, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 3 reads --
|
||||
Customer[id=3, name=Shafi Goldwasser, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 2 reads --
|
||||
Customer[id=4, name=Frances Allen, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 4 reads --
|
||||
Customer[id=5, name=Adele Goldberg, [email protected]]
|
||||
|
||||
-- Stage 4 writes, Stage 3 reads --
|
||||
Customer[id=6, name=Karen Sparck Jones, [email protected]]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/08-mixed-stage-rolling-window.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/08-mixed-stage-rolling-window.txt).
|
||||
|
||||
Six pairs, six passing reads. If any one of them failed, the technique would not be
|
||||
zero-downtime for that rollout — it would just be a race against however long the
|
||||
rollout takes to finish, with correctness depending on luck rather than design.
|
||||
|
||||
<svg viewBox="0 0 720 200" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Three rolling deploys, each with an overlap window between two stages">
|
||||
<style>
|
||||
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:12px;fill:#1a1a1a}
|
||||
.lbl{font-weight:600}
|
||||
.s1{fill:#eef4fc;stroke:#3b6fb0}
|
||||
.s2{fill:#fdf0e6;stroke:#c0762c}
|
||||
.s3{fill:#eefaf0;stroke:#2f9e52}
|
||||
.s4{fill:#f6eefc;stroke:#7c3ba0}
|
||||
</style>
|
||||
<text x="10" y="20" class="lbl">Deploy 2 rollout</text>
|
||||
<rect x="10" y="30" width="220" height="24" class="s1"/><text x="18" y="46">Stage 1 (draining)</text>
|
||||
<rect x="240" y="30" width="220" height="24" class="s2"/><text x="248" y="46">Stage 2 (arriving)</text>
|
||||
<text x="470" y="46">← overlap: both true, both correct</text>
|
||||
|
||||
<text x="10" y="80" class="lbl">Deploy 3 rollout</text>
|
||||
<rect x="10" y="90" width="220" height="24" class="s2"/><text x="18" y="106">Stage 2 (draining)</text>
|
||||
<rect x="240" y="90" width="220" height="24" class="s3"/><text x="248" y="106">Stage 3 (arriving)</text>
|
||||
<text x="470" y="106">← overlap: both true, both correct</text>
|
||||
|
||||
<text x="10" y="140" class="lbl">Deploy 4a rollout</text>
|
||||
<rect x="10" y="150" width="220" height="24" class="s3"/><text x="18" y="166">Stage 3 (draining)</text>
|
||||
<rect x="240" y="150" width="220" height="24" class="s4"/><text x="248" y="166">Stage 4 (arriving)</text>
|
||||
<text x="470" y="166">← overlap: both true, both correct</text>
|
||||
</svg>
|
||||
|
||||
The diagram is the same shape three times because the guarantee is the same three
|
||||
times: whichever two stages are live together during a given rollout, a write from
|
||||
either one has to be readable correctly by the other. That's what the test above
|
||||
checks directly, and it's what the article's live 4-deploy run — a real load
|
||||
generator, hitting real HTTP endpoints, during a real rolling restart — is
|
||||
reproducing under actual timing pressure rather than a unit test's controlled
|
||||
ordering.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The live version of this proof, with two real replicas and continuous HTTP
|
||||
traffic: [chapter 11](11-the-load-generator.md) and the full run in
|
||||
[`docs/output/11-live-deploy-sequence.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/11-live-deploy-sequence.txt).
|
||||
|
||||
[← 7. The backfill window bug](07-the-backfill-window-bug.md) · [Next: 9. The contract migration →](09-the-contract-migration.md)
|
||||
Reference in New Issue
Block a user