asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity
Files
main
spring-boot-demo/db-migrations-expand-contract/docs/01-the-problem-and-the-plan.md
T
asmhatreandClaude Sonnet 5 e478eafda3 Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run 
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run
as an actual rolling deploy across two live replicas, with a load generator sending
continuous HTTP traffic through all four deploys (99.98% success, every residual
error traced to a root cause rather than left unexplained). Findings include a real
NOT NULL constraint trap in the expand migration, a backfill-window bug in the read
switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling
restart, the drain-before-SIGTERM fix needed to close a health-check gap during
graceful shutdown, and H2 silently discarding a concurrently committed INSERT during
an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an
H2-specific behavior rather than a property of the technique itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2026-09-16 19:22:48 +00:00

117 lines
6.0 KiB
Markdown
Raw Permalink Blame History

# 1. The problem and the plan
[Next: 2. The expand migration →](02-the-expand-migration.md)
## The problem
`customers` has a column called `email`. You want it called `email_address` — maybe
because a second `contact_email` table is coming and the naming needs to be
consistent, maybe because "email" collided with a reserved word in a tool you just
adopted. The reason doesn't matter. What matters is that this table has rows in it,
right now, in production, and something is reading and writing that column while you
work.
The naive fix is one migration:
```sql
ALTER TABLE customers RENAME COLUMN email TO email_address;
```
That statement is correct and it is also a production outage. The instant it commits,
every currently-running copy of your application — the ones you have not redeployed
yet, because a rolling deploy takes minutes, not zero seconds — starts issuing SQL
against a column that no longer exists. `INSERT INTO customers(name, email) VALUES
(?, ?)` becomes a 500 on every single request, on every replica that hasn't restarted
yet, until the rollout finishes. You have coupled a **schema change** to a **code
deploy**, and the two of them do not happen atomically across a fleet.
## The plan: expand, migrate, contract
Expand-contract (sometimes "parallel change") solves this by never letting the schema
and the code disagree about what's safe. Instead of one migration and one deploy, it's
four:
1. **Expand** — add the new column, alongside the old one. Nothing reads it yet.
Nothing that's running has to change.
2. **Migrate writes** — deploy code that writes to *both* columns. Every row created
or updated from this point on is consistent in both places.
3. **Migrate reads** — deploy code that reads from the new column instead of the old
one. This is a *separate* deploy from step 2, and the gap between them matters more
than it looks like it should — see [chapter 5](05-the-read-switch.md).
4. **Contract** — once every replica in the fleet is confirmed running the Stage 4
code from step 3, drop the old column. Nothing is reading or writing it anymore, so
dropping it is safe.
<svg viewBox="0 0 760 230" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Four deploys, each one changing exactly one thing">
<style>
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:13px;fill:#1a1a1a}
.hdr{font-weight:600;font-size:13px}
.box{fill:#eef4fc;stroke:#3b6fb0;stroke-width:1.4;rx:6}
.dim{fill:#f4f4f4;stroke:#999;stroke-width:1;rx:6}
.arrow{stroke:#555;stroke-width:1.6;marker-end:url(#a)}
</style>
<defs><marker id="a" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6 z" fill="#555"/></marker></defs>
<rect x="10" y="10" width="170" height="70" class="box"/>
<text x="20" y="30" class="hdr">1. EXPAND</text>
<text x="20" y="48">add email_address</text>
<text x="20" y="64">schema only, no deploy</text>
<rect x="200" y="10" width="170" height="70" class="box"/>
<text x="210" y="30" class="hdr">2. MIGRATE WRITES</text>
<text x="210" y="48">write both columns</text>
<text x="210" y="64">code deploy</text>
<rect x="390" y="10" width="170" height="70" class="box"/>
<text x="400" y="30" class="hdr">3. MIGRATE READS</text>
<text x="400" y="48">read email_address</text>
<text x="400" y="64">code deploy</text>
<rect x="580" y="10" width="170" height="70" class="box"/>
<text x="590" y="30" class="hdr">4. CONTRACT</text>
<text x="590" y="48">drop email</text>
<text x="590" y="64">schema, then cleanup</text>
<line x1="180" y1="45" x2="198" y2="45" class="arrow"/>
<line x1="370" y1="45" x2="388" y2="45" class="arrow"/>
<line x1="560" y1="45" x2="578" y2="45" class="arrow"/>
<rect x="10" y="110" width="740" height="100" class="dim"/>
<text x="20" y="132" class="hdr">Same database, the whole time</text>
<text x="20" y="152">customers.email [always present until step 4]</text>
<text x="20" y="172">customers.email_address [present from step 1 onward, populated from step 2 onward]</text>
<text x="20" y="194">Two adjacent stages serve real traffic against this table at once during every rollout above.</text>
</svg>
The diagram's bottom half is the fact the rest of this article keeps coming back to:
at every point during a rolling deploy, two adjacent stages are running against the
same table at the same time. Deploy 2's rollout has Stage 1 and Stage 2 replicas live
together for however long the rollout takes; Deploy 3's rollout has Stage 2 and Stage
3 together; Deploy 4a's has Stage 3 and Stage 4 together. Each of those overlaps is a
window where the "old" code and the "new" code both have to produce correct answers
against a schema neither one fully owns. [Chapter 8](08-the-rolling-window-proof.md)
is the test that checks every one of those six write/read combinations directly, and
the article's live load-generator run reproduces the same overlaps under real HTTP
traffic and real timing.
## What never changes
The four deploys change exactly one thing about how the *database* is used. They
change nothing about the *API*:
```java
public record Customer(long id, String name, String email) {
}
```
[`Customer.java`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/Customer.java)
never mentions `email_address` — a caller of this API cannot tell which stage
answered a given request just by looking at the response shape, and neither can the
article's own load generator. That's deliberate: expand-contract is a technique for
changing storage without changing the contract clients depend on.
## Going deeper
- The companion module for this article is `db-migrations-expand-contract` in
[`spring-boot-demo`](https://ankurm.com/git.app/asmhatre/spring-boot-demo) — every
chapter from here on links to a real file or a real captured transcript in it.
- Martin Fowler's [ParallelChange](https://martinfowler.com/bliki/ParallelChange.html)
is the canonical name and description of this pattern outside a specific database or
framework.
[Next: 2. The expand migration →](02-the-expand-migration.md)
View Git Blame Copy Permalink