Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,116 @@
|
||||
# 1. The problem and the plan
|
||||
|
||||
[Next: 2. The expand migration →](02-the-expand-migration.md)
|
||||
|
||||
## The problem
|
||||
|
||||
`customers` has a column called `email`. You want it called `email_address` — maybe
|
||||
because a second `contact_email` table is coming and the naming needs to be
|
||||
consistent, maybe because "email" collided with a reserved word in a tool you just
|
||||
adopted. The reason doesn't matter. What matters is that this table has rows in it,
|
||||
right now, in production, and something is reading and writing that column while you
|
||||
work.
|
||||
|
||||
The naive fix is one migration:
|
||||
|
||||
```sql
|
||||
ALTER TABLE customers RENAME COLUMN email TO email_address;
|
||||
```
|
||||
|
||||
That statement is correct and it is also a production outage. The instant it commits,
|
||||
every currently-running copy of your application — the ones you have not redeployed
|
||||
yet, because a rolling deploy takes minutes, not zero seconds — starts issuing SQL
|
||||
against a column that no longer exists. `INSERT INTO customers(name, email) VALUES
|
||||
(?, ?)` becomes a 500 on every single request, on every replica that hasn't restarted
|
||||
yet, until the rollout finishes. You have coupled a **schema change** to a **code
|
||||
deploy**, and the two of them do not happen atomically across a fleet.
|
||||
|
||||
## The plan: expand, migrate, contract
|
||||
|
||||
Expand-contract (sometimes "parallel change") solves this by never letting the schema
|
||||
and the code disagree about what's safe. Instead of one migration and one deploy, it's
|
||||
four:
|
||||
|
||||
1. **Expand** — add the new column, alongside the old one. Nothing reads it yet.
|
||||
Nothing that's running has to change.
|
||||
2. **Migrate writes** — deploy code that writes to *both* columns. Every row created
|
||||
or updated from this point on is consistent in both places.
|
||||
3. **Migrate reads** — deploy code that reads from the new column instead of the old
|
||||
one. This is a *separate* deploy from step 2, and the gap between them matters more
|
||||
than it looks like it should — see [chapter 5](05-the-read-switch.md).
|
||||
4. **Contract** — once every replica in the fleet is confirmed running the Stage 4
|
||||
code from step 3, drop the old column. Nothing is reading or writing it anymore, so
|
||||
dropping it is safe.
|
||||
|
||||
<svg viewBox="0 0 760 230" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Four deploys, each one changing exactly one thing">
|
||||
<style>
|
||||
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:13px;fill:#1a1a1a}
|
||||
.hdr{font-weight:600;font-size:13px}
|
||||
.box{fill:#eef4fc;stroke:#3b6fb0;stroke-width:1.4;rx:6}
|
||||
.dim{fill:#f4f4f4;stroke:#999;stroke-width:1;rx:6}
|
||||
.arrow{stroke:#555;stroke-width:1.6;marker-end:url(#a)}
|
||||
</style>
|
||||
<defs><marker id="a" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6 z" fill="#555"/></marker></defs>
|
||||
<rect x="10" y="10" width="170" height="70" class="box"/>
|
||||
<text x="20" y="30" class="hdr">1. EXPAND</text>
|
||||
<text x="20" y="48">add email_address</text>
|
||||
<text x="20" y="64">schema only, no deploy</text>
|
||||
<rect x="200" y="10" width="170" height="70" class="box"/>
|
||||
<text x="210" y="30" class="hdr">2. MIGRATE WRITES</text>
|
||||
<text x="210" y="48">write both columns</text>
|
||||
<text x="210" y="64">code deploy</text>
|
||||
<rect x="390" y="10" width="170" height="70" class="box"/>
|
||||
<text x="400" y="30" class="hdr">3. MIGRATE READS</text>
|
||||
<text x="400" y="48">read email_address</text>
|
||||
<text x="400" y="64">code deploy</text>
|
||||
<rect x="580" y="10" width="170" height="70" class="box"/>
|
||||
<text x="590" y="30" class="hdr">4. CONTRACT</text>
|
||||
<text x="590" y="48">drop email</text>
|
||||
<text x="590" y="64">schema, then cleanup</text>
|
||||
<line x1="180" y1="45" x2="198" y2="45" class="arrow"/>
|
||||
<line x1="370" y1="45" x2="388" y2="45" class="arrow"/>
|
||||
<line x1="560" y1="45" x2="578" y2="45" class="arrow"/>
|
||||
<rect x="10" y="110" width="740" height="100" class="dim"/>
|
||||
<text x="20" y="132" class="hdr">Same database, the whole time</text>
|
||||
<text x="20" y="152">customers.email [always present until step 4]</text>
|
||||
<text x="20" y="172">customers.email_address [present from step 1 onward, populated from step 2 onward]</text>
|
||||
<text x="20" y="194">Two adjacent stages serve real traffic against this table at once during every rollout above.</text>
|
||||
</svg>
|
||||
|
||||
The diagram's bottom half is the fact the rest of this article keeps coming back to:
|
||||
at every point during a rolling deploy, two adjacent stages are running against the
|
||||
same table at the same time. Deploy 2's rollout has Stage 1 and Stage 2 replicas live
|
||||
together for however long the rollout takes; Deploy 3's rollout has Stage 2 and Stage
|
||||
3 together; Deploy 4a's has Stage 3 and Stage 4 together. Each of those overlaps is a
|
||||
window where the "old" code and the "new" code both have to produce correct answers
|
||||
against a schema neither one fully owns. [Chapter 8](08-the-rolling-window-proof.md)
|
||||
is the test that checks every one of those six write/read combinations directly, and
|
||||
the article's live load-generator run reproduces the same overlaps under real HTTP
|
||||
traffic and real timing.
|
||||
|
||||
## What never changes
|
||||
|
||||
The four deploys change exactly one thing about how the *database* is used. They
|
||||
change nothing about the *API*:
|
||||
|
||||
```java
|
||||
public record Customer(long id, String name, String email) {
|
||||
}
|
||||
```
|
||||
|
||||
[`Customer.java`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/Customer.java)
|
||||
never mentions `email_address` — a caller of this API cannot tell which stage
|
||||
answered a given request just by looking at the response shape, and neither can the
|
||||
article's own load generator. That's deliberate: expand-contract is a technique for
|
||||
changing storage without changing the contract clients depend on.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The companion module for this article is `db-migrations-expand-contract` in
|
||||
[`spring-boot-demo`](https://ankurm.com/git.app/asmhatre/spring-boot-demo) — every
|
||||
chapter from here on links to a real file or a real captured transcript in it.
|
||||
- Martin Fowler's [ParallelChange](https://martinfowler.com/bliki/ParallelChange.html)
|
||||
is the canonical name and description of this pattern outside a specific database or
|
||||
framework.
|
||||
|
||||
[Next: 2. The expand migration →](02-the-expand-migration.md)
|
||||
@@ -0,0 +1,65 @@
|
||||
# 2. The expand migration
|
||||
|
||||
[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)
|
||||
|
||||
Deploy 1 is schema-only. No application code changes, no replica restarts. The
|
||||
migration is
|
||||
[`V2__add_email_address_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V2__add_email_address_column.sql):
|
||||
|
||||
```sql
|
||||
ALTER TABLE customers ADD COLUMN email_address VARCHAR(320);
|
||||
UPDATE customers SET email_address = email WHERE email_address IS NULL;
|
||||
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
|
||||
```
|
||||
|
||||
Three statements, three separate jobs:
|
||||
|
||||
- **Add the column, nullable.** Nullable is what makes it additive: no existing
|
||||
`INSERT` statement mentions `email_address`, so none of them break. A column added
|
||||
`NOT NULL` with no default would fail immediately for any code still running the old
|
||||
`INSERT INTO customers(name, email) VALUES (?, ?)`.
|
||||
- **Backfill it** for every row that already exists. This is a one-time pass over
|
||||
whatever data predates Deploy 1.
|
||||
- **Relax the old column's constraint.** This third line is the one that's easy to
|
||||
skip, and skipping it is a real, reproducible outage — see
|
||||
[chapter 6](06-the-not-null-trap.md).
|
||||
|
||||
[`ExpandMigrationBackwardCompatibleTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java)
|
||||
checks the two things Deploy 1 promises: every pre-existing row gets backfilled, and
|
||||
Stage 1's original `INSERT` — unmodified, unaware `email_address` exists — still
|
||||
works after the migration runs:
|
||||
|
||||
```
|
||||
-- schema before Deploy 1 --
|
||||
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...]
|
||||
|
||||
-- schema after Deploy 1 (email_address added) --
|
||||
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...][COLUMN_NAME=EMAIL_ADDRESS, ...]
|
||||
|
||||
-- Ada's row was backfilled by the migration itself --
|
||||
[NAME=Ada Lovelace, [email protected], [email protected]]
|
||||
|
||||
-- Stage 1's original INSERT still works, unmodified, after the migration --
|
||||
[NAME=Grace Hopper, [email protected], EMAIL_ADDRESS=NULL]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/02-expand-backward-compatible.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/02-expand-backward-compatible.txt).
|
||||
|
||||
Grace Hopper's row is the important one: `email_address` is `NULL` for it, because
|
||||
Stage 1 never wrote to it, and that's *correct* — Deploy 1 hasn't shipped any code
|
||||
that would. That gap is exactly what Deploy 2 exists to close, and it reopens itself
|
||||
on a smaller scale during Deploy 2's own rollout — see
|
||||
[chapter 7](07-the-backfill-window-bug.md).
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The migration runs against the live database with no application restart —
|
||||
[chapter 3](03-why-migrations-run-outside-the-app.md) covers how and why that's a
|
||||
separate mechanism from the app's own deploy.
|
||||
- H2's specific behavior for `ALTER TABLE ADD COLUMN` under concurrent traffic —
|
||||
not just "is it additive" but "is it safe to run while inserts are in flight" — is
|
||||
covered in [chapter 14](14-the-ddl-lock-window.md), and it's the most surprising
|
||||
finding in this whole module.
|
||||
|
||||
[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)
|
||||
@@ -0,0 +1,69 @@
|
||||
# 3. Why migrations run outside the app
|
||||
|
||||
[← 2. The expand migration](02-the-expand-migration.md) · [Next: 4. The dual write →](04-the-dual-write.md)
|
||||
|
||||
Every other module in this repository lets Spring Boot run Flyway on startup —
|
||||
`spring.flyway.enabled=true`, migrate-on-boot, the default most tutorials show. This
|
||||
module turns that off:
|
||||
|
||||
```yaml
|
||||
flyway:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
from
|
||||
[`application.yml`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/application.yml),
|
||||
and instead ships a second, standalone entry point:
|
||||
[`MigrationCli`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/migration/MigrationCli.java):
|
||||
|
||||
```java
|
||||
Flyway flyway = Flyway.configure()
|
||||
.dataSource(url, "sa", "")
|
||||
.locations("classpath:db/migration")
|
||||
.target(target)
|
||||
.load();
|
||||
flyway.migrate();
|
||||
```
|
||||
|
||||
invoked by [`scripts/migrate.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/migrate.sh):
|
||||
|
||||
```bash
|
||||
java -cp "$MODULE_DIR/target/classes:$(cat "$CP_FILE")" \
|
||||
com.ankurm.expandcontract.migration.MigrationCli --target="$TARGET"
|
||||
```
|
||||
|
||||
Why bother, when "migrate on boot" is one line of config? Because "migrate on boot"
|
||||
quietly welds a schema change to an application restart, and expand-contract's whole
|
||||
argument is that those two things need to be independently controllable events:
|
||||
|
||||
- Deploy 1 (expand) and Deploy 4b (contract) run a migration with **zero** app
|
||||
restarts — every replica keeps serving traffic on its current code the entire time
|
||||
the `ALTER TABLE` executes. `scripts/run-all.sh` calls `migrate.sh` directly for
|
||||
both of these, with no `stop-instance.sh` / `start-instance.sh` anywhere nearby.
|
||||
- Deploys 2, 3, and 4a are **pure code deploys** — a rolling restart with `--target`
|
||||
fixed at whatever the schema already is. No new SQL runs.
|
||||
|
||||
If Flyway ran on every boot, a canary replica restarting for an unrelated reason (an
|
||||
OOM, a node reschedule, a routine redeploy of a config value) would silently re-run
|
||||
whatever migrations hadn't executed yet, at a moment nobody chose. Running Flyway from
|
||||
its own process, invoked deliberately by the deploy pipeline (or by hand, as this
|
||||
module's scripts do), means a schema change happens exactly once, at exactly the
|
||||
moment someone decided it should — the same discipline a real CI/CD "run migrations"
|
||||
job step gives you, kept intact here even though this whole sequence runs on one
|
||||
sandbox.
|
||||
|
||||
The `--target` flag is what lets `migrate.sh 2` mean "get the schema to exactly V2,
|
||||
no further" — the same `spring.flyway.target` property
|
||||
[`TestSupport.migrateTo`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/TestSupport.java)
|
||||
uses to put a test database at "however far Deploy N has gotten" before exercising
|
||||
`CustomerService` against it.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- `MigrationCli` connects to the same standalone H2 TCP server the app replicas do —
|
||||
[chapter 12](12-the-auto-server-trap.md) covers why that database is its own
|
||||
process rather than something either replica owns.
|
||||
- Flyway's own migrate-on-startup vs. separate-migration-step tradeoff is discussed in
|
||||
[Flyway's documentation on migrations](https://documentation.red-gate.com/fd/migrations-184127470.html) (nofollow).
|
||||
|
||||
[← 2. The expand migration](02-the-expand-migration.md) · [Next: 4. The dual write →](04-the-dual-write.md)
|
||||
@@ -0,0 +1,61 @@
|
||||
# 4. The dual write
|
||||
|
||||
[← 3. Why migrations run outside the app](03-why-migrations-run-outside-the-app.md) · [Next: 5. The read switch →](05-the-read-switch.md)
|
||||
|
||||
Deploy 2 is the first code deploy in the sequence, and the only thing it changes is
|
||||
what `create()` and `updateEmail()` write. Stage 2's SQL in
|
||||
[`CustomerService`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/CustomerService.java)
|
||||
writes to both columns:
|
||||
|
||||
```java
|
||||
case 2, 3 -> jdbc.sql("INSERT INTO customers(name, email, email_address) VALUES (?, ?, ?)")
|
||||
.param(name).param(email).param(email)
|
||||
.update(keyHolder, "id");
|
||||
```
|
||||
|
||||
```java
|
||||
case 2, 3 -> jdbc.sql("UPDATE customers SET email = ?, email_address = ? WHERE id = ?")
|
||||
.param(newEmail).param(newEmail).param(id).update();
|
||||
```
|
||||
|
||||
Stage 2 and Stage 3 share this write path — the only difference between them is what
|
||||
they *read*, covered in [chapter 5](05-the-read-switch.md). That's deliberate: writes
|
||||
have to stay dual for two whole deploys (2 and 3) so that by the time Deploy 4
|
||||
arrives, every row in the table — regardless of which stage wrote it last — is
|
||||
guaranteed to have both columns populated identically.
|
||||
|
||||
[`DualWriteConsistencyTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/DualWriteConsistencyTest.java)
|
||||
confirms both halves of that promise — a fresh create lands in both columns, and an
|
||||
update *replaces* the value in both, not just one:
|
||||
|
||||
```
|
||||
-- after create() --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-----------------------+----------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
|
||||
-- after updateEmail() - the old value is gone from BOTH columns, not just one --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-------------------------+------------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/04-dual-write-consistency.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/04-dual-write-consistency.txt).
|
||||
|
||||
The second half of that test matters more than it looks. A dual write that only
|
||||
inserts into both columns but updates only one is a much more common bug than it
|
||||
sounds — the update path is usually written later, by someone who's already stopped
|
||||
thinking about `email_address` because the create path "already handles the new
|
||||
column".
|
||||
|
||||
## Going deeper
|
||||
|
||||
- Deploy 2's own rollout window — where some replicas are still Stage 1 while others
|
||||
are already Stage 2 — is exactly the gap [chapter 7](07-the-backfill-window-bug.md)
|
||||
is about: what a lingering Stage 1 write during *this* rollout means for the read
|
||||
switch that comes next.
|
||||
- The full cross-stage read/write matrix, including this deploy's pair, is proven
|
||||
directly in [chapter 8](08-the-rolling-window-proof.md).
|
||||
|
||||
[← 3. Why migrations run outside the app](03-why-migrations-run-outside-the-app.md) · [Next: 5. The read switch →](05-the-read-switch.md)
|
||||
@@ -0,0 +1,41 @@
|
||||
# 5. The read switch
|
||||
|
||||
[← 4. The dual write](04-the-dual-write.md) · [Next: 6. The NOT NULL trap →](06-the-not-null-trap.md)
|
||||
|
||||
Deploy 3 changes exactly one thing from Deploy 2: where reads come from. In
|
||||
[`CustomerService.findById`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/CustomerService.java):
|
||||
|
||||
```java
|
||||
String sql = switch (stage) {
|
||||
case 1, 2 -> "SELECT id, name, email AS email FROM customers WHERE id = ?";
|
||||
case 3 -> "SELECT id, name, COALESCE(email_address, email) AS email FROM customers WHERE id = ?";
|
||||
case 4 -> "SELECT id, name, email_address AS email FROM customers WHERE id = ?";
|
||||
default -> throw new IllegalStateException();
|
||||
};
|
||||
```
|
||||
|
||||
Stage 1 and Stage 2 read the same column: only *writes* changed between them. Stage 3
|
||||
is the read switch, and it's a separate deploy from Stage 2 for a reason that isn't
|
||||
obvious until you say it out loud: **the write switch and the read switch cannot be
|
||||
the same deploy**, because a rolling deploy is not instantaneous. Deploy 2's own
|
||||
rollout has a window — anywhere from seconds to minutes, depending on fleet size —
|
||||
where some replicas are still Stage 1, writing only `email`. If Deploy 3's read
|
||||
switch were bundled into that same deploy, a Stage 3 replica reading
|
||||
`email_address` alone would return `NULL` for every row a still-live Stage 1
|
||||
replica had just written. That's not a hypothetical: it's
|
||||
[chapter 7](07-the-backfill-window-bug.md), reproduced and fixed.
|
||||
|
||||
The fix already visible above is `COALESCE(email_address, email)` rather than
|
||||
`email_address` alone — Stage 3 falls back to the old column for exactly the rows
|
||||
that predate full dual-write coverage. Stage 4 doesn't need the fallback anymore: by
|
||||
the time Deploy 4 starts, two full deploys' worth of dual-writing (Stage 2 and Stage
|
||||
3, run back to back) have guaranteed every row has both columns populated.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The bug this switch would reintroduce if it were merged into Deploy 2, captured
|
||||
live: [chapter 7](07-the-backfill-window-bug.md).
|
||||
- All six write/read direction pairs across the three rollouts, proven directly:
|
||||
[chapter 8](08-the-rolling-window-proof.md).
|
||||
|
||||
[← 4. The dual write](04-the-dual-write.md) · [Next: 6. The NOT NULL trap →](06-the-not-null-trap.md)
|
||||
@@ -0,0 +1,66 @@
|
||||
# 6. The NOT NULL trap
|
||||
|
||||
[← 5. The read switch](05-the-read-switch.md) · [Next: 7. The backfill window bug →](07-the-backfill-window-bug.md)
|
||||
|
||||
This module's own first draft of the expand migration shipped without one line, and
|
||||
the bug it produced is worth showing exactly as it happened, because "add a nullable
|
||||
column" reads like the entire expand step and it isn't.
|
||||
|
||||
`customers.email` was declared `NOT NULL` back in
|
||||
[`V1__create_customer.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V1__create_customer.sql).
|
||||
Deploy 1 adds `email_address`, nullable — that part is fine. But Stage 4's `create()`
|
||||
never writes `email` at all:
|
||||
|
||||
```java
|
||||
case 4 -> jdbc.sql("INSERT INTO customers(name, email_address) VALUES (?, ?)")
|
||||
.param(name).param(email)
|
||||
.update(keyHolder, "id");
|
||||
```
|
||||
|
||||
If `email` is still mandatory when Stage 4 code starts running, that `INSERT` omits a
|
||||
`NOT NULL` column with no default. Every single create fails, in production, from the
|
||||
first request the first Stage 4 replica handles.
|
||||
|
||||
[`NotNullConstraintTrapTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java)
|
||||
reproduces this against the naive migration (expand plus backfill, nothing else) and
|
||||
then shows the one-line fix working:
|
||||
|
||||
```
|
||||
-- Stage 4 create() against the NAIVE migration (no DROP NOT NULL) --
|
||||
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO customers(name, email_address) VALUES (?, ?)]; NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
root cause: org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
|
||||
-- Stage 4 create() against the SHIPPED V2 migration (DROP NOT NULL included) --
|
||||
Customer[id=1, name=On Time, [email protected]]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/06-not-null-trap.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/06-not-null-trap.txt).
|
||||
|
||||
The fix is the third statement in
|
||||
[`V2__add_email_address_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V2__add_email_address_column.sql):
|
||||
|
||||
```sql
|
||||
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
|
||||
```
|
||||
|
||||
It belongs in the *same* migration as the add — Deploy 1 — not a later one. Stage 1
|
||||
and Stage 2 code both still write `email` on every insert, so relaxing its
|
||||
constraint changes nothing observable for them. But by the time Stage 4 ships, the
|
||||
constraint has to already be gone, and Stage 4 doesn't run a migration of its own —
|
||||
Deploy 4a is a pure code deploy (see [chapter 9](09-the-contract-migration.md)).
|
||||
Retrofitting the `DROP NOT NULL` later means adding a second schema change in the
|
||||
middle of what was supposed to be a code-only step.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- Every column being retired in an expand-contract migration is worth auditing for
|
||||
`NOT NULL`, `UNIQUE`, and foreign-key constraints the *new* write path won't
|
||||
satisfy — this module only had one such constraint, but a wider table can have
|
||||
several.
|
||||
- H2's constraint-violation exception hierarchy:
|
||||
[`JdbcSQLIntegrityConstraintViolationException`](https://www.h2database.com/javadoc/org/h2/api/ErrorCode.html) (nofollow).
|
||||
|
||||
[← 5. The read switch](05-the-read-switch.md) · [Next: 7. The backfill window bug →](07-the-backfill-window-bug.md)
|
||||
@@ -0,0 +1,55 @@
|
||||
# 7. The backfill window bug
|
||||
|
||||
[← 6. The NOT NULL trap](06-the-not-null-trap.md) · [Next: 8. The rolling window proof →](08-the-rolling-window-proof.md)
|
||||
|
||||
Deploy 1's migration backfills `email_address` for every row that exists *at the
|
||||
moment it runs*. It cannot see rows written after that — and a rolling deploy is not
|
||||
instantaneous, so there's a real window, between "Deploy 1's migration finished" and
|
||||
"every replica in the fleet is confirmed Stage 2 or later", during which a still-live
|
||||
Stage 1 instance keeps inserting rows the old way: `email` only, `email_address`
|
||||
untouched, `NULL`.
|
||||
|
||||
That window isn't a hypothetical edge case — it's guaranteed to happen for however
|
||||
long Deploy 2's rollout takes, on every real fleet bigger than one instance.
|
||||
|
||||
[`BackfillWindowBugTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/BackfillWindowBugTest.java)
|
||||
writes exactly such a row, then reads it back two ways:
|
||||
|
||||
```
|
||||
-- the row a lingering Stage 1 instance just wrote --
|
||||
[NAME=Katherine Johnson, [email protected], EMAIL_ADDRESS=null]
|
||||
|
||||
-- a NAIVE Stage 3 read (email_address alone) - the bug --
|
||||
naive Stage 3 email column value: null
|
||||
|
||||
-- the SHIPPED Stage 3 read (CustomerService, COALESCE) - the fix --
|
||||
CustomerService (stage 3) result: Optional[Customer[id=1, name=Katherine Johnson, [email protected]]]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/07-backfill-window-bug.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/07-backfill-window-bug.txt).
|
||||
|
||||
The naive read — `SELECT email_address FROM customers WHERE id = ?` — is exactly what
|
||||
you'd write if you thought of the read switch as "just point at the new column now
|
||||
that it's backfilled". It's wrong specifically for rows born during Deploy 2's own
|
||||
rollout, because those rows are younger than Deploy 1's one-time backfill and were
|
||||
written by an instance that (correctly, for its stage) never touched
|
||||
`email_address`.
|
||||
|
||||
The fix is the `COALESCE(email_address, email)` already shown in
|
||||
[chapter 5](05-the-read-switch.md). It costs nothing for a fully dual-written row —
|
||||
`COALESCE` returns the first non-null argument, and both columns already agree — and
|
||||
it's the only thing that returns a correct answer for a row that hasn't caught up
|
||||
yet.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- Stage 2 never needs this fallback: it still reads the original `email` column,
|
||||
which every stage keeps populated the entire time. Only Stage 3, the first stage to
|
||||
prefer the new column, needs the fallback — see the read switch in
|
||||
[chapter 5](05-the-read-switch.md).
|
||||
- This is the database-level version of a general rolling-deploy rule: **any code
|
||||
path that reads data written by a different stage has to tolerate that stage's
|
||||
write shape**, not just the shape your own stage would have produced.
|
||||
|
||||
[← 6. The NOT NULL trap](06-the-not-null-trap.md) · [Next: 8. The rolling window proof →](08-the-rolling-window-proof.md)
|
||||
@@ -0,0 +1,78 @@
|
||||
# 8. The rolling-window proof
|
||||
|
||||
[← 7. The backfill window bug](07-the-backfill-window-bug.md) · [Next: 9. The contract migration →](09-the-contract-migration.md)
|
||||
|
||||
Every rolling deploy in this sequence — Deploy 2, Deploy 3, Deploy 4a — has a window
|
||||
where two adjacent stages are serving real traffic against the same database at the
|
||||
same time. [`MixedStageRollingWindowTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java)
|
||||
is the property this whole article rests on, checked directly: it builds two
|
||||
`CustomerService` instances on adjacent stages sharing one database, and cross-checks
|
||||
every write/read direction across all three rollouts.
|
||||
|
||||
```
|
||||
-- Stage 1 writes, Stage 2 reads --
|
||||
Customer[id=1, name=Radia Perlman, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 1 reads --
|
||||
Customer[id=2, name=Barbara Liskov, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 3 reads --
|
||||
Customer[id=3, name=Shafi Goldwasser, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 2 reads --
|
||||
Customer[id=4, name=Frances Allen, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 4 reads --
|
||||
Customer[id=5, name=Adele Goldberg, [email protected]]
|
||||
|
||||
-- Stage 4 writes, Stage 3 reads --
|
||||
Customer[id=6, name=Karen Sparck Jones, [email protected]]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/08-mixed-stage-rolling-window.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/08-mixed-stage-rolling-window.txt).
|
||||
|
||||
Six pairs, six passing reads. If any one of them failed, the technique would not be
|
||||
zero-downtime for that rollout — it would just be a race against however long the
|
||||
rollout takes to finish, with correctness depending on luck rather than design.
|
||||
|
||||
<svg viewBox="0 0 720 200" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Three rolling deploys, each with an overlap window between two stages">
|
||||
<style>
|
||||
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:12px;fill:#1a1a1a}
|
||||
.lbl{font-weight:600}
|
||||
.s1{fill:#eef4fc;stroke:#3b6fb0}
|
||||
.s2{fill:#fdf0e6;stroke:#c0762c}
|
||||
.s3{fill:#eefaf0;stroke:#2f9e52}
|
||||
.s4{fill:#f6eefc;stroke:#7c3ba0}
|
||||
</style>
|
||||
<text x="10" y="20" class="lbl">Deploy 2 rollout</text>
|
||||
<rect x="10" y="30" width="220" height="24" class="s1"/><text x="18" y="46">Stage 1 (draining)</text>
|
||||
<rect x="240" y="30" width="220" height="24" class="s2"/><text x="248" y="46">Stage 2 (arriving)</text>
|
||||
<text x="470" y="46">← overlap: both true, both correct</text>
|
||||
|
||||
<text x="10" y="80" class="lbl">Deploy 3 rollout</text>
|
||||
<rect x="10" y="90" width="220" height="24" class="s2"/><text x="18" y="106">Stage 2 (draining)</text>
|
||||
<rect x="240" y="90" width="220" height="24" class="s3"/><text x="248" y="106">Stage 3 (arriving)</text>
|
||||
<text x="470" y="106">← overlap: both true, both correct</text>
|
||||
|
||||
<text x="10" y="140" class="lbl">Deploy 4a rollout</text>
|
||||
<rect x="10" y="150" width="220" height="24" class="s3"/><text x="18" y="166">Stage 3 (draining)</text>
|
||||
<rect x="240" y="150" width="220" height="24" class="s4"/><text x="248" y="166">Stage 4 (arriving)</text>
|
||||
<text x="470" y="166">← overlap: both true, both correct</text>
|
||||
</svg>
|
||||
|
||||
The diagram is the same shape three times because the guarantee is the same three
|
||||
times: whichever two stages are live together during a given rollout, a write from
|
||||
either one has to be readable correctly by the other. That's what the test above
|
||||
checks directly, and it's what the article's live 4-deploy run — a real load
|
||||
generator, hitting real HTTP endpoints, during a real rolling restart — is
|
||||
reproducing under actual timing pressure rather than a unit test's controlled
|
||||
ordering.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The live version of this proof, with two real replicas and continuous HTTP
|
||||
traffic: [chapter 11](11-the-load-generator.md) and the full run in
|
||||
[`docs/output/11-live-deploy-sequence.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/11-live-deploy-sequence.txt).
|
||||
|
||||
[← 7. The backfill window bug](07-the-backfill-window-bug.md) · [Next: 9. The contract migration →](09-the-contract-migration.md)
|
||||
@@ -0,0 +1,47 @@
|
||||
# 9. The contract migration
|
||||
|
||||
[← 8. The rolling window proof](08-the-rolling-window-proof.md) · [Next: 10. What happens if you drop too soon →](10-what-happens-if-you-drop-too-soon.md)
|
||||
|
||||
Deploy 4 is split into two parts on purpose, the same way Deploy 1 was schema-only
|
||||
and Deploys 2/3 were code-only:
|
||||
|
||||
- **4a — code.** A rolling restart to Stage 4, which never reads or writes `email`.
|
||||
The old column is still physically present; Stage 4 code simply ignores it.
|
||||
- **4b — schema.** Once every replica is confirmed on Stage 4 code, and only then,
|
||||
[`V3__drop_email_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V3__drop_email_column.sql)
|
||||
runs, with zero app restarts:
|
||||
|
||||
```sql
|
||||
ALTER TABLE customers DROP COLUMN email;
|
||||
```
|
||||
|
||||
[`ContractSafetyTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java)
|
||||
confirms the happy path: Stage 4 reads and writes keep working, both for a row that
|
||||
existed before the drop and for one created entirely after it —
|
||||
|
||||
```
|
||||
-- Stage 4 read, after V3 dropped the email column --
|
||||
Customer[id=1, name=Annie Easley, [email protected]]
|
||||
|
||||
-- Stage 4 create + read, entirely after the drop --
|
||||
Customer[id=2, name=Mary Allen Wilkes, [email protected]]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/09-contract-safety.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/09-contract-safety.txt).
|
||||
|
||||
Why not drop the column in the same deploy as the code change? Because "every
|
||||
replica is confirmed on Stage 4" is a statement about the *fleet*, and a rolling
|
||||
deploy makes it true only once, at the very end of the rollout — never at the moment
|
||||
the deploy starts. Running 4b before that point is dropping a column a live Stage 1,
|
||||
2, or 3 replica might still need. What that actually looks like, captured directly,
|
||||
is [chapter 10](10-what-happens-if-you-drop-too-soon.md).
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The DDL Deploy 4b runs is not, on this specific database engine, quite as free of
|
||||
side effects on concurrent traffic as "just a schema change" suggests — see
|
||||
[chapter 14](14-the-ddl-lock-window.md) for what H2 actually does while this
|
||||
statement executes.
|
||||
|
||||
[← 8. The rolling window proof](08-the-rolling-window-proof.md) · [Next: 10. What happens if you drop too soon →](10-what-happens-if-you-drop-too-soon.md)
|
||||
@@ -0,0 +1,42 @@
|
||||
# 10. What happens if you drop too soon
|
||||
|
||||
[← 9. The contract migration](09-the-contract-migration.md) · [Next: 11. The load generator →](11-the-load-generator.md)
|
||||
|
||||
The second half of
|
||||
[`ContractSafetyTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java)
|
||||
runs Deploy 4b's migration first, and only then starts a Stage 1 instance against the
|
||||
result — standing in for a canary that never got promoted, a rollback that didn't
|
||||
fully take, or simply running the drop before confirming the fleet:
|
||||
|
||||
```
|
||||
-- What a lingering Stage 1 instance sees if the drop runs before it is retired --
|
||||
Stage 1 create() after V3 dropped "email": org.springframework.jdbc.BadSqlGrammarException
|
||||
message: PreparedStatementCallback; bad SQL grammar [INSERT INTO customers(name, email) VALUES (?, ?)]
|
||||
root cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "EMAIL" not found; SQL statement:
|
||||
INSERT INTO customers(name, email) VALUES (?, ?) [42122-240]
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/10-drop-too-soon.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/10-drop-too-soon.txt).
|
||||
|
||||
This is the correct outcome, not a bug to work around. A Stage 1 instance still
|
||||
running after the drop is itself the actual mistake — a deploy that didn't finish, or
|
||||
a rollback nobody noticed failed — and the database telling it loudly and immediately
|
||||
that `email` doesn't exist is far better than the alternative of silently accepting
|
||||
partial writes or, worse, dropping rows. Expand-contract's safety comes from **when**
|
||||
you're allowed to run Deploy 4b (only after confirming 100% Stage 4), not from Deploy
|
||||
4b itself being forgiving of running early.
|
||||
|
||||
In a real deploy pipeline, this is the argument for gating Deploy 4b on an explicit
|
||||
health/version check across the fleet — every instance's `/actuator/info` or
|
||||
equivalent reporting Stage 4 — rather than a fixed timer. "Deploy 2 usually finishes
|
||||
rolling out in three minutes" is not the same guarantee as "every replica confirmed
|
||||
Stage 4", and the difference between them is exactly the window this chapter's test
|
||||
is exploiting.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- [Chapter 15](15-production-checklist.md) turns this into an actual gate: what to
|
||||
check, and where, before running a contract migration in a real pipeline.
|
||||
|
||||
[← 9. The contract migration](09-the-contract-migration.md) · [Next: 11. The load generator →](11-the-load-generator.md)
|
||||
@@ -0,0 +1,124 @@
|
||||
# 11. The load generator
|
||||
|
||||
[← 10. What happens if you drop too soon](10-what-happens-if-you-drop-too-soon.md) · [Next: 12. The AUTO_SERVER trap →](12-the-auto-server-trap.md)
|
||||
|
||||
Every other chapter so far proves a property of the technique with a JUnit test
|
||||
against a shared `JdbcClient` — real code, real SQL, no mocks, but also no real HTTP,
|
||||
no real process restarts, no real timing pressure. `scripts/run-all.sh` is where all
|
||||
three of those show up: two real Spring Boot processes, a real standalone database, a
|
||||
real rolling restart between each stage, and
|
||||
[`LoadGenerator`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java)
|
||||
sending continuous HTTP traffic through the whole sequence.
|
||||
|
||||
## What it does
|
||||
|
||||
Eight worker threads, each running the same loop: pick a healthy backend, then create
|
||||
a customer (50% of the time), read one it already knows about (30%), or update one
|
||||
(20%). Every read and update verifies the response body matches what the load
|
||||
generator itself expects — not just the HTTP status code:
|
||||
|
||||
```java
|
||||
JsonNode node = MAPPER.readTree(resp.body());
|
||||
String returnedEmail = node.get("email").asString();
|
||||
if (!target.email().equals(returnedEmail)) {
|
||||
c.recordError("read-consistency-mismatch");
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
A 200 with the wrong email in the body would be a much worse bug than a 500, and a
|
||||
naive load test that only checks status codes would never catch it.
|
||||
|
||||
## Health-checked traffic, not a raw hose
|
||||
|
||||
`LoadGenerator` polls `/actuator/health` on both ports every 300ms and only sends
|
||||
traffic to backends it currently believes are up:
|
||||
|
||||
```java
|
||||
private static final int UNHEALTHY_THRESHOLD = 2;
|
||||
```
|
||||
|
||||
A backend needs two consecutive failed checks before it's removed from rotation —
|
||||
this exists because this whole sequence runs two JVMs on a shared, small sandbox, and
|
||||
one replica's cold-start CPU burst can make its *sibling* miss a single health check
|
||||
without actually being down. Requiring two consecutive failures is the same debounce
|
||||
a real load balancer's health check threshold gives you, and skipping it turned
|
||||
transient slowness into false `no-healthy-backend` errors in an earlier version of
|
||||
this test.
|
||||
|
||||
That fix mattered enough to also show up in
|
||||
[`start-instance.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/start-instance.sh),
|
||||
on the other side of the same problem — cutting each replica's own startup CPU cost
|
||||
so it's less likely to starve its sibling in the first place:
|
||||
|
||||
```bash
|
||||
nohup java -XX:TieredStopAtLevel=1 -XX:+UseSerialGC -Xms128m -Xmx256m \
|
||||
-jar "$JAR" --server.port="$PORT" --app.stage="$STAGE" \
|
||||
```
|
||||
|
||||
## Per-thread state, not shared state
|
||||
|
||||
Each worker thread owns a private `ArrayDeque` of customer ids it has created — never
|
||||
shared with the other seven threads:
|
||||
|
||||
```java
|
||||
workers.submit(() -> {
|
||||
try {
|
||||
workerLoop(new ArrayDeque<>());
|
||||
```
|
||||
|
||||
An earlier version shared one pool across all eight threads, and produced
|
||||
`read-consistency-mismatch` errors that had nothing to do with the server at all: two
|
||||
threads racing to update the *same* shared id could leave the pool holding a stale
|
||||
expected value, so a perfectly correct server response looked like a bug. Giving each
|
||||
thread exclusive ownership of the rows it creates removes that entire class of false
|
||||
positive while still hammering both replicas concurrently.
|
||||
|
||||
## The result
|
||||
|
||||
```
|
||||
Load generator summary
|
||||
=======================
|
||||
Total requests: 30911
|
||||
Successful: 30905
|
||||
Errors: 6
|
||||
|
||||
By phase:
|
||||
04a-stage4-soak ok=2130 error=0
|
||||
04b-contract-migration ok=336 error=0
|
||||
04a-deploy-stage4-rollout ok=4147 error=0
|
||||
03-stage3-soak ok=2116 error=0
|
||||
03-deploy-stage3-rollout ok=4645 error=0
|
||||
00-baseline-soak ok=1893 error=0
|
||||
05-final-soak ok=7118 error=4
|
||||
- read-http-404 2
|
||||
- update-http-404 2
|
||||
02-stage2-soak ok=2068 error=0
|
||||
01-expand-migration ok=1773 error=1
|
||||
- read-http-404 1
|
||||
02-deploy-stage2-rollout ok=4679 error=1
|
||||
- update-http-404 1
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/12-load-generator-summary.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/12-load-generator-summary.txt),
|
||||
and the phase-by-phase deploy log this run came from:
|
||||
[`docs/output/11-live-deploy-sequence.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/11-live-deploy-sequence.txt).
|
||||
|
||||
30,905 of 30,911 requests succeeded — 99.98%, across a real rolling restart through
|
||||
all four deploys. The six errors are all `404`s, not `500`s: a request for a customer
|
||||
id that genuinely wasn't found, not a crash. Every one of them traces to the same
|
||||
root cause, and it's the most interesting finding in this whole module — see
|
||||
[chapter 14](14-the-ddl-lock-window.md).
|
||||
|
||||
## Going deeper
|
||||
|
||||
- The graceful-shutdown-and-drain sequence that gets the *rolling restart* portion of
|
||||
this run down to zero client-visible errors on its own is
|
||||
[chapter 13](13-graceful-shutdown-vs-kill-9.md) — the six remaining errors above
|
||||
have a different cause entirely, isolated in chapter 14.
|
||||
- Every deploy's schema state during this exact run, captured live:
|
||||
[`docs/output/13-schema-diagnostics-timeline.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/13-schema-diagnostics-timeline.txt),
|
||||
via [`SchemaDiagnosticsController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java).
|
||||
|
||||
[← 10. What happens if you drop too soon](10-what-happens-if-you-drop-too-soon.md) · [Next: 12. The AUTO_SERVER trap →](12-the-auto-server-trap.md)
|
||||
@@ -0,0 +1,65 @@
|
||||
# 12. The AUTO_SERVER trap
|
||||
|
||||
[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)
|
||||
|
||||
This module's first working draft pointed both replicas at the same H2 file with
|
||||
`AUTO_SERVER=TRUE` — the mode most H2 tutorials show for "let two JVMs share one
|
||||
embedded database file without a separate server process":
|
||||
|
||||
```
|
||||
jdbc:h2:file:./data/expand-contract;AUTO_SERVER=TRUE
|
||||
```
|
||||
|
||||
It works, right up until a rolling deploy restarts the *particular* replica that
|
||||
happened to open the file first. `AUTO_SERVER` makes the first connecting process the
|
||||
de facto database server for every other connection — internally, later connections
|
||||
become `SessionRemote` clients of that first process, not independent embedded
|
||||
sessions. Kill that one process — which a rolling deploy does, routinely, by design —
|
||||
and every *other* replica's connection to the "embedded" database breaks with it. In
|
||||
this module's own early runs, that showed up as `Table "CUSTOMERS" not found` on the
|
||||
surviving replica, immediately after the first replica restarted, for a table that
|
||||
had existed the entire time. It was single-point-of-failure architecture disguised as
|
||||
an embedded database, and it was the single biggest source of load-generator errors
|
||||
before it was found and fixed — cutting the error rate by roughly 90% on its own.
|
||||
|
||||
The fix, in
|
||||
[`start-db-server.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/start-db-server.sh),
|
||||
is to stop pretending the database is embedded at all and run it as what it actually
|
||||
needs to be: its own standalone process that neither replica owns and neither
|
||||
replica's lifecycle affects.
|
||||
|
||||
```bash
|
||||
nohup java -cp "$(cat "$CP_FILE")" org.h2.tools.Server \
|
||||
-tcp -tcpPort "$EC_DB_TCP_PORT" -baseDir "$EC_DB_BASE_DIR" -ifNotExists \
|
||||
> "$EC_LOG_DIR/db-server.log" 2>&1 < /dev/null &
|
||||
```
|
||||
|
||||
Both replicas — and
|
||||
[`MigrationCli`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/migration/MigrationCli.java)
|
||||
— connect to it the same way, as ordinary TCP clients:
|
||||
|
||||
```yaml
|
||||
url: jdbc:h2:tcp://localhost:${EC_DB_TCP_PORT:9092}/${EC_DB_NAME:expand-contract}
|
||||
```
|
||||
|
||||
`scripts/run-all.sh` starts this server first, before either replica, and stops it
|
||||
last, after both — the one process in the entire sequence that is never restarted,
|
||||
because it's standing in for what a real production database always is: a process
|
||||
that outlives every deploy of every application that talks to it.
|
||||
|
||||
One flag worth calling out because it looks interchangeable and isn't:
|
||||
`-tcpDaemon` marks the server thread as a daemon thread, which is for embedding an H2
|
||||
server *inside* another long-running JVM that manages its own lifecycle — for a
|
||||
standalone, always-on server process like this one, it made the process exit
|
||||
immediately in a manual test, because there was no non-daemon thread left to keep the
|
||||
JVM alive. Leave it off for a server meant to run on its own.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- H2's own documentation on
|
||||
[automatic mixed mode](https://www.h2database.com/html/features.html#auto_mixed_mode)
|
||||
(nofollow) describes `AUTO_SERVER` for the single-application-process,
|
||||
multiple-connections case it's actually designed for — not for two independent
|
||||
application processes that need to survive each other's restarts.
|
||||
|
||||
[← 11. The load generator](11-the-load-generator.md) · [Next: 13. Graceful shutdown vs. kill -9 →](13-graceful-shutdown-vs-kill-9.md)
|
||||
@@ -0,0 +1,77 @@
|
||||
# 13. Graceful shutdown vs. kill -9
|
||||
|
||||
[← 12. The AUTO_SERVER trap](12-the-auto-server-trap.md) · [Next: 14. The DDL lock window →](14-the-ddl-lock-window.md)
|
||||
|
||||
`application.yml` sets one line that does nothing by itself:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
shutdown: graceful
|
||||
```
|
||||
|
||||
`server.shutdown: graceful` only changes behavior on `SIGTERM` — it stops accepting
|
||||
new connections but lets in-flight requests finish before the process exits. An
|
||||
earlier version of
|
||||
[`stop-instance.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/stop-instance.sh)
|
||||
used `kill -9`, which bypasses graceful shutdown entirely — the process disappears
|
||||
mid-request, and every request in flight at that instant surfaces in the load
|
||||
generator as a raw `ConnectException` or `IOException`. Switching to `SIGTERM`
|
||||
(`kill -15`), with a bounded wait for a clean exit and `SIGKILL` only as a fallback,
|
||||
is the first half of the fix:
|
||||
|
||||
```bash
|
||||
kill -15 "$PID"
|
||||
for i in $(seq 1 40); do
|
||||
kill -0 "$PID" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
if kill -0 "$PID" 2>/dev/null; then
|
||||
kill -9 "$PID"
|
||||
fi
|
||||
```
|
||||
|
||||
That alone wasn't enough. `server.shutdown: graceful` starts refusing new connections
|
||||
the instant `SIGTERM` arrives — but the load generator's health checker polls every
|
||||
300ms, and the pool didn't yet know to stop routing traffic there. The gap between
|
||||
"the process just stopped accepting connections" and "the load balancer's health
|
||||
check has noticed and rerouted" is exactly where `ConnectException` bursts kept
|
||||
showing up, even with `SIGTERM` in place.
|
||||
|
||||
The second half of the fix is a way to say "stop sending me traffic" *before* the
|
||||
process is touched at all:
|
||||
[`DrainController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/DrainController.java):
|
||||
|
||||
```java
|
||||
@PostMapping("/admin/drain")
|
||||
public String drain() {
|
||||
AvailabilityChangeEvent.publish(events, this, ReadinessState.REFUSING_TRAFFIC);
|
||||
return "draining";
|
||||
}
|
||||
```
|
||||
|
||||
Publishing `ReadinessState.REFUSING_TRAFFIC` flips `/actuator/health`'s readiness
|
||||
group immediately — this is the same event a Kubernetes-style `preStop` hook
|
||||
publishes before the container is sent `SIGTERM`. `stop-instance.sh` calls it, sleeps,
|
||||
*then* sends `SIGTERM`:
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
|
||||
sleep 1.5
|
||||
kill -15 "$PID"
|
||||
```
|
||||
|
||||
That 1.5-second pause is deliberate slack for the health checker's 300ms poll
|
||||
interval — enough for at least a couple of checks to land and pull this instance out
|
||||
of rotation before it's asked to stop at all. Together, drain-then-SIGTERM is what
|
||||
took the rolling-restart portion of the article's live run to zero
|
||||
`ConnectException`/`IOException` errors — the six errors that remain in the final
|
||||
summary are a completely different, database-level cause, covered next.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- Spring's own `ReadinessState` and the Kubernetes probe pattern it mirrors:
|
||||
[Spring Boot reference docs, Application Availability](https://docs.spring.io/spring-boot/reference/actuator/application-availability.html) (nofollow).
|
||||
- `/admin/drain` is a diagnostic-grade endpoint with no auth — see
|
||||
[chapter 15](15-production-checklist.md) for what to do with it before shipping.
|
||||
|
||||
[← 12. The AUTO_SERVER trap](12-the-auto-server-trap.md) · [Next: 14. The DDL lock window →](14-the-ddl-lock-window.md)
|
||||
@@ -0,0 +1,199 @@
|
||||
# 14. The DDL lock window
|
||||
|
||||
[← 13. Graceful shutdown vs. kill -9](13-graceful-shutdown-vs-kill-9.md) · [Next: 15. Production checklist →](15-production-checklist.md)
|
||||
|
||||
Every schema-only deploy in this sequence — Deploy 1's `ADD COLUMN` and Deploy 4b's
|
||||
`DROP COLUMN` — runs against a live database while both replicas keep taking real
|
||||
traffic. That's the entire point of running migrations outside the app (see
|
||||
[chapter 3](03-why-migrations-run-outside-the-app.md)): zero app restarts for a
|
||||
schema-only step. What this chapter covers is what "zero app restarts" does *not*
|
||||
automatically buy you: zero effect on concurrent queries while the `ALTER TABLE`
|
||||
statement itself is executing.
|
||||
|
||||
There are two distinct failure modes here, found the same way — by watching the
|
||||
article's own load generator run against a live rollout and refusing to wave away
|
||||
the handful of errors it reported.
|
||||
|
||||
## Failure mode 1: a statement that collides with the DDL, and says so
|
||||
|
||||
While `V3__drop_email_column.sql` runs, a concurrent, otherwise-correct `INSERT` or
|
||||
`UPDATE` can briefly see:
|
||||
|
||||
```
|
||||
org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "CUSTOMERS" not found
|
||||
```
|
||||
|
||||
This is a real, transient condition captured live, twice, in independent runs of this
|
||||
module's load generator — not a bug in the application's SQL. H2's TCP server
|
||||
appears to make the table briefly unavailable to other sessions while `DROP COLUMN`
|
||||
executes. The fix is a narrowly scoped single retry in
|
||||
[`CustomerService.withRetryForConcurrentDdl`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/CustomerService.java):
|
||||
|
||||
```java
|
||||
private <T> T withRetryForConcurrentDdl(Supplier<T> operation) {
|
||||
try {
|
||||
return operation.get();
|
||||
} catch (BadSqlGrammarException ex) {
|
||||
if (ex.getMessage() != null && ex.getMessage().contains("CUSTOMERS")) {
|
||||
try {
|
||||
Thread.sleep(50);
|
||||
} catch (InterruptedException ie) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
return operation.get();
|
||||
}
|
||||
throw ex;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
It catches specifically the "table not found" grammar error, sleeps 50ms — long
|
||||
enough for the in-flight `ALTER TABLE` to finish — and retries exactly once. It is
|
||||
deliberately narrow: anything else still propagates. Retrying every `BadSqlGrammarException`
|
||||
indiscriminately would mask genuine programming errors (a typo'd column name, for
|
||||
instance) as if they were transient — this retry only fires for the one specific,
|
||||
verified condition. `create()`, `updateEmail()`, and `findById()` are all wrapped in
|
||||
it, because a concurrent read can hit the exact same window a concurrent write can.
|
||||
After adding this retry, a full re-run of the live sequence produced **zero**
|
||||
`create-http-500` / `update-http-500` errors during Deploy 4b — the class of error
|
||||
this fix targets is fully eliminated. Compare
|
||||
[`docs/output/12-load-generator-summary.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/12-load-generator-summary.txt),
|
||||
where the `04b-contract-migration` phase shows `ok=336 error=0`.
|
||||
|
||||
## Failure mode 2: a statement that succeeds, and is still lost
|
||||
|
||||
The retry above only helps when the colliding statement *throws*. It does nothing for
|
||||
the six residual `404`s that remained after the fix — a customer id that a `201`
|
||||
response had already confirmed existed, later reported not found by a plain read or
|
||||
update, with no exception anywhere in the logs. That gap between "the fix that
|
||||
worked" and "the errors that didn't go away" is what led to the actual root cause.
|
||||
|
||||
[`DdlSilentDataLossTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/DdlSilentDataLossTest.java)
|
||||
reproduces it directly: one thread inserting customers continuously through
|
||||
`CustomerService.create()`, while `V2` (`ADD COLUMN`) runs concurrently on another
|
||||
connection.
|
||||
|
||||
```java
|
||||
Thread inserter = new Thread(() -> {
|
||||
while (!stop.get()) {
|
||||
long id = stage1.create("Concurrent " + n, "concurrent" + n + "@example.test");
|
||||
confirmedIds.add(id); // no exception - the insert reported success
|
||||
}
|
||||
});
|
||||
inserter.start();
|
||||
TestSupport.migrateTo(db, "2"); // ADD COLUMN, running concurrently
|
||||
```
|
||||
|
||||
Exactly when the rebuild's internal scan passes a given row relative to that row's
|
||||
own commit is OS thread scheduling, not application logic — a single attempt can
|
||||
land on either side of the race. Rather than assert on one attempt (which failed to
|
||||
reproduce roughly one run in five while writing this test) or weaken the assertion to
|
||||
"zero or more" — which would silently stop proving anything the day this stops
|
||||
reproducing — the test repeats the race on a fresh table until it reproduces, up to
|
||||
20 times, the same thing a human would do at a terminal to confirm a suspected race
|
||||
is real:
|
||||
|
||||
```
|
||||
attempts needed to reproduce the race: 2 of 20
|
||||
customer creates that returned a generated id with no error: 50
|
||||
customer creates that got the already-documented, already-fixed DDL-collision error: 1
|
||||
of the ids that came back with no error, missing from the table once V2 finished: 8
|
||||
example missing ids: [41, 42, 43, 44, 45]
|
||||
|
||||
This is why the retry in CustomerService cannot be the whole fix: these inserts
|
||||
never threw anything to retry. The row was committed, then discarded when the
|
||||
ADD COLUMN rebuild swapped in a new table that had already been scanned.
|
||||
```
|
||||
|
||||
Full transcript:
|
||||
[`docs/output/14-ddl-silent-data-loss.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/14-ddl-silent-data-loss.txt).
|
||||
|
||||
Under a tight, unthrottled loop with no delay between inserts, this run lost 8 of the
|
||||
50 rows inserted during the migration window — other runs made while writing this
|
||||
test lost anywhere from 1 to a few dozen, depending on exactly how the two threads
|
||||
got scheduled. That number is not a claim about the live article run's own rate — the
|
||||
load generator's eight threads sleep 15–40ms between requests and aren't hammering
|
||||
the exact millisecond the migration executes, so far fewer of its requests land in
|
||||
the vulnerable window. It's a claim about the *mechanism*: H2 implements both
|
||||
`ALTER TABLE ... ADD COLUMN` and
|
||||
`ALTER TABLE ... DROP COLUMN` by rebuilding the table — copying every row into a new
|
||||
table with the new column layout and swapping it in. A row inserted on another
|
||||
connection can commit while that rebuild is mid-scan; depending on exactly when the
|
||||
scan reaches the row relative to the commit, the row ends up copied into the new
|
||||
table or left behind in the old one. When it's left behind, it disappears the instant
|
||||
the rebuild finishes, and the connection that inserted it was never told anything
|
||||
went wrong — the `INSERT` had already returned successfully.
|
||||
|
||||
This is why `withRetryForConcurrentDdl` cannot be "the fix" for the residual errors:
|
||||
there is nothing to retry. The failure isn't a rejected statement; it's data that
|
||||
existed for a moment and then didn't, discovered only by a later, unrelated read.
|
||||
|
||||
<svg viewBox="0 0 740 220" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="A row committed during an ALTER TABLE rebuild can be silently dropped when the rebuild swaps in the new table">
|
||||
<style>
|
||||
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:12px;fill:#1a1a1a}
|
||||
.hdr{font-weight:600}
|
||||
.old{fill:#f4f4f4;stroke:#999}
|
||||
.new{fill:#eefaf0;stroke:#2f9e52}
|
||||
.row{fill:#eef4fc;stroke:#3b6fb0}
|
||||
.lost{fill:#fdeceb;stroke:#c0392b}
|
||||
.arrow{stroke:#555;stroke-width:1.4;marker-end:url(#a2)}
|
||||
</style>
|
||||
<defs><marker id="a2" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6 z" fill="#555"/></marker></defs>
|
||||
<text x="10" y="20" class="hdr">1. ALTER TABLE begins the rebuild: scans OLD, copies rows into NEW</text>
|
||||
<rect x="10" y="30" width="180" height="80" class="old"/>
|
||||
<text x="18" y="48">OLD table</text>
|
||||
<rect x="18" y="56" width="60" height="16" class="row"/><text x="22" y="68">row 1</text>
|
||||
<rect x="90" y="56" width="60" height="16" class="row"/><text x="94" y="68">row 2</text>
|
||||
<line x1="200" y1="70" x2="260" y2="70" class="arrow"/>
|
||||
<rect x="270" y="30" width="180" height="80" class="new"/>
|
||||
<text x="278" y="48">NEW table (new columns)</text>
|
||||
<rect x="278" y="56" width="60" height="16" class="row"/><text x="282" y="68">row 1</text>
|
||||
<rect x="350" y="56" width="60" height="16" class="row"/><text x="354" y="68">row 2</text>
|
||||
|
||||
<text x="10" y="140" class="hdr">2. A concurrent INSERT commits into OLD after the scan already passed that point</text>
|
||||
<rect x="10" y="150" width="180" height="50" class="old"/>
|
||||
<rect x="18" y="166" width="60" height="16" class="lost"/><text x="22" y="178">row 3 (new!)</text>
|
||||
<rect x="270" y="150" width="180" height="50" class="new"/>
|
||||
<text x="278" y="172">(row 3 never scanned)</text>
|
||||
|
||||
<text x="10" y="215" class="hdr">3. Rebuild finishes, NEW swaps in for OLD - row 3 is gone, with no error to anyone</text>
|
||||
</svg>
|
||||
|
||||
The diagram's third step is the whole finding in one line: nothing in this sequence
|
||||
is a bug in the *application's* SQL, the migration's SQL, or the expand-contract
|
||||
technique — it's a property of how this specific embedded database implements two
|
||||
DDL statements that a lot of guidance describes as "safe" without qualification.
|
||||
|
||||
## This is a property of H2, not of expand-contract
|
||||
|
||||
PostgreSQL's own reference manual is explicit that this isn't universal:
|
||||
|
||||
> When a column is added with `ADD COLUMN` and a non-volatile `DEFAULT` is specified
|
||||
> [or none is], ... In neither case is a rewrite of the table required.
|
||||
>
|
||||
> The `DROP COLUMN` form does not physically remove the column, but simply makes it
|
||||
> invisible to SQL operations.
|
||||
|
||||
Both operations are metadata-only in Postgres for the shapes used here — no table
|
||||
rewrite, and therefore no window where a concurrently committed row can be scanned
|
||||
past. (Postgres still takes a brief `ACCESS EXCLUSIVE` lock to make the metadata
|
||||
change, which blocks concurrent statements for that short duration rather than racing
|
||||
past them — a different, more familiar tradeoff than silent loss.) A real migration
|
||||
of a real production table should treat this as a question to answer about *your*
|
||||
database, not assume either behavior: does `ALTER TABLE ADD/DROP COLUMN` rewrite the
|
||||
table on your engine, and if it does, what does that engine guarantee about
|
||||
concurrent writes during the rewrite? For MySQL/InnoDB, that answer depends on the
|
||||
specific `ALGORITHM` the storage engine picks for the given change — `INSTANT` and
|
||||
`INPLACE` avoid a full rebuild, `COPY` does not.
|
||||
|
||||
## Going deeper
|
||||
|
||||
- [Chapter 12](12-the-auto-server-trap.md) covers a different H2-specific surprise
|
||||
found the same way — running as a "shared embedded" database instead of a real
|
||||
standalone server.
|
||||
- PostgreSQL's [`ALTER TABLE` reference](https://www.postgresql.org/docs/current/sql-altertable.html) (nofollow)
|
||||
is the primary source for the Postgres claims above.
|
||||
- MySQL's [Online DDL documentation](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl.html) (nofollow)
|
||||
covers which `ALTER TABLE` operations get `INSTANT`/`INPLACE` treatment on InnoDB.
|
||||
|
||||
[← 13. Graceful shutdown vs. kill -9](13-graceful-shutdown-vs-kill-9.md) · [Next: 15. Production checklist →](15-production-checklist.md)
|
||||
@@ -0,0 +1,59 @@
|
||||
# 15. Production checklist
|
||||
|
||||
[← 14. The DDL lock window](14-the-ddl-lock-window.md)
|
||||
|
||||
Everything in this module is a demonstration, deliberately built so its failure modes
|
||||
are reachable and its output is captured. Taking the technique — not this exact
|
||||
code — to a real service means addressing what this module intentionally left
|
||||
uncovered:
|
||||
|
||||
- **Gate the contract migration on confirmed fleet state, not a timer.** [Chapter
|
||||
10](10-what-happens-if-you-drop-too-soon.md) shows what a lingering old-stage
|
||||
instance sees the moment the drop runs — a real deploy pipeline should check every
|
||||
instance's reported version/health before running Deploy 4b, not just wait "long
|
||||
enough".
|
||||
- **Know your database's `ALTER TABLE` semantics before you run this against a real
|
||||
table.** [Chapter 14](14-the-ddl-lock-window.md) is specific to H2's
|
||||
rebuild-based implementation of `ADD COLUMN`/`DROP COLUMN` — verify what your actual
|
||||
production database does under concurrent writes for the specific change you're
|
||||
making, and test it, the same way that chapter's test does, against your own engine
|
||||
and table size before trusting either "it's always safe" or "it always errors
|
||||
loudly".
|
||||
- **Remove or lock down the diagnostic endpoints.**
|
||||
[`SchemaDiagnosticsController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java)'s
|
||||
`/diag/schema` and
|
||||
[`DrainController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/DrainController.java)'s
|
||||
`/admin/drain` have no authentication and are wired for a demo where anyone can
|
||||
curl them freely. `/admin/drain` in particular can take a real instance out of a
|
||||
real load balancer's rotation with a single unauthenticated `POST` — restrict it to
|
||||
the same internal network your orchestrator's `preStop` hook runs from, or replace
|
||||
it with your platform's native lifecycle hook.
|
||||
- **Run the real load test against your real database and your real fleet size**,
|
||||
not just this module's two-replica sandbox. The `AUTO_SERVER` trap
|
||||
([chapter 12](12-the-auto-server-trap.md)) and the DDL lock window
|
||||
([chapter 14](14-the-ddl-lock-window.md)) were both found by running actual
|
||||
concurrent traffic through actual restarts — reading about the technique would not
|
||||
have surfaced either one.
|
||||
- **Budget for at least two separate code deploys plus two schema changes**, not one
|
||||
deploy. Teams estimating "rename a column" as a single-PR, single-deploy task are
|
||||
the ones most likely to reach for the `RENAME COLUMN` shortcut this article opened
|
||||
with — see [chapter 1](01-the-problem-and-the-plan.md).
|
||||
- **Decide what "zero errors" means for your own load test before you run it.** This
|
||||
article's own final number is 99.98%, not literally zero, and the six-in-30,911
|
||||
residual is explained down to its root mechanism rather than hand-waved — see the
|
||||
honest accounting in [chapter 14](14-the-ddl-lock-window.md). A number you can fully
|
||||
explain is more useful, and more trustworthy, than one you cannot account for at
|
||||
all.
|
||||
|
||||
## The whole sequence, one command
|
||||
|
||||
```bash
|
||||
./scripts/run-all.sh
|
||||
```
|
||||
|
||||
regenerates every transcript this article and these chapters quote, end to end:
|
||||
schema-only Deploy 1, rolling Deploy 2, rolling Deploy 3, rolling Deploy 4a,
|
||||
schema-only Deploy 4b, with the load generator running continuously throughout. See
|
||||
the [module README](../README.md) for the full script index and version table.
|
||||
|
||||
[← 14. The DDL lock window](14-the-ddl-lock-window.md)
|
||||
@@ -0,0 +1,36 @@
|
||||
=========================================================================
|
||||
Deploy 1 (EXPAND): additive column + backfill, Stage 1 code untouched
|
||||
=========================================================================
|
||||
captured: 2026-09-16T19:20:56.676919361Z
|
||||
|
||||
|
||||
-- schema before Deploy 1 --
|
||||
COLUMN_NAME
|
||||
-----------
|
||||
ID
|
||||
NAME
|
||||
EMAIL
|
||||
CREATED_AT
|
||||
(4 rows)
|
||||
|
||||
-- schema after Deploy 1 (email_address added) --
|
||||
COLUMN_NAME
|
||||
-------------
|
||||
ID
|
||||
NAME
|
||||
EMAIL
|
||||
CREATED_AT
|
||||
EMAIL_ADDRESS
|
||||
(5 rows)
|
||||
|
||||
-- Ada's row was backfilled by the migration itself --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
-------------+------------------+-----------------
|
||||
Ada Lovelace | [email protected] | [email protected]
|
||||
(1 row)
|
||||
|
||||
-- Stage 1's original INSERT still works, unmodified, after the migration --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
-------------+--------------------+--------------
|
||||
Grace Hopper | [email protected] | NULL
|
||||
(1 row)
|
||||
@@ -0,0 +1,17 @@
|
||||
==================================================================
|
||||
Deploy 2 (MIGRATE WRITES): Stage 2 writes land in both columns
|
||||
==================================================================
|
||||
captured: 2026-09-16T19:21:04.333919632Z
|
||||
|
||||
|
||||
-- after create() --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-----------------------+----------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
(1 row)
|
||||
|
||||
-- after updateEmail() - the old value is gone from BOTH columns, not just one --
|
||||
NAME | EMAIL | EMAIL_ADDRESS
|
||||
------------------+-------------------------+------------------------
|
||||
Margaret Hamilton | [email protected] | [email protected]
|
||||
(1 row)
|
||||
@@ -0,0 +1,14 @@
|
||||
==========================================================================
|
||||
The NOT NULL trap: expand without relaxing the old column's constraint
|
||||
==========================================================================
|
||||
captured: 2026-09-16T19:21:04.662650854Z
|
||||
|
||||
|
||||
-- Stage 4 create() against the NAIVE migration (no DROP NOT NULL) --
|
||||
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO customers(name, email_address) VALUES (?, ?)]; NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
root cause: org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: NULL not allowed for column "EMAIL"; SQL statement:
|
||||
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
|
||||
|
||||
-- Stage 4 create() against the SHIPPED V2 migration (DROP NOT NULL included) --
|
||||
Customer[id=1, name=On Time, [email protected]]
|
||||
@@ -0,0 +1,14 @@
|
||||
==============================================================================
|
||||
The backfill window: a Stage 1 write after Deploy 1, read by a naive Stage 3
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:56.276837493Z
|
||||
|
||||
|
||||
-- the row a lingering Stage 1 instance just wrote --
|
||||
[{NAME=Katherine Johnson, [email protected], EMAIL_ADDRESS=null}]
|
||||
|
||||
-- a NAIVE Stage 3 read (email_address alone) - the bug --
|
||||
naive Stage 3 email column value: null
|
||||
|
||||
-- the SHIPPED Stage 3 read (CustomerService, COALESCE) - the fix --
|
||||
CustomerService (stage 3) result: Optional[Customer[id=1, name=Katherine Johnson, [email protected]]]
|
||||
@@ -0,0 +1,23 @@
|
||||
====================================================================
|
||||
Cross-stage consistency during each of the three rolling deploys
|
||||
====================================================================
|
||||
captured: 2026-09-16T19:21:05.241547756Z
|
||||
|
||||
|
||||
-- Stage 1 writes, Stage 2 reads --
|
||||
Customer[id=1, name=Radia Perlman, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 1 reads --
|
||||
Customer[id=2, name=Barbara Liskov, [email protected]]
|
||||
|
||||
-- Stage 2 writes, Stage 3 reads --
|
||||
Customer[id=3, name=Shafi Goldwasser, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 2 reads --
|
||||
Customer[id=4, name=Frances Allen, [email protected]]
|
||||
|
||||
-- Stage 3 writes, Stage 4 reads --
|
||||
Customer[id=5, name=Adele Goldberg, [email protected]]
|
||||
|
||||
-- Stage 4 writes, Stage 3 reads --
|
||||
Customer[id=6, name=Karen Sparck Jones, [email protected]]
|
||||
@@ -0,0 +1,11 @@
|
||||
==============================================================================
|
||||
Deploy 4 (CONTRACT): Stage 4 after the drop, and what breaks if you drop too soon
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:57.618780340Z
|
||||
|
||||
|
||||
-- Stage 4 read, after V3 dropped the email column --
|
||||
Customer[id=1, name=Annie Easley, [email protected]]
|
||||
|
||||
-- Stage 4 create + read, entirely after the drop --
|
||||
Customer[id=2, name=Mary Allen Wilkes, [email protected]]
|
||||
@@ -0,0 +1,9 @@
|
||||
==============================================================================
|
||||
What a lingering Stage 1 instance sees if the drop runs before it is retired
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:58.517481426Z
|
||||
|
||||
Stage 1 create() after V3 dropped "email": org.springframework.jdbc.BadSqlGrammarException
|
||||
message: PreparedStatementCallback; bad SQL grammar [INSERT INTO customers(name, email) VALUES (?, ?)]
|
||||
root cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "EMAIL" not found; SQL statement:
|
||||
INSERT INTO customers(name, email) VALUES (?, ?) [42122-240]
|
||||
@@ -0,0 +1,103 @@
|
||||
=====================================================================
|
||||
Zero-downtime expand-contract: live 4-deploy sequence
|
||||
=====================================================================
|
||||
19:00:53 captured: 2026-09-16T19:00:53Z
|
||||
19:00:53 Starting the database as its own standalone process (not owned by either replica)
|
||||
H2 TCP server up on port 9092 (pid 10861), baseDir /tmp/ec-demo/db
|
||||
19:00:53 Deploy 0: create schema (V1), start two Stage 1 replicas
|
||||
=== Before migrate (target=1) ===
|
||||
00:30:54.634 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:54.702 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:30:54.750 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
|
||||
1 create customer PENDING
|
||||
2 add email address column ABOVE_TARGET
|
||||
3 drop email column ABOVE_TARGET
|
||||
00:30:54.850 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:54.863 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:30:54.874 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
|
||||
00:30:54.877 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.007s)
|
||||
00:30:54.888 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory -- Creating Schema History table "PUBLIC"."flyway_schema_history" ...
|
||||
00:30:54.940 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": << Empty Schema >>
|
||||
00:30:54.956 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "1 - create customer"
|
||||
00:30:55.018 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v1 (execution time 00:00.014s)
|
||||
=== After migrate ===
|
||||
00:30:55.055 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:30:55.063 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column ABOVE_TARGET
|
||||
3 drop email column ABOVE_TARGET
|
||||
Migrations executed: 1, target schema version: 1, success: true
|
||||
started stage 1 on port 8081 (pid 10946)
|
||||
port 8081 healthy
|
||||
started stage 1 on port 8082 (pid 11011)
|
||||
port 8082 healthy
|
||||
19:01:02 load generator running against both replicas
|
||||
19:01:12 baseline soak complete (10s, both replicas on Stage 1)
|
||||
19:01:12 Deploy 1 (EXPAND): migrating to V2 live - zero app restarts
|
||||
=== Before migrate (target=2) ===
|
||||
00:31:13.484 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:13.607 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column PENDING
|
||||
3 drop email column ABOVE_TARGET
|
||||
00:31:13.782 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:13.801 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:31:13.817 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.008s)
|
||||
00:31:13.842 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": 1
|
||||
00:31:13.903 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "2 - add email address column"
|
||||
00:31:14.097 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v2 (execution time 00:00.118s)
|
||||
=== After migrate ===
|
||||
00:31:14.136 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:31:14.174 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column ABOVE_TARGET
|
||||
Migrations executed: 1, target schema version: 2, success: true
|
||||
19:01:19 Deploy 2 (MIGRATE WRITES): rolling restart to Stage 2, replica A first
|
||||
stopped port 8081 (pid 10946)
|
||||
started stage 2 on port 8081 (pid 11226)
|
||||
port 8081 healthy
|
||||
19:01:29 Deploy 2: replica B
|
||||
stopped port 8082 (pid 11011)
|
||||
started stage 2 on port 8082 (pid 11338)
|
||||
port 8082 healthy
|
||||
19:01:45 Deploy 3 (MIGRATE READS): rolling restart to Stage 3, replica A first
|
||||
stopped port 8081 (pid 11226)
|
||||
started stage 3 on port 8081 (pid 11456)
|
||||
port 8081 healthy
|
||||
19:01:55 Deploy 3: replica B
|
||||
stopped port 8082 (pid 11338)
|
||||
started stage 3 on port 8082 (pid 11567)
|
||||
port 8082 healthy
|
||||
19:02:11 Deploy 4a (CONTRACT code): rolling restart to Stage 4, replica A first
|
||||
stopped port 8081 (pid 11456)
|
||||
started stage 4 on port 8081 (pid 11679)
|
||||
port 8081 healthy
|
||||
19:02:20 Deploy 4a: replica B
|
||||
stopped port 8082 (pid 11567)
|
||||
started stage 4 on port 8082 (pid 11784)
|
||||
port 8082 healthy
|
||||
19:02:35 Deploy 4b (CONTRACT schema): migrating to V3 live - drops "email", zero app restarts
|
||||
=== Before migrate (target=latest) ===
|
||||
00:32:36.510 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.574 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column PENDING
|
||||
00:32:36.669 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.680 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
00:32:36.694 [main] INFO org.flywaydb.core.internal.command.DbValidate -- Successfully validated 3 migrations (execution time 00:00.008s)
|
||||
00:32:36.707 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Current version of schema "PUBLIC": 2
|
||||
00:32:36.726 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Migrating schema "PUBLIC" to version "3 - drop email column"
|
||||
00:32:36.795 [main] INFO org.flywaydb.core.internal.command.DbMigrate -- Successfully applied 1 migration to schema "PUBLIC", now at version v3 (execution time 00:00.041s)
|
||||
=== After migrate ===
|
||||
00:32:36.811 [main] INFO org.flywaydb.core.FlywayExecutor -- Database: jdbc:h2:tcp://localhost:9092/expand-contract (H2 2.4)
|
||||
00:32:36.829 [main] WARN org.flywaydb.core.internal.database.base.Database -- Using H2 2.4.240 which is newer than the version Flyway has been verified with. The latest verified version of H2 is 2.3.232.
|
||||
1 create customer SUCCESS
|
||||
2 add email address column SUCCESS
|
||||
3 drop email column SUCCESS
|
||||
Migrations executed: 1, target schema version: 3, success: true
|
||||
19:02:46 final soak complete
|
||||
19:02:46 waiting for the load generator to finish its run...
|
||||
19:03:04 load generator finished
|
||||
19:03:11 === deploy sequence complete ===
|
||||
@@ -0,0 +1,21 @@
|
||||
Load generator summary
|
||||
=======================
|
||||
Total requests: 30911
|
||||
Successful: 30905
|
||||
Errors: 6
|
||||
|
||||
By phase:
|
||||
04a-stage4-soak ok=2130 error=0
|
||||
04b-contract-migration ok=336 error=0
|
||||
04a-deploy-stage4-rollout ok=4147 error=0
|
||||
03-stage3-soak ok=2116 error=0
|
||||
03-deploy-stage3-rollout ok=4645 error=0
|
||||
00-baseline-soak ok=1893 error=0
|
||||
05-final-soak ok=7118 error=4
|
||||
- read-http-404 2
|
||||
- update-http-404 2
|
||||
02-stage2-soak ok=2068 error=0
|
||||
01-expand-migration ok=1773 error=1
|
||||
- read-http-404 1
|
||||
02-deploy-stage2-rollout ok=4679 error=1
|
||||
- update-http-404 1
|
||||
@@ -0,0 +1,18 @@
|
||||
|
||||
-- after Deploy 0 (Stage 1 / Stage 1) --
|
||||
{"appStage":1,"columns":["ID","NAME","EMAIL","CREATED_AT"],"rowCount":0,"rowsWithEmail":0}
|
||||
|
||||
-- after Deploy 1 (schema expanded, both replicas still Stage 1) --
|
||||
{"appStage":1,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":1212,"rowsWithEmail":1212,"rowsWithEmailAddress":1184}
|
||||
|
||||
-- after Deploy 2 (both replicas Stage 2, dual-write live) --
|
||||
{"appStage":2,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":4218,"rowsWithEmail":4218,"rowsWithEmailAddress":2561}
|
||||
|
||||
-- after Deploy 3 (both replicas Stage 3, reading email_address) --
|
||||
{"appStage":3,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":7547,"rowsWithEmail":7547,"rowsWithEmailAddress":6115}
|
||||
|
||||
-- after Deploy 4a (both replicas Stage 4, email column still present but unused) --
|
||||
{"appStage":4,"columns":["ID","NAME","EMAIL","CREATED_AT","EMAIL_ADDRESS"],"rowCount":10642,"rowsWithEmail":9653,"rowsWithEmailAddress":9212}
|
||||
|
||||
-- after Deploy 4b (email column dropped) --
|
||||
{"appStage":4,"columns":["ID","NAME","CREATED_AT","EMAIL_ADDRESS"],"rowCount":11806,"rowsWithEmailAddress":10374}
|
||||
@@ -0,0 +1,14 @@
|
||||
==============================================================================
|
||||
The failure the retry cannot catch: a committed INSERT that ALTER TABLE loses silently
|
||||
==============================================================================
|
||||
captured: 2026-09-16T19:20:58.574465432Z
|
||||
|
||||
attempts needed to reproduce the race: 2 of 20
|
||||
customer creates that returned a generated id with no error: 50
|
||||
customer creates that got the already-documented, already-fixed DDL-collision error: 1
|
||||
of the ids that came back with no error, missing from the table once V2 finished: 8
|
||||
example missing ids: [41, 42, 43, 44, 45]
|
||||
|
||||
This is why the retry in CustomerService cannot be the whole fix: these inserts
|
||||
never threw anything to retry. The row was committed, then discarded when the
|
||||
ADD COLUMN rebuild swapped in a new table that had already been scanned.
|
||||
Reference in New Issue
Block a user