deploy: db:push can fail and exit 0, so check the database instead

Verified against a scratch database on 2026-08-23. A push whose foreign key
violated existing rows printed the Postgres error, created the new table, left the
constraint uncreated, and returned zero. `bun db:push` wraps it, so a plan step
that reads `bun db:push` can report success over a half-applied schema.

The rollout already survived this twice only because the agent decided on its own
to verify per host by query rather than trusting the exit code. That instinct is
now the instruction.

Also documents the shape that cannot go through push at all — a new table plus a
foreign key onto it from an existing column, where the seed has to happen between
the two halves — and what direct SQL is allowed to look like when a plan calls for
it: the Drizzle schema stays the source of truth, the SQL comes from the plan
rather than from the agent, it is idempotent, push is run afterwards and expected
to be a no-op as proof the two agree, and every host is verified by query.

And one refusal: never `--force`. Its job is answering the destructive prompts
unattended, which on five hosts holding real data is not a convenience.

This is deploy documentation, not commander doctrine — a developer running
something similar should be able to reach the same conclusions, and the commander
architecture does not need to carry it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-23 03:49:48 +00:00
co-authored by Claude Opus 5
parent b640ec486d
commit 04a4ab33dd
+45
View File
@@ -192,6 +192,51 @@ assemble at the end from memory is a report that will be wrong about the host th
If a host fails, **continue to the next one**. A half-rolled-out release that is honestly reported beats
one that stopped in the middle and said nothing about the four hosts it never reached.
### Schema changes, and why `db:push` cannot be trusted
**`drizzle-kit push` can fail and still exit 0.** Verified on 2026-08-23 against a scratch database: a
change that violated a foreign key printed the Postgres error, created part of the change, left the rest
undone, and returned zero. `bun db:push` wraps it, so a plan step that reads `bun db:push` can report
success while the schema is half-applied.
So on any release whose plan says the schema changed:
**Never take the exit code as the answer. Check the database.** Query for the specific thing the plan
introduced — the table, the column — on that host, against that host's own database. `Changes applied` on
stdout is not evidence, and neither is exit 0. A report that says "schema ✓" without a query behind it is
a guess.
**Some changes cannot go through `db:push` at all, and the plan should say so.** The shape that fails is
create-then-constrain: a new table plus a foreign key onto it from an existing column. `push` creates the
table empty and adds the constraint in the same run, so every existing row violates it. There is no
ordering of `push` invocations that fixes this, because the seed has to happen between the two halves.
**When the plan says the schema needs direct SQL — last resort, and only when the plan says it.**
This is allowed and it is not a licence to improvise. The rules, in order:
1. **The Drizzle schema is the source of truth, always.** The SQL you run exists to make the database match
what the schema code already says. If you find yourself writing SQL the schema does not describe, stop —
that is a code change and it is not yours to make on five production hosts at 3am.
2. **The plan must contain the exact SQL.** Written by whoever cut the release, reviewed before the tag.
Do not compose it yourself from the diff, and do not adapt it per host: same statements, every host, or
the fleet has diverged in a way nothing will detect.
3. **Idempotent, or guarded.** `create table if not exists`, `insert … on conflict do nothing`,
`alter table … add constraint` only after checking `pg_constraint`. A deploy that is safe to run twice
is a deploy that can be retried, and being retryable is what stops a partial rollout becoming a
forensic exercise.
4. **Then run `bun db:push` anyway**, and expect it to be a no-op. That is the proof the two agree: if
`push` still wants to change something after your SQL, the SQL was wrong or incomplete, and that is a
finding for the report rather than something to push past.
5. **Verify per host, by query**, and put the result in the report per host. Not "applied on all five" —
the specific check, and its answer, for each.
6. **A host where the SQL fails is a failed host.** Report it, move to the next, and do not attempt a
repair you invented.
**One thing to refuse outright:** `--force`. It exists to answer prompts unattended, and the prompts it
answers are the destructive ones. On five hosts holding real data, a flag whose whole job is to say yes to
"this will truncate a table" is not a convenience. If a change needs `--force` to apply, it needs a human.
### Skip the host you are running on
Entirely. Do not pull it, do not restart anything on it, and **name it in the report as skipped**.