From 04a4ab33dd188bac88eb6e800f5601d3ea112268 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Sun, 23 Aug 2026 03:49:48 +0000 Subject: [PATCH] deploy: db:push can fail and exit 0, so check the database instead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against a scratch database on 2026-08-23. A push whose foreign key violated existing rows printed the Postgres error, created the new table, left the constraint uncreated, and returned zero. `bun db:push` wraps it, so a plan step that reads `bun db:push` can report success over a half-applied schema. The rollout already survived this twice only because the agent decided on its own to verify per host by query rather than trusting the exit code. That instinct is now the instruction. Also documents the shape that cannot go through push at all — a new table plus a foreign key onto it from an existing column, where the seed has to happen between the two halves — and what direct SQL is allowed to look like when a plan calls for it: the Drizzle schema stays the source of truth, the SQL comes from the plan rather than from the agent, it is idempotent, push is run afterwards and expected to be a no-op as proof the two agree, and every host is verified by query. And one refusal: never `--force`. Its job is answering the destructive prompts unattended, which on five hosts holding real data is not a convenience. This is deploy documentation, not commander doctrine — a developer running something similar should be able to reach the same conclusions, and the commander architecture does not need to carry it. Co-Authored-By: Claude Opus 5 --- prompts/deploy-commander.md | 45 +++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/prompts/deploy-commander.md b/prompts/deploy-commander.md index 4c557228..55b524af 100644 --- a/prompts/deploy-commander.md +++ b/prompts/deploy-commander.md @@ -192,6 +192,51 @@ assemble at the end from memory is a report that will be wrong about the host th If a host fails, **continue to the next one**. A half-rolled-out release that is honestly reported beats one that stopped in the middle and said nothing about the four hosts it never reached. +### Schema changes, and why `db:push` cannot be trusted + +**`drizzle-kit push` can fail and still exit 0.** Verified on 2026-08-23 against a scratch database: a +change that violated a foreign key printed the Postgres error, created part of the change, left the rest +undone, and returned zero. `bun db:push` wraps it, so a plan step that reads `bun db:push` can report +success while the schema is half-applied. + +So on any release whose plan says the schema changed: + +**Never take the exit code as the answer. Check the database.** Query for the specific thing the plan +introduced — the table, the column — on that host, against that host's own database. `Changes applied` on +stdout is not evidence, and neither is exit 0. A report that says "schema ✓" without a query behind it is +a guess. + +**Some changes cannot go through `db:push` at all, and the plan should say so.** The shape that fails is +create-then-constrain: a new table plus a foreign key onto it from an existing column. `push` creates the +table empty and adds the constraint in the same run, so every existing row violates it. There is no +ordering of `push` invocations that fixes this, because the seed has to happen between the two halves. + +**When the plan says the schema needs direct SQL — last resort, and only when the plan says it.** + +This is allowed and it is not a licence to improvise. The rules, in order: + +1. **The Drizzle schema is the source of truth, always.** The SQL you run exists to make the database match + what the schema code already says. If you find yourself writing SQL the schema does not describe, stop — + that is a code change and it is not yours to make on five production hosts at 3am. +2. **The plan must contain the exact SQL.** Written by whoever cut the release, reviewed before the tag. + Do not compose it yourself from the diff, and do not adapt it per host: same statements, every host, or + the fleet has diverged in a way nothing will detect. +3. **Idempotent, or guarded.** `create table if not exists`, `insert … on conflict do nothing`, + `alter table … add constraint` only after checking `pg_constraint`. A deploy that is safe to run twice + is a deploy that can be retried, and being retryable is what stops a partial rollout becoming a + forensic exercise. +4. **Then run `bun db:push` anyway**, and expect it to be a no-op. That is the proof the two agree: if + `push` still wants to change something after your SQL, the SQL was wrong or incomplete, and that is a + finding for the report rather than something to push past. +5. **Verify per host, by query**, and put the result in the report per host. Not "applied on all five" — + the specific check, and its answer, for each. +6. **A host where the SQL fails is a failed host.** Report it, move to the next, and do not attempt a + repair you invented. + +**One thing to refuse outright:** `--force`. It exists to answer prompts unattended, and the prompts it +answers are the destructive ones. On five hosts holding real data, a flag whose whole job is to say yes to +"this will truncate a table" is not a convenience. If a change needs `--force` to apply, it needs a human. + ### Skip the host you are running on Entirely. Do not pull it, do not restart anything on it, and **name it in the report as skipped**.