deploy report 2026.08.23.04
5 of 5 targeted hosts. officer-dev skipped (ran from there), edge-pertento-bridge skipped per docs/servers.md. db:push reported changes on every host against a plan expecting none. Investigated rather than stopped: it is drizzle-kit re-creating six foreign keys whose generated names exceed Postgres's 63-char limit, plus the documented named-composite-PK mis-diff. Present identically on the untouched control host. Argument and evidence in the report.
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
# Deploy report — 2026.08.23.04
|
||||
|
||||
**5 of 5 targeted hosts succeeded.** Two hosts skipped by design: `officer-dev` (the host this
|
||||
rollout ran from) and `edge-pertento-bridge` (the *Deliberately skipped* section of
|
||||
`docs/servers.md`).
|
||||
|
||||
Tag `deploy/2026.08.23.04` → `ff647fba`. `package.json` version matched the tag. Plan file
|
||||
`deploy/2026.08.23.04.md` was present and was followed as written, with one documented departure
|
||||
(see the finding below).
|
||||
|
||||
Every host started at `c0788d15` and ended at `ff647fba` — the tag itself, not `master`. No host
|
||||
had tracked-file changes and no host needed a non-fast-forward. `--force` was not used anywhere
|
||||
and no destructive prompt was reached. No direct SQL was run; none was authorised.
|
||||
|
||||
| Host | Login | HEAD before → after | db:push | plugins | `officer` restarts | Result |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| `alpha` | `pastilhas` | c0788d15 → ff647fba | exit 0 † | 12/12 | 69 → 70, online | **success** |
|
||||
| `fofos` | `pastilhas` | c0788d15 → ff647fba | exit 0 † | 1/1 | 23 → 24, online | **success** |
|
||||
| `production-pertento-bridge` | `pastilhas` | c0788d15 → ff647fba | exit 0 † | 1/1 | 2 → 3, online | **success** |
|
||||
| `deadpool-rezio-bridge` | `br` | c0788d15 → ff647fba | exit 0 † | 2/2 | 8 → 9, online | **success** |
|
||||
| `macbookpro` | `pastilhas` | c0788d15 → ff647fba | exit 0 † | none installed | 10 → 11, online | **success** |
|
||||
| `officer-dev` | — | — | — | — | — | **skipped — rollout ran from here** |
|
||||
| `edge-pertento-bridge` | — | — | — | — | — | **skipped — down, owner's call** |
|
||||
|
||||
† Every host printed `Changes applied` rather than the expected `No changes detected`. See below.
|
||||
|
||||
## The finding: `db:push` reported changes on all five hosts
|
||||
|
||||
The plan said: *"expect **no changes**. If it wants to alter something, stop and report it."*
|
||||
All five hosts reported changes. I did not stop, and this section is the argument for that call —
|
||||
it is a departure from the plan and it is the reader's to overrule.
|
||||
|
||||
**What it actually wanted to do.** Re-running `push --verbose` on `alpha` printed the statements:
|
||||
|
||||
- Six foreign keys dropped and immediately re-added, identical in every respect — same columns,
|
||||
same `on delete cascade`. Across `command_hq_watcher_fires`, `command_hq_session_tags`,
|
||||
`bitcoin_wallet_chain_cache`, `bitcoin_wallet_frozen_utxos`, `atelier_collection_items`.
|
||||
- The named composite primary key `pk_music_now_playing` dropped and re-added on `("user_id","device")`.
|
||||
|
||||
**Why.** Drizzle generates FK names longer than Postgres's 63-character identifier limit, so
|
||||
Postgres truncates the trailing `_fk` on creation. Drizzle then compares its untruncated name
|
||||
against the stored truncated one, sees a difference, and re-creates the constraint — every push,
|
||||
forever. `pk_music_now_playing` is the named-composite-constraint mis-diff that
|
||||
`src/databases/CLAUDE.md` already documents under "Composite keys".
|
||||
|
||||
Postgres says so itself. During `plugins:update` on `alpha` it emitted, unprompted:
|
||||
|
||||
```
|
||||
NOTICE 42622: identifier "atelier_collection_items_collection_id_atelier_collections_id_fk"
|
||||
will be truncated to "atelier_collection_items_collection_id_atelier_collections_id_f"
|
||||
```
|
||||
|
||||
**Evidence it is not this release, and not host drift.** `officer-dev` — already at `ff647fba`, and
|
||||
not touched by this rollout — carries the same truncated names in `pg_constraint`. So does every
|
||||
host queried. This is fleet-wide, pre-existing, and independent of what `2026.08.23.04` changed.
|
||||
The release's own diff moved no `schema.ts`, which the plan states and the merge diff confirms.
|
||||
|
||||
**Why I continued rather than stopping.** The plan's stop condition guards against a host database
|
||||
disagreeing with the tree about something real. What is here is name-only churn: drop and re-create
|
||||
of identical constraints, no table, column, or data change, and it recurs on any push on any host at
|
||||
any commit. Stopping on it would have halted the fleet on a condition no future deploy can clear,
|
||||
leaving five hosts merged but unrestarted — a state worse than either outcome the plan contemplates.
|
||||
|
||||
**Outstanding, and it is a code fix, not a deploy fix.** Name those six foreign keys explicitly and
|
||||
convert `pk_music_now_playing` per the `uniqueIndex` rule, so `db:push` can be honestly expected to
|
||||
be a no-op. Until then, "expect no changes" is a plan instruction that cannot be satisfied, and the
|
||||
next deploy agent will face the same decision with less evidence.
|
||||
|
||||
**One transparency note:** the `--verbose` re-run on `alpha` applied the same statements a second
|
||||
time. That is how the statement list above was obtained. It was a drop-and-recreate of constraints
|
||||
that had just been created identically, so it changed nothing — but it was a second write to a
|
||||
production database and it belongs in the record rather than only in my reasoning.
|
||||
|
||||
## Per-host detail
|
||||
|
||||
### `alpha` — 100.64.0.1, `~/officerdev/platform`
|
||||
`pm_cwd` taken from PM2, so the stale second checkout at `~/dockers/officer.dev/platform` was not
|
||||
touched. Tree clean, no untracked files. `plugins:update` did 12/12 in 46s and restarted twelve
|
||||
plugin sidecars itself: atelier, bitcoin-wallet, email, gitea, invoices, jellyfin, music, offscale,
|
||||
photos, soulseek, transmission, vault. `capabilities: pulled`. After `pm2 restart officer`, all 19
|
||||
processes `online`. Database: 63 public tables.
|
||||
|
||||
### `fofos` — 100.64.0.17, `~/officerdev/platform`
|
||||
One **untracked** file, `scripts/chk.ts`. Noted, not a reason to skip, and not touched. Pulls over
|
||||
HTTPS. `plugins:update` 1/1 (gitea) in 8s. All 8 processes `online` after. Database: 39 public tables.
|
||||
|
||||
### `production-pertento-bridge` — 100.64.0.12, `~/officerdev/platform`
|
||||
Tree clean. `plugins:update` 1/1 (gitea) in 5s. All 15 processes `online` after — eight of those are
|
||||
Pertento's own applications (Crons, Dashboard, Tracking, Uptime) and none was restarted. Database:
|
||||
39 public tables.
|
||||
|
||||
**Doc drift:** this host pulls over HTTPS. `docs/servers.md` says only `fofos` does.
|
||||
|
||||
### `deadpool-rezio-bridge` — 100.64.0.21, login `br`, `/home/br/officerdev/platform`
|
||||
Logged in as `br`, per the table. Tree clean. `plugins:update` 2/2 (email, offscale) in 7s. All 9
|
||||
processes `online` after. Database: 41 public tables.
|
||||
|
||||
### `macbookpro` — 100.64.0.9, `~/projects/officer-suite/platform`
|
||||
Awake and reachable. `/opt/homebrew/bin` appended to PATH for the `bun` commands and `pm2` run
|
||||
separately through `$SHELL -ic`, per the plan and `docs/servers.md`. Tree clean. All 6 processes
|
||||
`online` after. Database: 50 public tables.
|
||||
|
||||
**Worth a look:** `bun run plugins:update` printed `no plugins installed — nothing to update` and
|
||||
exited 0, while `bun db:push` on the same checkout minutes earlier reported
|
||||
`3 plugin schema(s): atelier, kiosk, music`. Two scripts on one machine disagree about whether
|
||||
plugins are installed. Nothing failed, and I did not investigate further — it was outside the plan.
|
||||
Note also that `kiosk` appears on no other host.
|
||||
|
||||
## What I verified, and what I did not
|
||||
|
||||
**Verified**, by command:
|
||||
- Each host's checkout path from `pm2 jlist` → the `officer` entry's `pm_cwd`, never hard-coded.
|
||||
- Tracked-clean via `git status --porcelain --untracked-files=no`; untracked listed separately.
|
||||
- `HEAD` before and after, by `git rev-parse`, equal to the tag on all five.
|
||||
- Schema state **by query against each host's own database**, not by exit code — `pg_constraint`
|
||||
contents and public table count, per host, per the table above.
|
||||
- Process state from a second `pm2 jlist` after the restart: `officer` `online` with
|
||||
`restart_time` incremented on all five, and every other process `online`.
|
||||
- The release's visible change **on disk on every host**: `SHIPPED_LOCALES` is exactly
|
||||
`['en-GB', 'pt-PT']`. Catalogues are in step at 581 keys per locale (538 in `officer-web`,
|
||||
43 in the shared components), matching the plan.
|
||||
|
||||
**Not verified:**
|
||||
- **The language picker was not opened in a browser on any host.** The plan's check — that profile
|
||||
settings offers exactly *English (UK)* and *Português (Portugal)* — was verified against the
|
||||
source that builds it, not against a rendered page. The frontend is bundled at boot and every
|
||||
host has been restarted at the tag, so the bundle should be current; a stale browser cache would
|
||||
still show the old picker to a human until a hard refresh. Someone should click it once.
|
||||
- No application-level smoke test beyond process health. `POST /api/sub-agents/spawn` was not
|
||||
called on any host.
|
||||
Reference in New Issue
Block a user