diff --git a/deploy/2026.08.23.01.report.md b/deploy/2026.08.23.01.report.md new file mode 100644 index 00000000..55775ad4 --- /dev/null +++ b/deploy/2026.08.23.01.report.md @@ -0,0 +1,134 @@ +# 2026.08.23.01 — deploy report + +**4 of 6 target hosts succeeded.** `officer-dev` skipped by the plan (the commander runs there) and +verified already current. Two hosts were **not reached at all**, both for access reasons that have nothing +to do with this release: `edge-pertento-bridge` will not complete an SSH handshake, and +`production-pertento-bridge` presented a **changed host key**, which is not something a deploy agent +clears. Neither was pulled, pushed to, or restarted. No host failed a step; nothing came back `errored`. + +Deployed 2026-08-23 from `officer-dev`. Tag `deploy/2026.08.23.01` → `dee0b89b`, matching `package.json` +`2026.08.23.01`. + +**Hosts landed on `e6f1388a`, one commit past the tag.** The plan says `git pull --ff-only`, which follows +`master`, and `master` had moved on by one commit — `plan: localizing the web UI, and the scanner that +sizes it`, three files: `CLAUDE.md`, `docs/localization.md`, `scripts/i18n/scan-hardcoded.ts`. Documentation +and a standalone script, no runtime code, no schema and no dependency. Recorded here because "deployed the +tag" would not be true of any host. + +## Summary + +| Host | HEAD before | HEAD after | Steps | `officer` restart | Schema | +| ---------------------------- | ----------- | ---------- | -------- | ----------------- | ------- | +| `officer-dev` | `e6f1388a` | — | skipped | not touched (237) | ✓ | +| `alpha` | `68d60e40` | `e6f1388a` | all 4 | 66 → 67, online | ✓ | +| `fofos` | `68d60e40` | `e6f1388a` | all 4 | 20 → 21, online | ✓ | +| `macbookpro` | `8b4a5b3b` | `e6f1388a` | all 4 | 7 → 8, online | ✓ | +| `edge-pertento-bridge` | unknown | — | **none** | not touched | unknown | +| `production-pertento-bridge` | unknown | — | **none** | not touched | unknown | +| `deadpool-rezio-bridge` | `68d60e40` | `e6f1388a` | all 4 | 5 → 6, online | ✓ | + +Checkout paths came from each `officer` process's `pm_cwd`, not from a hard-coded list. "Schema ✓" is a +live check, not an inference from `db:push` exiting zero: `command_hq_spawned` exists and +`command_hq_sessions.name` exists, queried per host against that host's own database. + +## Per host + +### `officer-dev` — SKIPPED, and current + +The commander session runs inside `officer-claude-code` here, so the whole host was skipped as the plan +requires: no pull, no `db:push`, no `plugins:update`, no restart. It was already at `e6f1388a` and both +schema objects are present, so it is running this release rather than lagging. `officer` shows 237 restarts, +untouched by this deploy. Nothing is pending on it — the owner restarts it deliberately if they want to. + +### `alpha` — done + +Tree clean, tracked and untracked. Pulled `68d60e40` → `e6f1388a`. `bun db:push` applied changes; +`backfill-session-models` reported 0 seeded. `bun run plugins:update` returned **12/12 updated in 48s**, +exit 0, and restarted all twelve plugin sidecars itself — atelier, bitcoin-wallet, email, gitea, invoices, +jellyfin, music, offscale, photos, soulseek, transmission, vault. Then `pm2 restart officer`: 66 → 67, +`online`. Every other PM2 entry `online`. + +### `fofos` — done + +Tree clean. Pulled `68d60e40` → `e6f1388a`. `db:push` applied changes. `plugins:update`: **1/1 in 8s** +(gitea), which restarted `officer-gitea`. `pm2 restart officer`: 20 → 21, `online`. No trace of the +`dubious ownership` / `EACCES` pair that broke its capabilities sync for three deploys — `capabilities: +pulled` came back clean, as it has since the 2026-08-21 `chown`. + +### `macbookpro` — done + +Awake and reachable. Tree clean. Pulled `8b4a5b3b` → `e6f1388a` — two releases behind, so it picked up +2026.08.22.01 and .02 as well. `db:push` applied changes (3 plugin schemas generated: atelier, kiosk, +music). `plugins:update`: `no plugins installed — nothing to update`, exit 0. `pm2 restart officer` was run +as a **separate** command per `docs/servers.md`: 7 → 8, `online`. + +`PATH` was appended, never replaced — `export PATH="$PATH:/opt/homebrew/bin"` inside the interactive shell, +so `bun` resolves and `pm2` stays where nvm put it. `.zshrc` still fails on a missing `starship` on every +non-interactive login; noise, not a fault, and it has been there for at least three deploys. + +### `deadpool-rezio-bridge` — done + +Logged in as **`br`**, not `pastilhas`. Tree clean. Pulled `68d60e40` → `e6f1388a`. `db:push` applied +changes. `plugins:update`: **2/2 in 8s** (email, offscale), restarting both sidecars. `pm2 restart officer`: +5 → 6, `online`. + +### `edge-pertento-bridge` — NOT REACHED + +**Four attempts, ConnectTimeout up to 60s: `Connection timed out during banner exchange`.** The TCP +connection succeeds — `nc -z 100.64.0.15 22` reports the port open and `ssh -v` logs `Connection +established` — and then sshd never sends its version string. Tailscale shows the node `active; direct`, so +the tunnel is up and this is not a tailnet problem. + +That is a host-side sshd or load problem, not a credential one and not a rate-limit one. Nothing was run +against it: it is still on whatever it was before this release, which I could not read, because reading it +needs the same handshake. It also did not answer HTTPS on its tailnet address during the probe. + +**This host needs a look before the next release**, and the check is one command from anywhere on the +tailnet — `ssh edge-pertento-bridge hostname`. If it still hangs, the box wants a console. + +### `production-pertento-bridge` — NOT REACHED, host key changed + +``` +WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! +The fingerprint for the ED25519 key sent by the remote host is +SHA256:63S0GKa7eqv2a4cGbT7UqsDqupFrdd+RQsUYJ8Mg1ww +Offending ECDSA key in /home/pastilhas/.ssh/known_hosts:37 +``` + +**I did not clear the entry and did not connect.** Accepting a new host key is a trust decision, and a +deploy agent that removes a `known_hosts` line to get its release out has removed the only warning the +owner would ever get. `StrictHostKeyChecking` did exactly what it is for. + +What is known, all read locally without touching the host: + +- `known_hosts` holds three keys for this name: ED25519 `SHA256:dw7kG2MDftuLDT1CduEO+OVYuMVHsx02Cnwikj7GS/4` + (line 33), RSA (line 36), ECDSA (line 37, the one ssh names as offending). +- The host now offers ED25519 `SHA256:63S0GKa7eqv2a4cGbT7UqsDqupFrdd+RQsUYJ8Mg1ww` — a different key from + the ED25519 already trusted. +- Tailscale shows it `active; direct`, port 22 accepts TCP, and it did not answer HTTPS on 100.64.0.12 + during the probe. + +The benign explanation is a rebuild or a regenerated host key — this is `PPROD`, a bridge that has been +reinstalled before. The other explanation is the one the warning is about. **Settle it out of band**: read +`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` on the machine's own console, compare it to +`63S0GK…`, and only then drop the stale lines with +`ssh-keygen -f ~/.ssh/known_hosts -R production-pertento-bridge`. Do not compare it over SSH; that is the +channel in question. + +Until then this host is a release behind — it was at `68d60e40` as of the 2026.08.22.02 report. + +## What I did not do + +- **Did not deploy `edge-pertento-bridge` or `production-pertento-bridge`**, and did not force either. + Neither was pulled, neither had a process restarted, and neither has this release's schema. +- **Did not touch `~/.ssh/known_hosts`.** +- **Did not run `bun install`** — the plan forbids it and `bun.lock` is unchanged across + `deploy/2026.08.22.02..HEAD`. +- **Did not run `bun run docs:check`** — not the commander's job; it is run before the tag. +- **Did not restart anything the plan did not name.** `officer` only, per host. The plugin sidecars that + bounced were restarted by `plugins:update` itself, which is that script's own behaviour: 12 on `alpha`, + 1 on `fofos`, 2 on `deadpool-rezio-bridge`. +- **Did not touch `officer-dev`** at all. + +The local record `data/deploys/2026.08.23.01.done` was written before this report, as the protocol requires: +the record is what prevents a second attempt, and pushing this file is paperwork that is allowed to fail.