pastilhas
b5d9f591ee
edge-pertento-bridge never completes an SSH banner exchange; production-pertento-bridge presented a changed host key and was left alone. Neither was pulled or restarted.
135 lines
8.1 KiB
Markdown
135 lines
8.1 KiB
Markdown
# 2026.08.23.01 — deploy report
|
|
|
|
**4 of 6 target hosts succeeded.** `officer-dev` skipped by the plan (the commander runs there) and
|
|
verified already current. Two hosts were **not reached at all**, both for access reasons that have nothing
|
|
to do with this release: `edge-pertento-bridge` will not complete an SSH handshake, and
|
|
`production-pertento-bridge` presented a **changed host key**, which is not something a deploy agent
|
|
clears. Neither was pulled, pushed to, or restarted. No host failed a step; nothing came back `errored`.
|
|
|
|
Deployed 2026-08-23 from `officer-dev`. Tag `deploy/2026.08.23.01` → `dee0b89b`, matching `package.json`
|
|
`2026.08.23.01`.
|
|
|
|
**Hosts landed on `e6f1388a`, one commit past the tag.** The plan says `git pull --ff-only`, which follows
|
|
`master`, and `master` had moved on by one commit — `plan: localizing the web UI, and the scanner that
|
|
sizes it`, three files: `CLAUDE.md`, `docs/localization.md`, `scripts/i18n/scan-hardcoded.ts`. Documentation
|
|
and a standalone script, no runtime code, no schema and no dependency. Recorded here because "deployed the
|
|
tag" would not be true of any host.
|
|
|
|
## Summary
|
|
|
|
| Host | HEAD before | HEAD after | Steps | `officer` restart | Schema |
|
|
| ---------------------------- | ----------- | ---------- | -------- | ----------------- | ------- |
|
|
| `officer-dev` | `e6f1388a` | — | skipped | not touched (237) | ✓ |
|
|
| `alpha` | `68d60e40` | `e6f1388a` | all 4 | 66 → 67, online | ✓ |
|
|
| `fofos` | `68d60e40` | `e6f1388a` | all 4 | 20 → 21, online | ✓ |
|
|
| `macbookpro` | `8b4a5b3b` | `e6f1388a` | all 4 | 7 → 8, online | ✓ |
|
|
| `edge-pertento-bridge` | unknown | — | **none** | not touched | unknown |
|
|
| `production-pertento-bridge` | unknown | — | **none** | not touched | unknown |
|
|
| `deadpool-rezio-bridge` | `68d60e40` | `e6f1388a` | all 4 | 5 → 6, online | ✓ |
|
|
|
|
Checkout paths came from each `officer` process's `pm_cwd`, not from a hard-coded list. "Schema ✓" is a
|
|
live check, not an inference from `db:push` exiting zero: `command_hq_spawned` exists and
|
|
`command_hq_sessions.name` exists, queried per host against that host's own database.
|
|
|
|
## Per host
|
|
|
|
### `officer-dev` — SKIPPED, and current
|
|
|
|
The commander session runs inside `officer-claude-code` here, so the whole host was skipped as the plan
|
|
requires: no pull, no `db:push`, no `plugins:update`, no restart. It was already at `e6f1388a` and both
|
|
schema objects are present, so it is running this release rather than lagging. `officer` shows 237 restarts,
|
|
untouched by this deploy. Nothing is pending on it — the owner restarts it deliberately if they want to.
|
|
|
|
### `alpha` — done
|
|
|
|
Tree clean, tracked and untracked. Pulled `68d60e40` → `e6f1388a`. `bun db:push` applied changes;
|
|
`backfill-session-models` reported 0 seeded. `bun run plugins:update` returned **12/12 updated in 48s**,
|
|
exit 0, and restarted all twelve plugin sidecars itself — atelier, bitcoin-wallet, email, gitea, invoices,
|
|
jellyfin, music, offscale, photos, soulseek, transmission, vault. Then `pm2 restart officer`: 66 → 67,
|
|
`online`. Every other PM2 entry `online`.
|
|
|
|
### `fofos` — done
|
|
|
|
Tree clean. Pulled `68d60e40` → `e6f1388a`. `db:push` applied changes. `plugins:update`: **1/1 in 8s**
|
|
(gitea), which restarted `officer-gitea`. `pm2 restart officer`: 20 → 21, `online`. No trace of the
|
|
`dubious ownership` / `EACCES` pair that broke its capabilities sync for three deploys — `capabilities:
|
|
pulled` came back clean, as it has since the 2026-08-21 `chown`.
|
|
|
|
### `macbookpro` — done
|
|
|
|
Awake and reachable. Tree clean. Pulled `8b4a5b3b` → `e6f1388a` — two releases behind, so it picked up
|
|
2026.08.22.01 and .02 as well. `db:push` applied changes (3 plugin schemas generated: atelier, kiosk,
|
|
music). `plugins:update`: `no plugins installed — nothing to update`, exit 0. `pm2 restart officer` was run
|
|
as a **separate** command per `docs/servers.md`: 7 → 8, `online`.
|
|
|
|
`PATH` was appended, never replaced — `export PATH="$PATH:/opt/homebrew/bin"` inside the interactive shell,
|
|
so `bun` resolves and `pm2` stays where nvm put it. `.zshrc` still fails on a missing `starship` on every
|
|
non-interactive login; noise, not a fault, and it has been there for at least three deploys.
|
|
|
|
### `deadpool-rezio-bridge` — done
|
|
|
|
Logged in as **`br`**, not `pastilhas`. Tree clean. Pulled `68d60e40` → `e6f1388a`. `db:push` applied
|
|
changes. `plugins:update`: **2/2 in 8s** (email, offscale), restarting both sidecars. `pm2 restart officer`:
|
|
5 → 6, `online`.
|
|
|
|
### `edge-pertento-bridge` — NOT REACHED
|
|
|
|
**Four attempts, ConnectTimeout up to 60s: `Connection timed out during banner exchange`.** The TCP
|
|
connection succeeds — `nc -z 100.64.0.15 22` reports the port open and `ssh -v` logs `Connection
|
|
established` — and then sshd never sends its version string. Tailscale shows the node `active; direct`, so
|
|
the tunnel is up and this is not a tailnet problem.
|
|
|
|
That is a host-side sshd or load problem, not a credential one and not a rate-limit one. Nothing was run
|
|
against it: it is still on whatever it was before this release, which I could not read, because reading it
|
|
needs the same handshake. It also did not answer HTTPS on its tailnet address during the probe.
|
|
|
|
**This host needs a look before the next release**, and the check is one command from anywhere on the
|
|
tailnet — `ssh edge-pertento-bridge hostname`. If it still hangs, the box wants a console.
|
|
|
|
### `production-pertento-bridge` — NOT REACHED, host key changed
|
|
|
|
```
|
|
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
|
|
The fingerprint for the ED25519 key sent by the remote host is
|
|
SHA256:63S0GKa7eqv2a4cGbT7UqsDqupFrdd+RQsUYJ8Mg1ww
|
|
Offending ECDSA key in /home/pastilhas/.ssh/known_hosts:37
|
|
```
|
|
|
|
**I did not clear the entry and did not connect.** Accepting a new host key is a trust decision, and a
|
|
deploy agent that removes a `known_hosts` line to get its release out has removed the only warning the
|
|
owner would ever get. `StrictHostKeyChecking` did exactly what it is for.
|
|
|
|
What is known, all read locally without touching the host:
|
|
|
|
- `known_hosts` holds three keys for this name: ED25519 `SHA256:dw7kG2MDftuLDT1CduEO+OVYuMVHsx02Cnwikj7GS/4`
|
|
(line 33), RSA (line 36), ECDSA (line 37, the one ssh names as offending).
|
|
- The host now offers ED25519 `SHA256:63S0GKa7eqv2a4cGbT7UqsDqupFrdd+RQsUYJ8Mg1ww` — a different key from
|
|
the ED25519 already trusted.
|
|
- Tailscale shows it `active; direct`, port 22 accepts TCP, and it did not answer HTTPS on 100.64.0.12
|
|
during the probe.
|
|
|
|
The benign explanation is a rebuild or a regenerated host key — this is `PPROD`, a bridge that has been
|
|
reinstalled before. The other explanation is the one the warning is about. **Settle it out of band**: read
|
|
`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` on the machine's own console, compare it to
|
|
`63S0GK…`, and only then drop the stale lines with
|
|
`ssh-keygen -f ~/.ssh/known_hosts -R production-pertento-bridge`. Do not compare it over SSH; that is the
|
|
channel in question.
|
|
|
|
Until then this host is a release behind — it was at `68d60e40` as of the 2026.08.22.02 report.
|
|
|
|
## What I did not do
|
|
|
|
- **Did not deploy `edge-pertento-bridge` or `production-pertento-bridge`**, and did not force either.
|
|
Neither was pulled, neither had a process restarted, and neither has this release's schema.
|
|
- **Did not touch `~/.ssh/known_hosts`.**
|
|
- **Did not run `bun install`** — the plan forbids it and `bun.lock` is unchanged across
|
|
`deploy/2026.08.22.02..HEAD`.
|
|
- **Did not run `bun run docs:check`** — not the commander's job; it is run before the tag.
|
|
- **Did not restart anything the plan did not name.** `officer` only, per host. The plugin sidecars that
|
|
bounced were restarted by `plugins:update` itself, which is that script's own behaviour: 12 on `alpha`,
|
|
1 on `fofos`, 2 on `deadpool-rezio-bridge`.
|
|
- **Did not touch `officer-dev`** at all.
|
|
|
|
The local record `data/deploys/2026.08.23.01.done` was written before this report, as the protocol requires:
|
|
the record is what prevents a second attempt, and pushing this file is paperwork that is allowed to fail.
|