deploy the tag, and stop spending a minute on a host that is known to be down

Two things the 2026.08.23.01 rollout showed.

`edge-pertento-bridge` is down and staying down for some days, and the deploy
agent had no way to know: it spent four attempts and up to sixty seconds apiece
before reporting it unreachable. Correct behaviour against a list that had not
been told, so `docs/servers.md` now has a Deliberately skipped section, with the
host's row kept verbatim inside it so restoring it is one move. A host there is
named as skipped in the report and costs nothing else.

And the procedure said `git pull --ff-only`, which follows `master` rather than
the tag being deployed. A docs commit landed mid-rollout, so every host came up on
`e6f1388a` — one past `deploy/2026.08.23.01` — and "deployed the tag" was true of
none of them. Harmless this time, and it need not have been: it is now fetch plus
`merge --ff-only <tag>`. A host that is AHEAD of the tag is reported, never reset.

Also lands the report itself, which the agent wrote and could not push.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-23 02:12:45 +00:00
co-authored by Claude Opus 5
parent b5d9f591ee
commit 510fa3f524
2 changed files with 34 additions and 2 deletions
+20 -1
View File
@@ -29,13 +29,32 @@ Seven, all running.
| `alpha` | 100.64.0.1 | `pastilhas` | `~/officerdev/platform` | `officer.pastilhas.dev` | | `alpha` | 100.64.0.1 | `pastilhas` | `~/officerdev/platform` | `officer.pastilhas.dev` |
| `fofos` | 100.64.0.17 | `pastilhas` | `~/officerdev/platform` | `officer.fofosfluffybastards.com` | | `fofos` | 100.64.0.17 | `pastilhas` | `~/officerdev/platform` | `officer.fofosfluffybastards.com` |
| `macbookpro` | 100.64.0.9 | `pastilhas` | `~/projects/officer-suite/platform` | `macbookpro.pastilhas.dev` | | `macbookpro` | 100.64.0.9 | `pastilhas` | `~/projects/officer-suite/platform` | `macbookpro.pastilhas.dev` |
| `edge-pertento-bridge` | 100.64.0.15 | `pastilhas` | `~/officerdev/platform` | `edge.pertento.ai` |
| `production-pertento-bridge` | 100.64.0.12 | `pastilhas` | `~/officerdev/platform` | `officer.pertento.ai` | | `production-pertento-bridge` | 100.64.0.12 | `pastilhas` | `~/officerdev/platform` | `officer.pertento.ai` |
| `deadpool-rezio-bridge` | 100.64.0.21 | **`br`** | `/home/br/officerdev/platform` | `officer.rezio.dev` `[pending]` | | `deadpool-rezio-bridge` | 100.64.0.21 | **`br`** | `/home/br/officerdev/platform` | `officer.rezio.dev` `[pending]` |
Only `platform.officer.dev` is public (77.42.39.155). The rest resolve to tailnet addresses and are Only `platform.officer.dev` is public (77.42.39.155). The rest resolve to tailnet addresses and are
reachable from inside only. reachable from inside only.
## Deliberately skipped
A host here is NOT a deploy target. A rollout names it as skipped and moves on — it is not a failure,
and it must not be reported as one.
| Host | Since | Why |
| ---------------------- | ---------- | ---------------------------------------------------------------- |
| `edge-pertento-bridge` | 2026-08-23 | Down, and staying down for some days. Owner's call, not a fault. |
Its row, kept so restoring it is one move:
```
| `edge-pertento-bridge` | 100.64.0.15 | `pastilhas` | `~/officerdev/platform` | `edge.pertento.ai` |
```
On 2026-08-23 the deploy agent spent four attempts and up to 60s per try on this host before reporting
it unreachable — TCP on 22 accepted, then no SSH banner. That work was correct and entirely wasted, which
is what this section exists to prevent: a host that is known to be down should cost a line in a report,
not a retry budget.
### Names that disagree with themselves ### Names that disagree with themselves
Three hosts answer to more than one name, and `tailscale status` is not always the one in use: Three hosts answer to more than one name, and `tailscale status` is not always the one in use:
+14 -1
View File
@@ -136,6 +136,11 @@ That document holds every machine, its tailnet name, **its login user**, its che
platform runs on it. Read it at the start of every deploy, and treat the hosts marked as having no platform runs on it. Read it at the start of every deploy, and treat the hosts marked as having no
platform as out of scope. platform as out of scope.
It also has a **Deliberately skipped** section. A host listed there is not a target: name it as skipped
in the report and spend nothing on it. On 2026-08-23 a host that was known to be down cost four attempts
and a minute apiece before being reported unreachable — correct behaviour against a list that had not been
told. Read that section before you try to reach anything.
**The login is not always `pastilhas`.** `deadpool-rezio-bridge` runs as `br`, permanently, and more will. **The login is not always `pastilhas`.** `deadpool-rezio-bridge` runs as `br`, permanently, and more will.
Take the user from the table — an orchestrator that hard-codes one fails on that host silently, because Take the user from the table — an orchestrator that hard-codes one fails on that host silently, because
`Permission denied (publickey)` looks like a broken key rather than a wrong username. `Permission denied (publickey)` looks like a broken key rather than a wrong username.
@@ -163,7 +168,15 @@ assemble at the end from memory is a report that will be wrong about the host th
`--porcelain` counts untracked files, and on 2026-08-21 that skipped a host over one stray script a `--porcelain` counts untracked files, and on 2026-08-21 that skipped a host over one stray script a
fast-forward could never have touched. Untracked files get a note in the report, not a skip. Never fast-forward could never have touched. Untracked files get a note in the report, not a skip. Never
discard someone's work to ship a release. discard someone's work to ship a release.
2. `git pull --ff-only`. A non-fast-forward means that checkout has diverged: skip and report. 2. **`git fetch origin --tags` then `git merge --ff-only <tag>`** — the TAG you are deploying, not
`master`. `git pull --ff-only` follows the branch, so a commit pushed while you are rolling out lands
on some hosts and not others, and "deployed the tag" stops being true of any of them. That happened on
2026-08-23: every host came up one commit past the tag because a docs commit landed mid-rollout. It was
harmless and it need not have been.
A non-fast-forward means that checkout has diverged: skip and report. A tag that will not fast-forward
because the host is AHEAD is the same answer — report it, do not reset.
3. **Update the plugins.** `bun run plugins:update`. Sequential by design, one plugin failing does not 3. **Update the plugins.** `bun run plugins:update`. Sequential by design, one plugin failing does not
stop the rest, and it exits non-zero if any did — so a failure is something you report rather than stop the rest, and it exits non-zero if any did — so a failure is something you report rather than
something you have to notice. A failure here is usually an uncommitted change in that plugin's own something you have to notice. A failure here is usually an uncommitted change in that plugin's own