The deploy watcher kept logging `Connection closed by 77.42.39.155 port 22` followed by "make sure you have
the correct access rights", which reads as a credentials problem and is not one. `origin` pointed at
gitea.officer.dev, which resolves to the public address and arrives through Nginx Proxy Manager — where
every connection appears to come from one container IP, internet bots get that IP penalised by
srclimit_penalise, and SSH is refused for long stretches.
Gitea is a plugin ON THIS MACHINE, listening on 100.64.0.13:10023. The watcher was leaving the box and
coming back in through a proxy to read a repository sitting on its own disk.
origin now points at the local address; the public URL is kept as `gitea-public` so nothing is lost. No file
was edited to do it, which matters — the watcher is a running bash script and editing one in place is how a
loop starts executing half of two versions. Changing the remote's URL takes effect on the next tick with
nothing restarted.
Loopback would be better still and is not used: 127.0.0.1 fails host key verification because it has no
known_hosts entry, while the tailnet address is already trusted.
The cost of not knowing this, recorded in the prompt: a tag pushed at 21:47 on 2026-08-21 went unnoticed
until 23:07. Eighty minutes, forty-odd ticks, every one of them correctly logged and retried. The loop was
never at fault — it was asking through a door that is sometimes shut.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first watcher had an unbounded retry in it. A release whose deploy failed left no .done file, so the
next poll two minutes later spawned a fresh headless agent, and the one after that, all night, each one
costing money. The loop was careful and its logging was honest; the defect was structural.
It is structural because the loop is written FRESH by an agent every time one arms. A rule that lives only
in the prompt gets re-implemented from scratch each time, and the wrong implementation is the one that
looks right: test the .done file. That file only exists on success, so testing it is exactly the retry
storm.
So scripts/deploy-claim.sh owns it. `claim <version>` exits 0 once and 1 for ever after, whatever became
of the attempt that held it — a failed release is not retried, it is looked at. Handing the claim back is
`rm -rf` by a person, on purpose; a loop cannot undo its own claim, which is what makes the storm
impossible rather than unlikely.
Three choices in it worth keeping:
- `mkdir` is the claim. One atomic syscall that both tests and takes, with no window between checking and
acting — two pollers a second apart is not hypothetical when a watcher gets restarted.
- Bash, not bun. The thing that stops a retry storm must not need a healthy checkout; a deploy that dies
on `bun install` is precisely when it has to work.
- A version is digits and dots or it is refused. It arrives off a git tag and is concatenated into a path.
A pre-existing .done with no claim beside it still counts as deployed, so deploy/2026.08.21.01 — rolled
out by hand before any of this existed — is not re-attempted.
docs/servers.md corrected from the commander's own report, which was right and specific: on macbookpro
APPEND /opt/homebrew/bin for bun rather than replacing the PATH (this file said "do not help the PATH",
which is right for pm2 and is what made plugins:update fail there), `ls` is aliased to an eza that is not
installed, plugins:update bounces every sidecar before the plan's own restart step, and fofos cannot
publish affordances for what is probably the same ownership problem as its capabilities sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes, all from the owner's call.
The trigger is a tag, deploy/<version>, and nothing else. The code word it replaces could be written by
accident — every document about deploying contains it, this repository included, and so does every
conversation planning one. Defending that needs a rule about where in the subject line it sits, and a
rule like that is one somebody eventually gets wrong. A tag cannot be typed into prose, it names its own
version, and git ls-remote answers "is there one" without fetching.
The record of a deploy is LOCAL — data/deploys/<version>.done — not the pushed report. This is the part
that matters: making the report the marker means a failed push leaves the release looking undeployed and
the next tick rolls it out again. Pushes here fail for reasons unrelated to the rollout — gitea's public
SSH is rate-limited because every connection arrives from one proxy container IP that bots keep
penalised, which cost several retries tonight. So the lock cannot fail for a network reason, and the
report is an artefact: valuable, not load-bearing. Under data/ so a pull cannot clobber it.
And the commander skips the host it runs on ENTIRELY, naming it in the report. Restarting only officer
and leaving its own sidecar behind would work and is worse — one host in a state nobody chose, a step
behind, found later. The owner restarts it deliberately as the last step.
Tonight's deploy is backfilled: the lock file exists and deploy/2026.08.21.01 is tagged at bd41dff5, so
the mechanism has a precedent rather than starting from nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deadpool-rezio-bridge joins the list: 100.64.0.21, checkout /home/br/officerdev/platform, deployed by
hand and verified — 7/7 processes online, username NOT NULL, uq_users_username_lower present, HEAD at
bd41dff5.
Its login is `br` and always will be. That makes the SSH user a per-host fact rather than a constant,
so the table carries it and the deploy prompt reads it. Worth stating plainly because the failure is
misleading: an orchestrator assuming `pastilhas` gets "Permission denied (publickey)", which reads as a
broken key rather than a wrong username.
Also folded in what tonight established. Three hosts answer to names that disagree with
`tailscale status` — edge-bridge and gitea-bridge are the old ones — so the table says which to use.
`edge-bridge` was listed as "not SSH-reachable" and that was wrong: its SSH was down at the moment I
checked, not absent. The pm2/nvm PATH trap is written in, including that `-lc` is not enough and cost
two wrong conclusions, and that macbookpro must not be helped with an exported PATH because doing so
breaks pm2. And a dirty tree means TRACKED changes; plain --porcelain skipped a host over a stray file.
officer.rezio.dev is recorded under "known wrong, waiting on someone else": it points at gateway rather
than deadpool, which is worse than unavailable — it will serve the wrong box the moment gateway answers
on 443.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updating plugins was only possible through POST /api/plugins/update-all, which is authenticated — so a
deploy had to mint an API key to update the machine it was already logged into. A credential created,
used and hopefully deleted, on every host, for a local operation.
Same work through the same functions with the HTTP layer removed, not a second implementation beside
it. Sequential like the route (each update pushes a schema, republishes assets, rewrites the ecosystem
file and rebuilds the frontend — two at once is a race over one file each), one plugin's failure does
not stop the rest, and it exits non-zero if any failed so a deploy can stop rather than report a
success it did not have. Verified: 3/3 in 18s.
Its header says the one thing that is not obvious — it runs OUTSIDE the server process, so files,
schema, assets and the SPA bundle are current when it finishes but the route table is not, until
something restarts officer. A deploy restarts anyway, which is exactly why that would otherwise be
found out rather than known.
The prompt had a table of six hosts under a line saying docs/servers.md is authoritative. Both cannot
be true, and the copy is the one people trust while it rots. Removed, with the reason written in its
place so it does not come back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The agent watches for a commit whose subject BEGINS with the code word, rolls it out to the six
platforms in docs/servers.md, and reports back. It never decides what a deploy needs — the plan is
written for it in deploy/<version>.md, and an agent improvising a db:push is how a database gets a
schema nobody was ready for.
Four traps written in rather than left to be rediscovered, all verified tonight: take the checkout path
from PM2's pm_cwd because alpha has a stale second checkout that pulls successfully and changes
nothing; reach pm2 with `$SHELL -ic` because it lives under nvm in .zshrc and a login shell does not
read that, which is how I concluded twice that a machine had no pm2 while it was plainly running;
remotes differ per host so pull in place rather than assuming a URL; and macbookpro is a laptop that
may be asleep, which is a skip and not a failure.
Idempotence is the report's own existence — deploy/<version>.report.md on origin means deployed. A
watcher that restarts must not roll the same release out twice, and nothing else was standing between
it and doing so. The report's subject deliberately omits the code word, or a report would trigger a
deploy and loop.
Two gaps stated rather than papered over: there is no CLI for updating plugins, only an authenticated
route, so the plan has to say how or the report has to say it was skipped; and nothing wakes the agent
yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pastilhasandClaude Opus 5