Verified against a scratch database on 2026-08-23. A push whose foreign key
violated existing rows printed the Postgres error, created the new table, left the
constraint uncreated, and returned zero. `bun db:push` wraps it, so a plan step
that reads `bun db:push` can report success over a half-applied schema.
The rollout already survived this twice only because the agent decided on its own
to verify per host by query rather than trusting the exit code. That instinct is
now the instruction.
Also documents the shape that cannot go through push at all — a new table plus a
foreign key onto it from an existing column, where the seed has to happen between
the two halves — and what direct SQL is allowed to look like when a plan calls for
it: the Drizzle schema stays the source of truth, the SQL comes from the plan
rather than from the agent, it is idempotent, push is run afterwards and expected
to be a no-op as proof the two agree, and every host is verified by query.
And one refusal: never `--force`. Its job is answering the destructive prompts
unattended, which on five hosts holding real data is not a convenience.
This is deploy documentation, not commander doctrine — a developer running
something similar should be able to reach the same conclusions, and the commander
architecture does not need to carry it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things the 2026.08.23.01 rollout showed.
`edge-pertento-bridge` is down and staying down for some days, and the deploy
agent had no way to know: it spent four attempts and up to sixty seconds apiece
before reporting it unreachable. Correct behaviour against a list that had not
been told, so `docs/servers.md` now has a Deliberately skipped section, with the
host's row kept verbatim inside it so restoring it is one move. A host there is
named as skipped in the report and costs nothing else.
And the procedure said `git pull --ff-only`, which follows `master` rather than
the tag being deployed. A docs commit landed mid-rollout, so every host came up on
`e6f1388a` — one past `deploy/2026.08.23.01` — and "deployed the tag" was true of
none of them. Harmless this time, and it need not have been: it is now fetch plus
`merge --ff-only <tag>`. A host that is AHEAD of the tag is reported, never reset.
Also lands the report itself, which the agent wrote and could not push.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Update the docs before a deploy" is a step that gets skipped at two in the morning, and its absence is
invisible. Every piece of drift found today was mechanical — a count, a route name, a process list — so it is
checkable, and a check that exits non-zero cannot be quietly not done.
It found two things on its first run, both wrong for over a week:
platform/CLAUDE.md "27 entries as of 2026-08-13" the registry holds 18
officer-dev/CLAUDE.md "Officer is twenty PM2 entries" there are 13
The second was worse than a wrong number. It listed eleven processes that do not run here — vault, slskd,
headscale, transmission, invoiceshelf, wallet, photos, jellyfin, gitea, music, vnc — while omitting six that
do, and named two mybiblepal entries as sharing the daemon when nothing does. Accurate on 2026-08-07, and
nobody re-derives a list of processes while editing the sentence around it. Both are corrected, with what
they used to claim recorded, because the interesting part is how a maintained list decays rather than that it
did.
Three checks: the permission registry count, the PM2 entry count, and every settings route Spotlight offers
resolving to a section some screen declares. That last one is the class the counts cannot catch — a name, not
a number — and it is the drift that shipped a dead link to /settings/users.
THE RULE FOR ADDING ONE, written into the file: a check compares a documented claim against a value DERIVED
AT RUN TIME, never against a second hard-coded copy. A check with the answer written into it is one more
place to drift, and it drifts in the same edit that misses the doc.
What it deliberately cannot do is read prose and judge it. It looks only at claims with a machine-checkable
counterpart — a small fraction of the documentation, and the fraction that rots fastest.
The settings check parses source rather than importing it, and says why: those sections are object literals
inside components, so importing them drags React and a DOM into a CLI.
`skipped:` lines are printed rather than swallowed — a machine without pm2 cannot check that count, and
saying so beats reporting a pass it did not perform.
Recorded as required before tagging in CLAUDE.md, and in prompts/deploy-commander.md as explicitly NOT the
rollout agent's job: it executes a plan, and a prose edit between two hosts is not that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deploy watcher kept logging `Connection closed by 77.42.39.155 port 22` followed by "make sure you have
the correct access rights", which reads as a credentials problem and is not one. `origin` pointed at
gitea.officer.dev, which resolves to the public address and arrives through Nginx Proxy Manager — where
every connection appears to come from one container IP, internet bots get that IP penalised by
srclimit_penalise, and SSH is refused for long stretches.
Gitea is a plugin ON THIS MACHINE, listening on 100.64.0.13:10023. The watcher was leaving the box and
coming back in through a proxy to read a repository sitting on its own disk.
origin now points at the local address; the public URL is kept as `gitea-public` so nothing is lost. No file
was edited to do it, which matters — the watcher is a running bash script and editing one in place is how a
loop starts executing half of two versions. Changing the remote's URL takes effect on the next tick with
nothing restarted.
Loopback would be better still and is not used: 127.0.0.1 fails host key verification because it has no
known_hosts entry, while the tailnet address is already trusted.
The cost of not knowing this, recorded in the prompt: a tag pushed at 21:47 on 2026-08-21 went unnoticed
until 23:07. Eighty minutes, forty-odd ticks, every one of them correctly logged and retried. The loop was
never at fault — it was asking through a door that is sometimes shut.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first watcher had an unbounded retry in it. A release whose deploy failed left no .done file, so the
next poll two minutes later spawned a fresh headless agent, and the one after that, all night, each one
costing money. The loop was careful and its logging was honest; the defect was structural.
It is structural because the loop is written FRESH by an agent every time one arms. A rule that lives only
in the prompt gets re-implemented from scratch each time, and the wrong implementation is the one that
looks right: test the .done file. That file only exists on success, so testing it is exactly the retry
storm.
So scripts/deploy-claim.sh owns it. `claim <version>` exits 0 once and 1 for ever after, whatever became
of the attempt that held it — a failed release is not retried, it is looked at. Handing the claim back is
`rm -rf` by a person, on purpose; a loop cannot undo its own claim, which is what makes the storm
impossible rather than unlikely.
Three choices in it worth keeping:
- `mkdir` is the claim. One atomic syscall that both tests and takes, with no window between checking and
acting — two pollers a second apart is not hypothetical when a watcher gets restarted.
- Bash, not bun. The thing that stops a retry storm must not need a healthy checkout; a deploy that dies
on `bun install` is precisely when it has to work.
- A version is digits and dots or it is refused. It arrives off a git tag and is concatenated into a path.
A pre-existing .done with no claim beside it still counts as deployed, so deploy/2026.08.21.01 — rolled
out by hand before any of this existed — is not re-attempted.
docs/servers.md corrected from the commander's own report, which was right and specific: on macbookpro
APPEND /opt/homebrew/bin for bun rather than replacing the PATH (this file said "do not help the PATH",
which is right for pm2 and is what made plugins:update fail there), `ls` is aliased to an eza that is not
installed, plugins:update bounces every sidecar before the plan's own restart step, and fofos cannot
publish affordances for what is probably the same ownership problem as its capabilities sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes, all from the owner's call.
The trigger is a tag, deploy/<version>, and nothing else. The code word it replaces could be written by
accident — every document about deploying contains it, this repository included, and so does every
conversation planning one. Defending that needs a rule about where in the subject line it sits, and a
rule like that is one somebody eventually gets wrong. A tag cannot be typed into prose, it names its own
version, and git ls-remote answers "is there one" without fetching.
The record of a deploy is LOCAL — data/deploys/<version>.done — not the pushed report. This is the part
that matters: making the report the marker means a failed push leaves the release looking undeployed and
the next tick rolls it out again. Pushes here fail for reasons unrelated to the rollout — gitea's public
SSH is rate-limited because every connection arrives from one proxy container IP that bots keep
penalised, which cost several retries tonight. So the lock cannot fail for a network reason, and the
report is an artefact: valuable, not load-bearing. Under data/ so a pull cannot clobber it.
And the commander skips the host it runs on ENTIRELY, naming it in the report. Restarting only officer
and leaving its own sidecar behind would work and is worse — one host in a state nobody chose, a step
behind, found later. The owner restarts it deliberately as the last step.
Tonight's deploy is backfilled: the lock file exists and deploy/2026.08.21.01 is tagged at bd41dff5, so
the mechanism has a precedent rather than starting from nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deadpool-rezio-bridge joins the list: 100.64.0.21, checkout /home/br/officerdev/platform, deployed by
hand and verified — 7/7 processes online, username NOT NULL, uq_users_username_lower present, HEAD at
bd41dff5.
Its login is `br` and always will be. That makes the SSH user a per-host fact rather than a constant,
so the table carries it and the deploy prompt reads it. Worth stating plainly because the failure is
misleading: an orchestrator assuming `pastilhas` gets "Permission denied (publickey)", which reads as a
broken key rather than a wrong username.
Also folded in what tonight established. Three hosts answer to names that disagree with
`tailscale status` — edge-bridge and gitea-bridge are the old ones — so the table says which to use.
`edge-bridge` was listed as "not SSH-reachable" and that was wrong: its SSH was down at the moment I
checked, not absent. The pm2/nvm PATH trap is written in, including that `-lc` is not enough and cost
two wrong conclusions, and that macbookpro must not be helped with an exported PATH because doing so
breaks pm2. And a dirty tree means TRACKED changes; plain --porcelain skipped a host over a stray file.
officer.rezio.dev is recorded under "known wrong, waiting on someone else": it points at gateway rather
than deadpool, which is worse than unavailable — it will serve the wrong box the moment gateway answers
on 443.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updating plugins was only possible through POST /api/plugins/update-all, which is authenticated — so a
deploy had to mint an API key to update the machine it was already logged into. A credential created,
used and hopefully deleted, on every host, for a local operation.
Same work through the same functions with the HTTP layer removed, not a second implementation beside
it. Sequential like the route (each update pushes a schema, republishes assets, rewrites the ecosystem
file and rebuilds the frontend — two at once is a race over one file each), one plugin's failure does
not stop the rest, and it exits non-zero if any failed so a deploy can stop rather than report a
success it did not have. Verified: 3/3 in 18s.
Its header says the one thing that is not obvious — it runs OUTSIDE the server process, so files,
schema, assets and the SPA bundle are current when it finishes but the route table is not, until
something restarts officer. A deploy restarts anyway, which is exactly why that would otherwise be
found out rather than known.
The prompt had a table of six hosts under a line saying docs/servers.md is authoritative. Both cannot
be true, and the copy is the one people trust while it rots. Removed, with the reason written in its
place so it does not come back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The agent watches for a commit whose subject BEGINS with the code word, rolls it out to the six
platforms in docs/servers.md, and reports back. It never decides what a deploy needs — the plan is
written for it in deploy/<version>.md, and an agent improvising a db:push is how a database gets a
schema nobody was ready for.
Four traps written in rather than left to be rediscovered, all verified tonight: take the checkout path
from PM2's pm_cwd because alpha has a stale second checkout that pulls successfully and changes
nothing; reach pm2 with `$SHELL -ic` because it lives under nvm in .zshrc and a login shell does not
read that, which is how I concluded twice that a machine had no pm2 while it was plainly running;
remotes differ per host so pull in place rather than assuming a URL; and macbookpro is a laptop that
may be asleep, which is a skip and not a failure.
Idempotence is the report's own existence — deploy/<version>.report.md on origin means deployed. A
watcher that restarts must not roll the same release out twice, and nothing else was standing between
it and doing so. The report's subject deliberately omits the code word, or a report would trigger a
deploy and loop.
Two gaps stated rather than papered over: there is no CLI for updating plugins, only an authenticated
route, so the plan has to say how or the report has to say it was skipped; and nothing wakes the agent
yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pastilhasandClaude Opus 5