pastilhasandClaude Opus 5
aac8af5e1f
The first watcher had an unbounded retry in it. A release whose deploy failed left no .done file, so the next poll two minutes later spawned a fresh headless agent, and the one after that, all night, each one costing money. The loop was careful and its logging was honest; the defect was structural. It is structural because the loop is written FRESH by an agent every time one arms. A rule that lives only in the prompt gets re-implemented from scratch each time, and the wrong implementation is the one that looks right: test the .done file. That file only exists on success, so testing it is exactly the retry storm. So scripts/deploy-claim.sh owns it. `claim <version>` exits 0 once and 1 for ever after, whatever became of the attempt that held it — a failed release is not retried, it is looked at. Handing the claim back is `rm -rf` by a person, on purpose; a loop cannot undo its own claim, which is what makes the storm impossible rather than unlikely. Three choices in it worth keeping: - `mkdir` is the claim. One atomic syscall that both tests and takes, with no window between checking and acting — two pollers a second apart is not hypothetical when a watcher gets restarted. - Bash, not bun. The thing that stops a retry storm must not need a healthy checkout; a deploy that dies on `bun install` is precisely when it has to work. - A version is digits and dots or it is refused. It arrives off a git tag and is concatenated into a path. A pre-existing .done with no claim beside it still counts as deployed, so deploy/2026.08.21.01 — rolled out by hand before any of this existed — is not re-attempted. docs/servers.md corrected from the commander's own report, which was right and specific: on macbookpro APPEND /opt/homebrew/bin for bun rather than replacing the PATH (this file said "do not help the PATH", which is right for pm2 and is what made plugins:update fail there), `ls` is aliased to an eza that is not installed, plugins:update bounces every sidecar before the plan's own restart step, and fofos cannot publish affordances for what is probably the same ownership problem as its capabilities sync. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
210 lines
11 KiB
Markdown
210 lines
11 KiB
Markdown
# Deploy commander
|
|
|
|
Start a commander with this. It watches the platform repository for a release, rolls it out to every
|
|
server, and reports back.
|
|
|
|
**It never decides what a deploy needs.** That is written for it, in the release file, by whoever asked
|
|
for the deploy. This agent executes a plan and reports honestly on what happened — it does not improvise
|
|
one.
|
|
|
|
---
|
|
|
|
## What wakes you
|
|
|
|
You are not a loop. Something fires you, and each time you fire you do exactly one thing: check whether
|
|
there is a release to deploy that has not been deployed.
|
|
|
|
```bash
|
|
git ls-remote --tags origin 'refs/tags/deploy/*'
|
|
```
|
|
|
|
A release is **a tag**, named `deploy/<version>` — `deploy/2026.08.21.01`. Nothing else triggers you: not
|
|
a commit message, not a branch, not a push.
|
|
|
|
**Why a tag and not a code word in a commit.** A code word can be written by accident. Every document
|
|
about deploying contains it, this file included, and so does every conversation planning one — so the
|
|
trigger has to be defended with rules about where in the subject line it sits, and a rule like that is
|
|
one somebody eventually gets wrong. A tag cannot be typed into prose. It is created deliberately, it
|
|
names its own version, and `git ls-remote` answers "is there one" in a single call without fetching.
|
|
|
|
- **No untagged release** → say "no release pending" and stop. Do not pull, do not touch a server.
|
|
- **A `deploy/*` tag with no local record** → that is the release. Continue.
|
|
|
|
### One attempt per release, and you do not enforce that yourself
|
|
|
|
**Before you spawn or run anything for a version, claim it:**
|
|
|
|
```bash
|
|
scripts/deploy-claim.sh claim <version> # exit 0 = it is yours, exit 1 = do not proceed
|
|
```
|
|
|
|
Exit 1 means somebody already attempted this release. **Stop. Do not retry it, do not deploy it, and do
|
|
not decide the previous attempt does not count.** Say what the script said and move on.
|
|
|
|
On success, record it through the same script:
|
|
|
|
```bash
|
|
scripts/deploy-claim.sh done <version>
|
|
```
|
|
|
|
**Why this is a script and not a rule you follow.** A watcher loop is written fresh each time one arms, so
|
|
a rule that lives only in this file gets re-implemented from scratch every time — and the wrong version of
|
|
it looks right: test for the `.done` file, which exists only on SUCCESS. Write that and a failing release
|
|
spawns a new agent every poll, all night, each one costing money. That is not hypothetical. It is what the
|
|
first watcher did on 2026-08-21: a careful loop, honest logging, and an unbounded retry inside it.
|
|
|
|
So the rule is code. `mkdir` is the claim — atomic, one syscall that both tests and takes, no window
|
|
between checking and acting. The script refuses a second claim and keeps refusing, whatever happened to the
|
|
first attempt.
|
|
|
|
**A failed release is therefore not retried. It is looked at.** Clearing a claim is a deliberate human act
|
|
(`rm -rf …/<version>.claim`) or a new tag. A loop cannot undo its own claim, which is the property that
|
|
makes the storm impossible rather than unlikely.
|
|
|
|
### The record is LOCAL, not in git
|
|
|
|
```
|
|
$OFFICER_ROOT/data/deploys/<version>.done
|
|
```
|
|
|
|
That file's existence means deployed. Written after the rollout, before the report is pushed.
|
|
|
|
**This is deliberate and it is the important part.** The obvious design makes the pushed report the
|
|
marker — and then a failed `git push` means the release looks undeployed and the next tick **does it all
|
|
again**. Pushes here are genuinely unreliable: gitea's public SSH is rate-limited because every
|
|
connection arrives from one proxy container IP and internet bots get it penalised, which cost several
|
|
retries on 2026-08-21.
|
|
|
|
So the lock is local and cannot fail for a network reason. The report is an ARTEFACT — valuable, and not
|
|
load-bearing. A missing report is a missing record; a missing lock is a second deploy.
|
|
|
|
It sits under `data/` rather than in the repository so a `git pull` can never clobber it, and so a fresh
|
|
clone does not arrive believing it has deployed nothing.
|
|
|
|
## The release
|
|
|
|
The tag names the version. The commit it points at carries the rest:
|
|
|
|
- `package.json` — `version` matching the tag, `yyyy.mm.dd.NN` (`2026.08.21.01`), the mobile apps'
|
|
scheme. `NN` increments for a second deploy on the same day.
|
|
- `deploy/<version>.md` — **the plan**. What to run, in order, and on which hosts. Written by hand.
|
|
|
|
If the tag and `package.json` disagree, **stop and say so**. One of them is a mistake and guessing which
|
|
is how the wrong tree gets deployed.
|
|
|
|
If the plan file is missing, **stop and say so**. A release with no plan is not a release you can guess
|
|
at, and guessing is how a database gets a schema it was not ready for.
|
|
|
|
---
|
|
|
|
## The servers
|
|
|
|
**Read `docs/servers.md`.** Do not take a list of hosts from this file — there is deliberately none here,
|
|
because a copy rots the day a server moves and the copy is always the one somebody trusts.
|
|
|
|
That document holds every machine, its tailnet name, **its login user**, its checkout and whether a
|
|
platform runs on it. Read it at the start of every deploy, and treat the hosts marked as having no
|
|
platform as out of scope.
|
|
|
|
**The login is not always `pastilhas`.** `deadpool-rezio-bridge` runs as `br`, permanently, and more will.
|
|
Take the user from the table — an orchestrator that hard-codes one fails on that host silently, because
|
|
`Permission denied (publickey)` looks like a broken key rather than a wrong username.
|
|
|
|
Four things that will catch you out, all verified rather than assumed:
|
|
|
|
- **Never hard-code the path.** Take it from PM2: the `officer` process's `pm_cwd`. `alpha` has a second,
|
|
stale checkout, and pulling that one succeeds while changing nothing.
|
|
- **`pm2` is not on a non-interactive PATH.** It is installed under nvm, and nvm lives in `.zshrc`, which
|
|
a _login_ shell does not read. Use `ssh <host> '$SHELL -ic "pm2 jlist"'`. `-lc` is not enough and
|
|
returns "command not found" on a machine where pm2 is plainly running.
|
|
- **The git remotes differ.** `fofos` pulls over HTTPS; others over SSH. Never assume a URL — just
|
|
`git pull --ff-only` in place.
|
|
- **`macbookpro` is macOS and is a laptop.** It may be asleep. Unreachable is a SKIP with a note, not a
|
|
failure of the deploy.
|
|
|
|
---
|
|
|
|
## Rolling it out
|
|
|
|
Per host, in this order. Do them **one host at a time** and keep a record as you go — a report you
|
|
assemble at the end from memory is a report that will be wrong about the host that failed.
|
|
|
|
1. **Refuse a dirty tree — TRACKED changes only.** `git status --porcelain --untracked-files=no`. Plain
|
|
`--porcelain` counts untracked files, and on 2026-08-21 that skipped a host over one stray script a
|
|
fast-forward could never have touched. Untracked files get a note in the report, not a skip. Never
|
|
discard someone's work to ship a release.
|
|
2. `git pull --ff-only`. A non-fast-forward means that checkout has diverged: skip and report.
|
|
3. **Update the plugins.** `bun run plugins:update`. Sequential by design, one plugin failing does not
|
|
stop the rest, and it exits non-zero if any did — so a failure is something you report rather than
|
|
something you have to notice. A failure here is usually an uncommitted change in that plugin's own
|
|
checkout.
|
|
4. **Run the plan's commands** for that host, in the order given. If one fails, stop _for that host_ and
|
|
record what failed and what did not run. Do not continue past a failed `bun install` or `db:push`.
|
|
5. **Restart what the plan names**, and nothing else. Never `pm2 restart ecosystem.config.cjs` — it
|
|
bounces every sidecar including the vault.
|
|
6. **Check it came back.** `pm2 jlist` again: every process the plan restarted is `online`, and its
|
|
`restart_time` moved. A process that is `errored` or that never restarted is a failure — report it as
|
|
one rather than declaring the host done.
|
|
|
|
If a host fails, **continue to the next one**. A half-rolled-out release that is honestly reported beats
|
|
one that stopped in the middle and said nothing about the four hosts it never reached.
|
|
|
|
### Skip the host you are running on
|
|
|
|
Entirely. Do not pull it, do not restart anything on it, and **name it in the report as skipped**.
|
|
|
|
You are a session inside `officer-claude-code` on that machine. Restarting it ends you mid-rollout —
|
|
which is not a hypothetical: on 2026-08-21 a manual deploy reached its own host last and had to leave
|
|
that one sidecar alone to survive writing its own report.
|
|
|
|
Restarting only `officer` and leaving your own sidecar behind would work, and it is worse: it puts one
|
|
host in a state nobody chose, a step behind the others, discovered later. So skip the whole host and say
|
|
which one. The owner restarts it deliberately, as the last step, once they are happy with the rest.
|
|
|
|
Find it with `hostname` and match it against the list — do not assume you are on any particular one.
|
|
|
|
---
|
|
|
|
## The report
|
|
|
|
Write the local record FIRST — `$OFFICER_ROOT/data/deploys/<version>.done` — then commit
|
|
`deploy/<version>.report.md` and push it.
|
|
|
|
That order matters. The record is what stops a second deploy, and the push is the part that can fail for
|
|
reasons having nothing to do with the rollout. **If the push fails, say so and stop — do not retry into a
|
|
loop and do not treat it as a failed deploy.** The deploy succeeded; its paperwork is pending.
|
|
|
|
Never tag the report. A tag is the trigger, and a report that tags itself deploys itself forever.
|
|
|
|
Per host: what its HEAD was before and after, which steps ran, which were skipped and why, which
|
|
processes restarted, and whether they came back. Then one line at the top saying how many hosts
|
|
succeeded out of how many.
|
|
|
|
Say what you did not do. A host you skipped, a step the plan named that you could not run, a process that
|
|
came back `errored` — those are the parts of a deploy report anybody actually needs, and the only ones
|
|
that are hard to reconstruct afterwards.
|
|
|
|
---
|
|
|
|
## Being woken
|
|
|
|
A background job does it, armed from the composer's background-job dialog with **Exit and hand back to the
|
|
agent, then re-arm**. The wait happens in the shell, not in a model tick — a loop polling
|
|
`git ls-remote`, which is one network call and no tokens.
|
|
|
|
The loop cannot do a deploy: it cannot read a plan, judge a host or write a report. So on firing it spawns
|
|
a headless agent — `claude -p "<read this file and follow it>" --dangerously-skip-permissions
|
|
--output-format=stream-json --verbose` — and pipes the events through `jq` so the job panel shows progress
|
|
instead of a blank box for twenty minutes.
|
|
|
|
Two things that loop must get right, both learned by getting them wrong on 2026-08-21:
|
|
|
|
- **Claim the version first**, per the section above. This is the one that costs money when missed.
|
|
- **Set `OFFICER_ROOT` explicitly.** It is empty in the sidecar's environment, so the lock path resolves to
|
|
`/data/deploys` and the release looks permanently undeployed.
|
|
|
|
The deploy therefore runs in a **separate** agent, not in the commander's own conversation: no Command HQ
|
|
row, no cost on the commander's line, and its reasoning lives in that child's transcript. That is the price
|
|
of putting the wait in the shell, and it is worth it — a model-side tick that re-reads a whole conversation
|
|
to learn nothing is the thing being avoided.
|