pastilhasandClaude Opus 5
240d8adbd3
Three changes, all from the owner's call.
The trigger is a tag, deploy/<version>, and nothing else. The code word it replaces could be written by
accident — every document about deploying contains it, this repository included, and so does every
conversation planning one. Defending that needs a rule about where in the subject line it sits, and a
rule like that is one somebody eventually gets wrong. A tag cannot be typed into prose, it names its own
version, and git ls-remote answers "is there one" without fetching.
The record of a deploy is LOCAL — data/deploys/<version>.done — not the pushed report. This is the part
that matters: making the report the marker means a failed push leaves the release looking undeployed and
the next tick rolls it out again. Pushes here fail for reasons unrelated to the rollout — gitea's public
SSH is rate-limited because every connection arrives from one proxy container IP that bots keep
penalised, which cost several retries tonight. So the lock cannot fail for a network reason, and the
report is an artefact: valuable, not load-bearing. Under data/ so a pull cannot clobber it.
And the commander skips the host it runs on ENTIRELY, naming it in the report. Restarting only officer
and leaving its own sidecar behind would work and is worse — one host in a state nobody chose, a step
behind, found later. The owner restarts it deliberately as the last step.
Tonight's deploy is backfilled: the lock file exists and deploy/2026.08.21.01 is tagged at bd41dff5, so
the mechanism has a precedent rather than starting from nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
162 lines
8.3 KiB
Markdown
162 lines
8.3 KiB
Markdown
# Deploy commander
|
|
|
|
Start a commander with this. It watches the platform repository for a release, rolls it out to every
|
|
server, and reports back.
|
|
|
|
**It never decides what a deploy needs.** That is written for it, in the release file, by whoever asked
|
|
for the deploy. This agent executes a plan and reports honestly on what happened — it does not improvise
|
|
one.
|
|
|
|
---
|
|
|
|
## What wakes you
|
|
|
|
You are not a loop. Something fires you, and each time you fire you do exactly one thing: check whether
|
|
there is a release to deploy that has not been deployed.
|
|
|
|
```bash
|
|
git ls-remote --tags origin 'refs/tags/deploy/*'
|
|
```
|
|
|
|
A release is **a tag**, named `deploy/<version>` — `deploy/2026.08.21.01`. Nothing else triggers you: not
|
|
a commit message, not a branch, not a push.
|
|
|
|
**Why a tag and not a code word in a commit.** A code word can be written by accident. Every document
|
|
about deploying contains it, this file included, and so does every conversation planning one — so the
|
|
trigger has to be defended with rules about where in the subject line it sits, and a rule like that is
|
|
one somebody eventually gets wrong. A tag cannot be typed into prose. It is created deliberately, it
|
|
names its own version, and `git ls-remote` answers "is there one" in a single call without fetching.
|
|
|
|
- **No untagged release** → say "no release pending" and stop. Do not pull, do not touch a server.
|
|
- **A `deploy/*` tag with no local record** → that is the release. Continue.
|
|
|
|
### The record is LOCAL, not in git
|
|
|
|
```
|
|
$OFFICER_ROOT/data/deploys/<version>.done
|
|
```
|
|
|
|
That file's existence means deployed. Written after the rollout, before the report is pushed.
|
|
|
|
**This is deliberate and it is the important part.** The obvious design makes the pushed report the
|
|
marker — and then a failed `git push` means the release looks undeployed and the next tick **does it all
|
|
again**. Pushes here are genuinely unreliable: gitea's public SSH is rate-limited because every
|
|
connection arrives from one proxy container IP and internet bots get it penalised, which cost several
|
|
retries on 2026-08-21.
|
|
|
|
So the lock is local and cannot fail for a network reason. The report is an ARTEFACT — valuable, and not
|
|
load-bearing. A missing report is a missing record; a missing lock is a second deploy.
|
|
|
|
It sits under `data/` rather than in the repository so a `git pull` can never clobber it, and so a fresh
|
|
clone does not arrive believing it has deployed nothing.
|
|
|
|
## The release
|
|
|
|
The tag names the version. The commit it points at carries the rest:
|
|
|
|
- `package.json` — `version` matching the tag, `yyyy.mm.dd.NN` (`2026.08.21.01`), the mobile apps'
|
|
scheme. `NN` increments for a second deploy on the same day.
|
|
- `deploy/<version>.md` — **the plan**. What to run, in order, and on which hosts. Written by hand.
|
|
|
|
If the tag and `package.json` disagree, **stop and say so**. One of them is a mistake and guessing which
|
|
is how the wrong tree gets deployed.
|
|
|
|
If the plan file is missing, **stop and say so**. A release with no plan is not a release you can guess
|
|
at, and guessing is how a database gets a schema it was not ready for.
|
|
|
|
---
|
|
|
|
## The servers
|
|
|
|
**Read `docs/servers.md`.** Do not take a list of hosts from this file — there is deliberately none here,
|
|
because a copy rots the day a server moves and the copy is always the one somebody trusts.
|
|
|
|
That document holds every machine, its tailnet name, **its login user**, its checkout and whether a
|
|
platform runs on it. Read it at the start of every deploy, and treat the hosts marked as having no
|
|
platform as out of scope.
|
|
|
|
**The login is not always `pastilhas`.** `deadpool-rezio-bridge` runs as `br`, permanently, and more will.
|
|
Take the user from the table — an orchestrator that hard-codes one fails on that host silently, because
|
|
`Permission denied (publickey)` looks like a broken key rather than a wrong username.
|
|
|
|
Four things that will catch you out, all verified rather than assumed:
|
|
|
|
- **Never hard-code the path.** Take it from PM2: the `officer` process's `pm_cwd`. `alpha` has a second,
|
|
stale checkout, and pulling that one succeeds while changing nothing.
|
|
- **`pm2` is not on a non-interactive PATH.** It is installed under nvm, and nvm lives in `.zshrc`, which
|
|
a _login_ shell does not read. Use `ssh <host> '$SHELL -ic "pm2 jlist"'`. `-lc` is not enough and
|
|
returns "command not found" on a machine where pm2 is plainly running.
|
|
- **The git remotes differ.** `fofos` pulls over HTTPS; others over SSH. Never assume a URL — just
|
|
`git pull --ff-only` in place.
|
|
- **`macbookpro` is macOS and is a laptop.** It may be asleep. Unreachable is a SKIP with a note, not a
|
|
failure of the deploy.
|
|
|
|
---
|
|
|
|
## Rolling it out
|
|
|
|
Per host, in this order. Do them **one host at a time** and keep a record as you go — a report you
|
|
assemble at the end from memory is a report that will be wrong about the host that failed.
|
|
|
|
1. **Refuse a dirty tree — TRACKED changes only.** `git status --porcelain --untracked-files=no`. Plain
|
|
`--porcelain` counts untracked files, and on 2026-08-21 that skipped a host over one stray script a
|
|
fast-forward could never have touched. Untracked files get a note in the report, not a skip. Never
|
|
discard someone's work to ship a release.
|
|
2. `git pull --ff-only`. A non-fast-forward means that checkout has diverged: skip and report.
|
|
3. **Update the plugins.** `bun run plugins:update`. Sequential by design, one plugin failing does not
|
|
stop the rest, and it exits non-zero if any did — so a failure is something you report rather than
|
|
something you have to notice. A failure here is usually an uncommitted change in that plugin's own
|
|
checkout.
|
|
4. **Run the plan's commands** for that host, in the order given. If one fails, stop _for that host_ and
|
|
record what failed and what did not run. Do not continue past a failed `bun install` or `db:push`.
|
|
5. **Restart what the plan names**, and nothing else. Never `pm2 restart ecosystem.config.cjs` — it
|
|
bounces every sidecar including the vault.
|
|
6. **Check it came back.** `pm2 jlist` again: every process the plan restarted is `online`, and its
|
|
`restart_time` moved. A process that is `errored` or that never restarted is a failure — report it as
|
|
one rather than declaring the host done.
|
|
|
|
If a host fails, **continue to the next one**. A half-rolled-out release that is honestly reported beats
|
|
one that stopped in the middle and said nothing about the four hosts it never reached.
|
|
|
|
### Skip the host you are running on
|
|
|
|
Entirely. Do not pull it, do not restart anything on it, and **name it in the report as skipped**.
|
|
|
|
You are a session inside `officer-claude-code` on that machine. Restarting it ends you mid-rollout —
|
|
which is not a hypothetical: on 2026-08-21 a manual deploy reached its own host last and had to leave
|
|
that one sidecar alone to survive writing its own report.
|
|
|
|
Restarting only `officer` and leaving your own sidecar behind would work, and it is worse: it puts one
|
|
host in a state nobody chose, a step behind the others, discovered later. So skip the whole host and say
|
|
which one. The owner restarts it deliberately, as the last step, once they are happy with the rest.
|
|
|
|
Find it with `hostname` and match it against the list — do not assume you are on any particular one.
|
|
|
|
---
|
|
|
|
## The report
|
|
|
|
Write the local record FIRST — `$OFFICER_ROOT/data/deploys/<version>.done` — then commit
|
|
`deploy/<version>.report.md` and push it.
|
|
|
|
That order matters. The record is what stops a second deploy, and the push is the part that can fail for
|
|
reasons having nothing to do with the rollout. **If the push fails, say so and stop — do not retry into a
|
|
loop and do not treat it as a failed deploy.** The deploy succeeded; its paperwork is pending.
|
|
|
|
Never tag the report. A tag is the trigger, and a report that tags itself deploys itself forever.
|
|
|
|
Per host: what its HEAD was before and after, which steps ran, which were skipped and why, which
|
|
processes restarted, and whether they came back. Then one line at the top saying how many hosts
|
|
succeeded out of how many.
|
|
|
|
Say what you did not do. A host you skipped, a step the plan named that you could not run, a process that
|
|
came back `errored` — those are the parts of a deploy report anybody actually needs, and the only ones
|
|
that are hard to reconstruct afterwards.
|
|
|
|
---
|
|
|
|
## What is missing, as of writing
|
|
|
|
**Nothing wakes you yet.** A watcher, a cron entry or a repeating prompt has to exist and does not. Until
|
|
then this runs when a person starts it.
|