Files
platform/todo/fleet/an-orchestrator-that-acts-on-every-server-at.md
T
pastilhasandClaude Opus 5 530e5880e1 todo: what tonight's deploy work left behind, and two items it closed
Closed:

- fleet/an-orchestrator-that-acts-on-every-server-at — built, and as none of the three shapes it
  proposed. Not a script, not a screen, not a chat capability: an agent handed a written procedure and a
  server list it has to read rather than one baked into it. All three verified facts in the item earned
  their place; two things it did not anticipate are now new items.
- fleet/a-way-to-mark-a-commit-this-one-ships-so-a — built as a TAG, which the item explicitly ruled out.
  The owner reversed it, and the argument that changed his mind is recorded: a codeword can be written by
  accident, because every document about deploying contains it, so the trigger needs rules about where in
  a subject line it sits and a rule like that is one somebody eventually gets wrong.

Opened, all from tonight and all with the evidence attached:

- a deploy verifies pm2 and not that the app serves anything — the commander raised this against itself
- the commander cannot finish its own host, so one machine is always a build behind
- the watcher loop is rewritten by an agent every arming, so its correctness is re-rolled each time
- plugins:update restarts every sidecar, so a report naming only officer understates what moved
- an agent spawned by a background job is invisible in the UI while being perfectly readable
- AskUserQuestion can fail to parse its own call
- a commander opened from /chat loses its commander controls, because isCommander comes from the screen
  rather than the session

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 20:27:00 +00:00

47 lines
2.7 KiB
Markdown

---
id: fleet/an-orchestrator-that-acts-on-every-server-at
title: "An orchestrator that acts on every server at once"
theme: fleet
status: done
created: 2026-08-20
date_source: assumed
order: 1
---
**An orchestrator that acts on every server at once** — pull and update every platform, restart a
process everywhere, report which host is behind. The list is the input; this is the thing that
reads it.
Three facts from writing the list decide the design, and all three are verified: the checkout
**path differs per host** (take it from PM2's `pm_cwd` for the `officer` process, never a
hard-coded path); `alpha` carries a **second, stale checkout** so pulling the wrong one succeeds
while changing nothing; and the **git remotes differ** — one host pulls over HTTPS and one points
at a different Gitea altogether. An orchestrator built on "ssh, cd to the same path, git pull"
is wrong on three counts before it starts.
Open: whether this is a script over SSH, an Officer screen, or a chat capability; and whether
`headscale` and `omega` — checkout present, nothing running — are targets at all. Note the
platform already has an **Update button** for a single server (`03abdeb2`); this is that, fanned
out, and it should probably share the code rather than reimplement the pull.
---
**Done 2026-08-21, and not as any of the three shapes this item proposed.** It is not a script, not a
screen and not a chat capability — it is an **agent** given a written procedure
(`prompts/deploy-commander.md`) and a list it has to read rather than a list baked into it. Tonight it
rolled `2026.08.21.02` and `2026.08.21.03` across six hosts unattended and wrote its own reports.
All three verified facts in this item survived contact and all three earned their place: it takes each
checkout from PM2's `pm_cwd`, so `alpha`'s stale second checkout was never touched; it pulls in place
rather than assuming a remote; and it takes the login user per host, so `deadpool` was reached as `br`.
Two things this item did not anticipate, both now in the procedure. It must **skip its own host entirely**
— restarting `officer` there kills the rollout mid-flight — which leaves that one machine needing a
deliberate restart afterwards (`todo/fleet/the-deploy-commander-cannot-finish-its-own-host`). And a
release needs **one attempt only**: the first watcher retried a failed deploy every two minutes, so that
rule is now `scripts/deploy-claim.sh` rather than a paragraph anyone re-implements.
It does **not** share code with the single-server Update button. Worth revisiting, but the overlap turned
out to be smaller than expected: the button restarts a process, and almost all of this is the reading,
skipping and reporting around that.