todo: what tonight's deploy work left behind, and two items it closed
Closed: - fleet/an-orchestrator-that-acts-on-every-server-at — built, and as none of the three shapes it proposed. Not a script, not a screen, not a chat capability: an agent handed a written procedure and a server list it has to read rather than one baked into it. All three verified facts in the item earned their place; two things it did not anticipate are now new items. - fleet/a-way-to-mark-a-commit-this-one-ships-so-a — built as a TAG, which the item explicitly ruled out. The owner reversed it, and the argument that changed his mind is recorded: a codeword can be written by accident, because every document about deploying contains it, so the trigger needs rules about where in a subject line it sits and a rule like that is one somebody eventually gets wrong. Opened, all from tonight and all with the evidence attached: - a deploy verifies pm2 and not that the app serves anything — the commander raised this against itself - the commander cannot finish its own host, so one machine is always a build behind - the watcher loop is rewritten by an agent every arming, so its correctness is re-rolled each time - plugins:update restarts every sidecar, so a report naming only officer understates what moved - an agent spawned by a background job is invisible in the UI while being perfectly readable - AskUserQuestion can fail to parse its own call - a commander opened from /chat loses its commander controls, because isCommander comes from the screen rather than the session Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: fleet/a-deploy-report-verifies-pm2-not-that-the-app
|
||||
title: 'A deploy verifies PM2, not that the app still works'
|
||||
theme: fleet
|
||||
status: open
|
||||
created: 2026-08-21
|
||||
date_source: stated
|
||||
order: 4
|
||||
---
|
||||
|
||||
**A deploy verifies PM2, not that the app still works.** The deploy commander raised this against itself,
|
||||
unprompted, in its own report: _"Nothing was verified beyond `pm2`."_ That is the most valuable line in
|
||||
either report so far and it should not be true twice more.
|
||||
|
||||
What step 6 checks today is that the process it restarted is `online` and that `restart_time` moved. That
|
||||
catches a crash on boot. It does not catch a server that starts, listens, and serves a 500 — or one that
|
||||
serves a stale bundle, which is the exact failure that cost an hour on 2026-08-21 when a rebuilt frontend
|
||||
was never picked up because the running process still held the previous `index.gen.html`.
|
||||
|
||||
The check that would have caught all of it is one line and needs nothing built:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/
|
||||
```
|
||||
|
||||
Two things to settle rather than assume. **The port differs per host** — take it from that host's `.env`
|
||||
(`OFFICERDEV_PORT`), not from a constant; `officer-dev` is 9000 and nothing guarantees the others are.
|
||||
And **a 200 on `/` is a low bar**: it proves the SPA shell is served. An endpoint that touches Postgres
|
||||
would prove more, and choosing one is the actual design question here — probably `/api/health` if it
|
||||
exists, and if it does not, that is the smaller item to do first.
|
||||
|
||||
Worth adding to the plan format too, so a release can name its own smoke test rather than relying on a
|
||||
generic one.
|
||||
@@ -2,7 +2,7 @@
|
||||
id: fleet/a-way-to-mark-a-commit-this-one-ships-so-a
|
||||
title: "A way to mark a commit \"this one ships\", so a release is not just `master`'s tip"
|
||||
theme: fleet
|
||||
status: open
|
||||
status: done
|
||||
created: 2026-08-21
|
||||
date_source: stated
|
||||
order: 2
|
||||
@@ -29,3 +29,22 @@ Shape to settle when it is built:
|
||||
be a chosen one.
|
||||
- The updater then does a checkout at that SHA, not a `pull` — same discipline the plugin-pinning
|
||||
item asks for: **verify `rev-parse HEAD` matches, or refuse.**
|
||||
|
||||
---
|
||||
|
||||
**Done 2026-08-21 — as a tag, which is what this item ruled out.** The owner reversed it on being shown
|
||||
the trade: "we don't even need a code word if we tag with deploy/date and the version". `deploy/<version>`,
|
||||
e.g. `deploy/2026.08.21.02`.
|
||||
|
||||
The argument that changed it was not about refs at all. A codeword **can be written by accident**: every
|
||||
document about deploying contains it, this file included, and so does every conversation planning one — so
|
||||
the trigger needs rules about where in the subject line it sits, and a rule like that is one somebody
|
||||
eventually gets wrong. A tag cannot be typed into prose. It also names its own version, which a codeword
|
||||
does not, and `git ls-remote --tags` answers "is there one" without fetching.
|
||||
|
||||
The cost this item correctly identified is real and was accepted: a tag is a second ref to mint and push.
|
||||
It is two commands in the release recipe.
|
||||
|
||||
What replaced the "un-marking is forward-only" worry: the deploy is idempotent by a **local** record, not
|
||||
by the ref. `scripts/deploy-claim.sh` takes one claim per version and never gives it back to a machine, so
|
||||
a tag being moved or deleted cannot cause a second rollout.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
id: fleet/an-orchestrator-that-acts-on-every-server-at
|
||||
title: "An orchestrator that acts on every server at once"
|
||||
theme: fleet
|
||||
status: open
|
||||
status: done
|
||||
created: 2026-08-20
|
||||
date_source: assumed
|
||||
order: 1
|
||||
@@ -23,3 +23,24 @@ Open: whether this is a script over SSH, an Officer screen, or a chat capability
|
||||
`headscale` and `omega` — checkout present, nothing running — are targets at all. Note the
|
||||
platform already has an **Update button** for a single server (`03abdeb2`); this is that, fanned
|
||||
out, and it should probably share the code rather than reimplement the pull.
|
||||
|
||||
---
|
||||
|
||||
**Done 2026-08-21, and not as any of the three shapes this item proposed.** It is not a script, not a
|
||||
screen and not a chat capability — it is an **agent** given a written procedure
|
||||
(`prompts/deploy-commander.md`) and a list it has to read rather than a list baked into it. Tonight it
|
||||
rolled `2026.08.21.02` and `2026.08.21.03` across six hosts unattended and wrote its own reports.
|
||||
|
||||
All three verified facts in this item survived contact and all three earned their place: it takes each
|
||||
checkout from PM2's `pm_cwd`, so `alpha`'s stale second checkout was never touched; it pulls in place
|
||||
rather than assuming a remote; and it takes the login user per host, so `deadpool` was reached as `br`.
|
||||
|
||||
Two things this item did not anticipate, both now in the procedure. It must **skip its own host entirely**
|
||||
— restarting `officer` there kills the rollout mid-flight — which leaves that one machine needing a
|
||||
deliberate restart afterwards (`todo/fleet/the-deploy-commander-cannot-finish-its-own-host`). And a
|
||||
release needs **one attempt only**: the first watcher retried a failed deploy every two minutes, so that
|
||||
rule is now `scripts/deploy-claim.sh` rather than a paragraph anyone re-implements.
|
||||
|
||||
It does **not** share code with the single-server Update button. Worth revisiting, but the overlap turned
|
||||
out to be smaller than expected: the button restarts a process, and almost all of this is the reading,
|
||||
skipping and reporting around that.
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
id: fleet/plugins-update-restarts-every-sidecar-so-the
|
||||
title: '`plugins:update` restarts every sidecar, so the report understates what moved'
|
||||
theme: fleet
|
||||
status: open
|
||||
created: 2026-08-21
|
||||
date_source: stated
|
||||
order: 7
|
||||
---
|
||||
|
||||
**`bun run plugins:update` restarts sidecars itself**, printing `sidecar: started` per plugin. On `alpha`
|
||||
that is twelve processes bouncing during step 3 — before the plan's own restart step, and not something
|
||||
any plan asked for.
|
||||
|
||||
Nothing is broken and this is not new behaviour. The problem is honesty: a deploy report whose "restarted"
|
||||
column reads `officer` is describing one restart when thirteen happened, and the whole value of these
|
||||
reports is that they can be trusted about what moved.
|
||||
|
||||
Two ways out, and they are not the same decision:
|
||||
|
||||
- **Make the report accurate** — the plan format grows a note, or the commander reads `restart_time` for
|
||||
every process rather than only the ones it restarted. Cheap, and it is the honest minimum.
|
||||
- **Make the script not do it** — a `--no-restart` flag, with the deploy restarting what the plan names.
|
||||
That is the better shape (a plan that says "restart `officer` only" should mean it) but it is a change
|
||||
to something that works, and a plugin whose sidecar is not restarted after an update may well be broken
|
||||
in a way nobody notices until later.
|
||||
|
||||
Do the first regardless. The second is a real design call.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
id: fleet/the-deploy-commander-cannot-finish-its-own-host
|
||||
title: 'The deploy commander cannot finish its own host'
|
||||
theme: fleet
|
||||
status: open
|
||||
created: 2026-08-21
|
||||
date_source: stated
|
||||
order: 5
|
||||
---
|
||||
|
||||
**The deploy commander cannot finish its own host, so one machine is always a build behind until somebody
|
||||
restarts it by hand.** After `2026.08.21.03` that skip was two releases deep on `officer-dev` —
|
||||
`restart_time` unchanged at 155 — with the code for both sitting on disk.
|
||||
|
||||
The skip itself is correct and should not be removed. The commander is a session inside
|
||||
`officer-claude-code` on that machine; restarting `officer` there ends the rollout mid-flight, and it is
|
||||
not hypothetical — a manual deploy on 2026-08-21 had to leave that one sidecar alone to survive writing
|
||||
its own report. Restarting only `officer` and leaving the sidecar would technically work and is worse: it
|
||||
puts one host in a state nobody chose, a step behind the rest, discovered later.
|
||||
|
||||
So the question is what finishes it, and there are three honest answers:
|
||||
|
||||
- **A deferred restart the platform runs after the session ends.** Cheapest. The commander writes a
|
||||
one-shot request, something outside the agent's process honours it, and the report says a restart is
|
||||
pending rather than outstanding. Needs a thing that is not the agent and not `officer` — `at`, a systemd
|
||||
transient unit, or PM2 itself.
|
||||
- **A second machine deploys this one.** Cleanest and needs no new mechanism at all: any other host's
|
||||
commander has no reason to skip `officer-dev`. Costs a second armed watcher and raises "who deploys the
|
||||
deployer" — two watchers must not both claim a release, which `scripts/deploy-claim.sh` already prevents
|
||||
per host but **not across hosts**, since the claim is local by design.
|
||||
- **Leave it manual and make it loud.** The report already says "outstanding". A notification would close
|
||||
most of the gap for none of the work.
|
||||
|
||||
Not a decision to make from the code — it is the owner's call which of those three shapes he wants.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
id: fleet/the-watcher-loop-is-rewritten-by-an-agent-every
|
||||
title: 'The watcher loop is rewritten by an agent every time it arms'
|
||||
theme: fleet
|
||||
status: open
|
||||
created: 2026-08-21
|
||||
date_source: stated
|
||||
order: 6
|
||||
---
|
||||
|
||||
**The watcher loop is rewritten from scratch by an agent every time one arms**, so its correctness is
|
||||
re-rolled on every arming. The first one on 2026-08-21 was careful, logged honestly, set a `timeout` on
|
||||
its network call, routed stderr away from the JSON it was parsing — and still contained an unbounded
|
||||
retry, because the obvious way to ask "have I done this release" is to test the `.done` file, and that
|
||||
file only exists on **success**.
|
||||
|
||||
`scripts/deploy-claim.sh` fixed that one instance by moving the rule into code. The general problem is
|
||||
untouched: the next loop can get something else subtly wrong, and nothing will notice until it costs
|
||||
money or misses a release.
|
||||
|
||||
Things a platform-owned loop would make constant rather than re-derived, all of them learned by an agent
|
||||
getting them wrong once:
|
||||
|
||||
- Claim the version before spawning anything.
|
||||
- `OFFICER_ROOT` set explicitly — it is **empty** in the sidecar's environment, so the lock path resolves
|
||||
to `/data/deploys` and every release looks permanently undeployed.
|
||||
- stderr kept out of the `stream-json` stream, or the `claude.ai connectors are disabled` warning corrupts
|
||||
the first `jq` parse.
|
||||
- `jq --unbuffered`, or the job panel shows nothing for twenty minutes and reads as hung.
|
||||
|
||||
The tension worth naming: an agent writing its own loop is also the reason this worked at all on the first
|
||||
night, from a prose description, with no watcher feature built. A fixed script is more correct and less
|
||||
general. The likely answer is a **script the loop calls** rather than a script that replaces it — which is
|
||||
what the claim rule already is, and it could absorb the four items above the same way.
|
||||
Reference in New Issue
Block a user