Compare commits

..
Author SHA1 Message Date
pastilhas 4a37cfd2ce release 2026.09.14.01 2026-09-14 20:11:15 +01:00
pastilhas 36db9a2ae0 command-hq: self-reported arm/fire/disarm for loops that outlive their turn
The existing arm path only fires off the harness's own task:started/
task:notification tracking for the live session that submitted the dialog,
so a loop started detached (setsid/nohup) or as a tracked job that dies
with its turn — which is every real commander on this machine — never
arms in Postgres and never records a fire, even after running correctly
for weeks. Adds POST /commanders/dir/:dirName/{arm,fire,disarm}, a second
door to the same armCommandHqWatcher/recordCommandHqWatcherFired/
disarmCommandHqWatcher functions, authenticated with the commander's own
long-lived OFFICER_API_KEY and keyed on dirName since that's what a loop
already knows about itself.
2026-09-14 19:48:11 +01:00
pastilhas 78896793aa correct the 2026.09.09.01 report: no cron moved those hosts
The agent inferred auto-pull.sh had pulled alpha and fofos ahead of the rollout. It had not. alpha was
rolled by hand by the session that cut the release, before being stopped; fofos was current because it
had been deployed to all day for plugin work.

The wrong reading is kept in the text rather than deleted — a real cron, on the right machine, with the
right effect, and nothing to do with this. That is what a plausible wrong answer looks like.
2026-09-09 23:32:37 +01:00
pastilhasandClaude Opus 5 9217d1faaa deploy report 2026.09.09.01 — 4 of 4 targets
plugin_ports verified by query on every host. alpha and fofos were
already at the tag; macbookpro and deadpool were real fast-forwards
and the only two that needed @anthropic-ai/sdk.

Two plugins did not update, both blocked by uncommitted local work:
av-club on alpha, fofos-website on fofos. Nothing was discarded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 23:28:31 +01:00
5 changed files with 269 additions and 7 deletions
+71
View File
@@ -0,0 +1,71 @@
# 2026.09.09.01 — report
**4 of 4 target hosts succeeded.** `officer-dev` skipped entirely, as the host the rollout ran on.
`macbruno` and both Pertento hosts skipped per the plan and `docs/servers.md`.
Every target ended at `be3139f9` — the commit `deploy/2026.09.09.01` points at. `plugin_ports` was
verified **by query on each host's own database**, not inferred from `db:push` exit codes.
**Two hosts were already at the tag before the rollout touched them, and the cause was a person, not a
process.** `alpha` was at `be3139f9` rather than the `19bbbab9` the plan recorded, and `fofos` was at
`be3139f9`. The deploy agent inferred `auto-pull.sh` on its minute crontab; **that inference is wrong
and is corrected here** so nobody spends time hunting a mechanism that was not involved.
What actually happened: the session that cut this release also rolled `alpha` by hand before being
stopped — the release is the releaser's job and the rollout is this commander's, and that line was
crossed. `fofos` was at `master` for an unrelated reason: it had been deployed to repeatedly through
the day during work on the `fofos-website` plugin.
So their merges were no-ops. Only `macbookpro` and `deadpool-rezio-bridge` were genuine fast-forwards,
and they are also the only two where `bun install` had work to do.
The `auto-pull` reading is worth keeping visible rather than deleting, because it is exactly the shape
of a plausible wrong answer: a real cron, on the right machine, with the right effect, that had nothing
to do with this. A host being unexpectedly current is evidence of *something*, and "the obvious
candidate" is not the same as the cause.
## Per host
| Host | HEAD before → after | `bun install` | `plugins:update` | `plugin_ports` by query | `officer` restart_time | health |
| --- | --- | --- | --- | --- | --- | --- |
| `officer-dev` | — | — | — | — | — | **SKIPPED** — the host this ran on |
| `alpha` | `be3139f9` → `be3139f9` | no changes | **16/17, exit 1** | ✓ present, 5 cols | 10 → 11, online | `401` (9000) |
| `fofos` | `be3139f9` → `be3139f9` | no changes | **4/5, exit 1** | ✓ present, 5 cols | 380 → 381, online | `401` (9000) |
| `macbookpro` | `c97da029` → `be3139f9` | **+ @anthropic-ai/sdk@0.71.2** | no plugins installed | ✓ present, 5 cols | 0 → 1, online | `401` (**9010**) |
| `deadpool-rezio-bridge` | `c97da029` → `be3139f9` | **+ @anthropic-ai/sdk@0.71.2** | 2/2, exit 0 | ✓ present, 5 cols | 0 → 1, online | `401` (9000) |
`401` on `/api/health` is healthy — the endpoint is behind auth. Ports were read from each `.env`.
The schema check run per host was `select to_regclass('public.plugin_ports')`, answered `plugin_ports`
on all four, with columns `id`, `app_name`, `name`, `port`, `allocated_at timestamptz`. Databases:
`officer_dev` on `alpha` and `macbookpro`, `officer` on `fofos` and `deadpool-rezio-bridge`.
No `db:push` prompted for anything — `< /dev/null` was used on every host, as the plan required.
## What the owner should act on
- **`plugins:update` failed on two hosts, both for uncommitted work in a plugin checkout.** Nothing was
discarded. Each is one `git commit` or `git checkout --` away from clean:
- `alpha` — `plugins/av-club` at `62972c1`, 3 modified: `docs/SECURITY.md`, `sidecar/config.ts`,
`sidecar/paths.ts`. That plugin did **not** update on `alpha` and is still on its old checkout.
- `fofos` — `plugins/fofos-website` at `7001eb0`, 1 modified: `frontend/Layout/Footer/Footer.tsx`.
Same consequence: that plugin did not update.
- **`alpha` could not fetch.** `git fetch origin --tags` failed with
`Connection closed by 77.42.39.155 port 22` — gitea's public SSH, rate-limited at the proxy. Its
`origin` is `git@gitea.officer.dev`. It made no difference this time because the tag was already
local, but a host that genuinely needed the tag would have failed there. **The remote on that host is
the thing that is wrong**; pointing it at the tailnet address removes the whole class. The same
recommendation was made in the `2026.08.30.03` report about `macbookpro`, which fetched cleanly today.
- **`macbruno` was not attempted.** The plan recorded it unreachable at cut time (`Connection timed
out`, 2026-09-09) and named it skipped, so no retry budget was spent on it. It is at `c97da029` as far
as anyone knows and will be a real fast-forward whenever it is next reached.
- **`officer-dev` still wants the marketplace column.** `plugins.release_tag` in `marketplace_db` is
out of scope for this rollout by the plan's own words — `bun db:push` covers `officer_db` only, and
`officer-marketplace` runs on this host alone, which is the host the rollout skips. Whoever restarts
`officer-dev` deliberately should apply it then. Nothing improvised a second push to reach it.
## Not done, deliberately
- `officer-dev` — not pulled, not restarted, nothing touched. It is already at `be3139f9` in code.
- `macbruno`, `production-pertento-bridge`, `edge-pertento-bridge` — skipped, no fault on any of them.
- The `bunx tsgo` error at `src/servers/sidecar/claude/claude-manager.ts:667` was neither checked nor
fixed. The plan names it as pre-existing and type-only; nothing in this rollout would have surfaced it.
+70
View File
@@ -0,0 +1,70 @@
# 2026.09.14.01 — command-hq self-reported arm/fire/disarm
**One commit**, `deploy/2026.09.09.01..36db9a2a`. Backend-only, no schema change, no new dependency.
## What it is
Three new routes, `POST /command-hq/commanders/dir/:dirName/{arm,fire,disarm}` — a second, self-reported
door onto the existing `armCommandHqWatcher`/`recordCommandHqWatcherFired`/`disarmCommandHqWatcher`
functions, for a commander loop that outlives its own turn (detached, or a tracked background job that
dies with the turn anyway) and so never produces the harness's own `task:started`/`task:notification`
events. Authenticated with the commander's own long-lived `OFFICER_API_KEY`. Full account in
`docs/command-hq.md` → "Self-reported arm".
No route removed, no existing route's contract changed, no frontend change. The only way this release
does anything is a host restarting `officer` to pick up the new routes — a commander whose loop already
calls them (this machine's four) gets nothing from the rollout on any OTHER host, because those hosts'
own commanders, if any, don't call them yet.
## Verified, not assumed
```bash
git diff --name-only deploy/2026.09.09.01..HEAD | grep -iE 'schema|databases/' # → nothing
git diff --name-only deploy/2026.09.09.01..HEAD -- package.json bun.lock # → nothing
git diff --name-only deploy/2026.09.09.01..HEAD -- docs/servers.md # → nothing, host list unchanged
bun run docs:check # → green
```
No schema change, no dependency change, no host-list change since the last release. `package.json`
version bumped to match this tag and nothing else in it changed.
## Every host — the commands
Standard steps only, nothing extra: dirty check (TRACKED only), `git fetch origin --tags`,
`git merge --ff-only deploy/2026.09.14.01`, `bun run plugins:update`, `pm2 restart officer`. No
`bun install`, no `db:push` — neither is needed and running either is not a finding, just unnecessary.
### Verify it came back — two checks, per host
No schema table to query this time, so the usual three checks drop to two:
1. **`pm2 jlist`** — `officer` is `online` and its `restart_time` moved.
2. **HTTP** — `/api/health` on the host's own port. `200` or `401` are both healthy. Port is **9010 on
`macbookpro`**, 9000 elsewhere.
## Per host
| Host | Login | Note |
| ----------------------- | -------------- | ----------------------------------------------------------|
| `alpha` | `pastilhas` | |
| `fofos` | `pastilhas` | |
| `macbookpro` | `pastilhas` | Laptop. Asleep is a SKIP with a note. Port **9010**. |
| `deadpool-rezio-bridge` | **`br`** | Not `pastilhas`. |
### Skipped, deliberately — not failures
- **`officer-dev`** — the host the rollout runs on. Skip entirely, per the standard rule.
- **`macbruno`** — reachability unknown at cut time; check live, unreachable is a SKIP not a failure.
- **Both Pertento hosts** — deliberately-skipped list in `docs/servers.md` since 2026-09-08.
## What "done" looks like
Four hosts attempted: `alpha`, `fofos`, `macbookpro`, `deadpool-rezio-bridge`. Each at
`deploy/2026.09.14.01`, `officer` back online, health answering on the right port. A host that fails
does not stop the rollout — its own line, what failed, what never ran.
## Known and carried, not introduced here
`bunx tsgo` reports one pre-existing type error in `src/servers/sidecar/claude/claude-manager.ts:667`
(`SdkUserMessage` vs `SDKUserMessage`, from the agent SDK's own types) — present before this release,
type-only, stripped at run time. Named here so it isn't mistaken for something this release did.
+39 -3
View File
@@ -62,11 +62,14 @@ for exact request/response shapes; the table below is the map, not the full cont
| GET | `/commanders/:sessionId/turns` | — | `{turns: [...]}` — the raw per-turn cost ledger, oldest first. What "cost over time" plots from |
| GET | `/commanders/:sessionId/fires` | — | `{fires: [...]}` — every trigger, oldest first. The full history behind `watcherLastFiredAt` |
| POST | `/commanders` | `{sessionId, categoryId?}` | files an EXISTING session; idempotent; 404 if the session isn't in the commanders directory. Not the creation path — see `/commanders/new` |
| PATCH | `/commanders/:sessionId` | `{categoryId?, name?, watcherArmed?}` | `name` proxies to `renameClaudeSession`; `watcherArmed: false` disarms (arming has no route — needs a live task id) |
| PATCH | `/commanders/:sessionId` | `{categoryId?, name?, watcherArmed?}` | `name` proxies to `renameClaudeSession`; `watcherArmed: false` disarms. This route still can't arm — see the `dir/:dirName` routes below for the one path that can from outside a running turn |
| DELETE | `/commanders/:sessionId` | — | deletes the TRANSCRIPT (`/clear` chain and all) via `deleteClaudeSession`, then the row. Not reversible, and not the same thing `/purge` does — see "The `/chat` ↔ `/command-hq` relationship" below |
| POST | `/purge` | — | sweeps rows whose transcript no longer resolves; returns `{removed}` |
| POST | `/commanders/:sessionId/tags` | `{tagId}` | |
| DELETE | `/commanders/:sessionId/tags/:tagId` | — | |
| POST | `/commanders/dir/:dirName/arm` | `{taskId}` | self-reported arm, for a loop the harness never sees fire — see "Self-reported arm" below. `:dirName` is the commander's own numeric id as a string, not the transcript uuid |
| POST | `/commanders/dir/:dirName/fire` | `{taskId}` | self-reported trigger; no-ops (`{ok:false}`) unless `taskId` matches the row's current `watcherTaskId` |
| POST | `/commanders/dir/:dirName/disarm` | — | self-reported clean exit; same effect as `PATCH {watcherArmed:false}`, resolved by dirName instead |
**A commander's conversation is `GET /chat/sessions/:id`, unchanged.** This router deliberately does not
duplicate it — `GET /commanders/:sessionId` returns Command HQ's own metadata: `{sessionId, categoryId,
@@ -463,8 +466,9 @@ occur.
`recordCommandHqTurn` in `queries.ts`; exposed read-only on `GET /commanders` and
`GET /commanders/:sessionId` (`watcherLastFiredAt` survives as a name in the JSON response — it is just
computed there, `MAX(fired_at)`, not read off a column), and `PATCH /commanders/:sessionId` accepts
`{watcherArmed: false}` to disarm (arming itself has no route — it needs a live task id, which only a
running turn has).
`{watcherArmed: false}` to disarm. Arming still has no route THERE — it needs a live task id, which only
a running turn has — but see "Self-reported arm" below for the route that exists precisely because a
loop that outlives its turn doesn't have one.
- **`armWatcher` threaded end-to-end, client to server.** `BackgroundJobDialog` computes it (`watching &&
onFire === 'handoff'`) → `sendWithHiddenPrefix` → `QueuedPrompt.armWatcher` → `sendPrompt`'s new last
parameter → the chat websocket message → `ClientMessage`'s `'chat'` variant → `startClaudeTurn`'s
@@ -537,6 +541,38 @@ OWN user id, not the row it belonged to). Found by re-reading every call site th
after the fact, not by any tool — worth remembering the next time a signature changes shape rather than
just its types: a compiler that stays quiet is not the same as a diff that is correct.
### Self-reported arm — 2026-09-14
The mechanism above has a gap it never surfaced until four real commanders on the owner's own machine were
checked against it: `pendingArmWatcher` is consumed on the harness's `task:started` for the live session
that submitted the arm dialog, and stays valid only as long as that session's own tracked-background-task
plumbing does. A loop that needs to outlive its turn — `setsid`/`nohup`/`disown`, or any tracked background
job that the harness itself drops when the turn ends, which multiple real commanders here do on purpose,
one of them (`docs`-adjacent `WATCHERS.md` for the OCR watcher) as an explicit standing decision favouring
job-panel visibility over survival — never produces that event. `watcherArmed` stayed `false` and
`command_hq_watcher_fires` stayed empty forever for every one of them, even while the loop genuinely polled
and genuinely fired, repeatedly, for weeks. Not cosmetic: `GET /commanders`'s card already papers over the
`watcherArmed` boolean with a live pid check (`row.watcherArmed || liveLoopPid(...) !== null`, `router.ts`),
so the badge was always right — but nothing papers over an empty `command_hq_watcher_fires`, so
`watcherLastFiredAt` read `null` forever on every card for a commander that had fired dozens of times.
Fixed by giving the loop a second, independent way to say the same thing the harness's own tracking says
for a session-scoped commander: three routes under `/commanders/dir/:dirName/{arm,fire,disarm}`,
authenticated with the commander's own long-lived `OFFICER_API_KEY` (the same credential its loop already
holds and already uses for `/api/notify` and `/api/tasks/*/run` — nothing new minted), keyed on `dirName`
because that is what a loop already knows about itself without needing its own transcript uuid. `arm` and
`disarm` call `armCommandHqWatcher`/`disarmCommandHqWatcher` directly — the same functions `websocket.ts`
calls, just reached from a second door. `fire` keeps the identical staleness guard `task:notification`
already has (`watcherTaskId` must match), so a superseded instance of a loop — an old pid a crash left
running, now replaced by a freshly-armed one — can't keep writing fires for a watch it no longer is. The
task id itself is no longer harness-issued for this path: the loop mints its own (pid + start time is
enough) at the top of the script and reuses it for every `fire` call until it restarts.
This does not change anything about the original harness-driven path — a commander whose loop genuinely is
a tracked background job on a live session still arms exactly as before, and most of what's written above
about `pendingArmWatcher` and the noticing in `createMessageHandler` is unchanged. It adds a second, parallel
way to reach the same two DB functions, for the class of commander the first path structurally can't reach.
## Later — explicitly deferred, not forgotten
Raised during spec and set aside for a second version, in the owner's words: _"that's not... that's out of
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "officer",
"version": "2026.09.09.01",
"version": "2026.09.14.01",
"private": true,
"type": "module",
"workspaces": [
+88 -3
View File
@@ -41,9 +41,12 @@ import {
setCommanderSessionId,
listCommandHqSessions,
getCommandHqSession,
getCommandHqSessionById,
setCommandHqSessionCategory,
purgeCommandHqSessions,
deleteCommandHqSession,
armCommandHqWatcher,
recordCommandHqWatcherFired,
disarmCommandHqWatcher,
getCommandHqCostSummary,
listCommandHqTurns,
@@ -721,9 +724,9 @@ commandHqRouter.post('/commanders', async (ctx) => {
* `name: string` — rename. Proxies straight to `renameClaudeSession`, the SAME function `/chat`'s own
* rename action calls — there is no separate name field on this table to disagree with it.
* `watcherArmed: false` — disarm a watcher, so the harness stops re-arming it on its next fire. Only
* `false` is accepted here: arming requires a live task id (`armCommandHqWatcher`), which only the turn
* that starts the watcher has — there is no product path to arm one from outside a running turn, and
* this route is not it.
* `false` is accepted here: arming through THIS route would need a live task id, which only the turn
* that starts the watcher has. A loop reporting its own task id arms through `POST .../dir/:dirName/arm`
* below instead — that route exists precisely because this one can't take it.
*/
commandHqRouter.patch('/commanders/:sessionId', async (ctx) => {
const user = ctx.get('user');
@@ -751,6 +754,88 @@ commandHqRouter.patch('/commanders/:sessionId', async (ctx) => {
return ctx.json({ ok: true });
});
/**
* The self-report counterpart to `armCommandHqWatcher`/`recordCommandHqWatcherFired`'s harness-driven
* calls in `websocket.ts`. Those fire off a `task:started`/`task:notification` event from the harness's
* OWN background-task tracking on the live session that submitted the arm dialog — which never happens
* for a loop that outlives its turn (`setsid`/`nohup`, or a tracked background job that dies with the
* turn anyway, which is most of them). `watcher_armed` stayed false and `command_hq_watcher_fires` stayed
* empty forever for exactly that class of commander — see `docs/command-hq.md` → "Self-reported arm".
*
* Authenticated with the commander's own `OFFICER_API_KEY` (minted per-commander in `credential.ts`), the
* same door its loop already uses for `/api/notify` and `/api/tasks/*​/run`. Keyed on `:dirName`, not
* `:sessionId` — a loop knows its own directory name (it's the folder it's running from; `dirName` is
* literally `String(id)`, see `createCommander`) and has no reason to also know its transcript's uuid.
*/
commandHqRouter.post('/commanders/dir/:dirName/arm', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row?.sessionId) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (!taskId) throw errors.BAD_REQUEST('taskId required');
await armCommandHqWatcher(user.id, row.sessionId, taskId);
return ctx.json({ ok: true });
});
/**
* `fire`'s own guard mirrors the harness-driven path exactly: only records a trigger when this row is
* CURRENTLY armed with a matching task id, so an old instance of a loop that a fresh restart superseded
* (new pid, new self-generated task id) can't keep writing fires for a watch it no longer is — the same
* protection `websocket.ts`'s `task:notification` handler gets from checking `watcherTaskId === taskId`.
*/
commandHqRouter.post('/commanders/dir/:dirName/fire', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (!taskId) throw errors.BAD_REQUEST('taskId required');
if (!row.watcherArmed || row.watcherTaskId !== taskId) {
return ctx.json({ ok: false, reason: 'not the current armed task' });
}
await recordCommandHqWatcherFired(row.id, taskId);
return ctx.json({ ok: true });
});
/**
* The loop's own clean-exit counterpart to `arm` — best-effort, same as every other disarm path: a loop
* killed without running its trap still leaves `watcher_armed: true` until the next arm or an explicit
* disarm, exactly as it already could before this route existed.
*
* Takes the SAME `taskId` guard as `fire`, and for the same reason: a restart replaces the old instance
* with a fresh arm before the old one's process necessarily notices it's been superseded (an inotify
* loop's kill can outrace the new one's `arm` call by seconds). An unguarded disarm from the OLD
* instance's exit trap would then clear the NEW instance's `watcher_armed` a moment after it was set —
* caught in exactly this shape while restarting a real duplicate on this machine, 2026-09-14. `taskId`
* is optional so an out-of-band disarm (nothing self-reported ever armed this row) still works — the
* guard only applies once there is a task id on the row to disagree with.
*/
commandHqRouter.post('/commanders/dir/:dirName/disarm', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row?.sessionId) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (taskId && row.watcherTaskId && row.watcherTaskId !== taskId) {
return ctx.json({ ok: false, reason: 'not the current armed task' });
}
await disarmCommandHqWatcher(user.id, row.sessionId);
return ctx.json({ ok: true });
});
/**
* DELETE /command-hq/commanders/:sessionId
*