command-hq: self-reported arm/fire/disarm for loops that outlive their turn

The existing arm path only fires off the harness's own task:started/
task:notification tracking for the live session that submitted the dialog,
so a loop started detached (setsid/nohup) or as a tracked job that dies
with its turn — which is every real commander on this machine — never
arms in Postgres and never records a fire, even after running correctly
for weeks. Adds POST /commanders/dir/:dirName/{arm,fire,disarm}, a second
door to the same armCommandHqWatcher/recordCommandHqWatcherFired/
disarmCommandHqWatcher functions, authenticated with the commander's own
long-lived OFFICER_API_KEY and keyed on dirName since that's what a loop
already knows about itself.
This commit is contained in:
2026-09-14 19:48:11 +01:00
parent 78896793aa
commit 36db9a2ae0
2 changed files with 127 additions and 6 deletions
+39 -3
View File
@@ -62,11 +62,14 @@ for exact request/response shapes; the table below is the map, not the full cont
| GET | `/commanders/:sessionId/turns` | — | `{turns: [...]}` — the raw per-turn cost ledger, oldest first. What "cost over time" plots from | | GET | `/commanders/:sessionId/turns` | — | `{turns: [...]}` — the raw per-turn cost ledger, oldest first. What "cost over time" plots from |
| GET | `/commanders/:sessionId/fires` | — | `{fires: [...]}` — every trigger, oldest first. The full history behind `watcherLastFiredAt` | | GET | `/commanders/:sessionId/fires` | — | `{fires: [...]}` — every trigger, oldest first. The full history behind `watcherLastFiredAt` |
| POST | `/commanders` | `{sessionId, categoryId?}` | files an EXISTING session; idempotent; 404 if the session isn't in the commanders directory. Not the creation path — see `/commanders/new` | | POST | `/commanders` | `{sessionId, categoryId?}` | files an EXISTING session; idempotent; 404 if the session isn't in the commanders directory. Not the creation path — see `/commanders/new` |
| PATCH | `/commanders/:sessionId` | `{categoryId?, name?, watcherArmed?}` | `name` proxies to `renameClaudeSession`; `watcherArmed: false` disarms (arming has no route — needs a live task id) | | PATCH | `/commanders/:sessionId` | `{categoryId?, name?, watcherArmed?}` | `name` proxies to `renameClaudeSession`; `watcherArmed: false` disarms. This route still can't arm — see the `dir/:dirName` routes below for the one path that can from outside a running turn |
| DELETE | `/commanders/:sessionId` | — | deletes the TRANSCRIPT (`/clear` chain and all) via `deleteClaudeSession`, then the row. Not reversible, and not the same thing `/purge` does — see "The `/chat` ↔ `/command-hq` relationship" below | | DELETE | `/commanders/:sessionId` | — | deletes the TRANSCRIPT (`/clear` chain and all) via `deleteClaudeSession`, then the row. Not reversible, and not the same thing `/purge` does — see "The `/chat` ↔ `/command-hq` relationship" below |
| POST | `/purge` | — | sweeps rows whose transcript no longer resolves; returns `{removed}` | | POST | `/purge` | — | sweeps rows whose transcript no longer resolves; returns `{removed}` |
| POST | `/commanders/:sessionId/tags` | `{tagId}` | | | POST | `/commanders/:sessionId/tags` | `{tagId}` | |
| DELETE | `/commanders/:sessionId/tags/:tagId` | — | | | DELETE | `/commanders/:sessionId/tags/:tagId` | — | |
| POST | `/commanders/dir/:dirName/arm` | `{taskId}` | self-reported arm, for a loop the harness never sees fire — see "Self-reported arm" below. `:dirName` is the commander's own numeric id as a string, not the transcript uuid |
| POST | `/commanders/dir/:dirName/fire` | `{taskId}` | self-reported trigger; no-ops (`{ok:false}`) unless `taskId` matches the row's current `watcherTaskId` |
| POST | `/commanders/dir/:dirName/disarm` | — | self-reported clean exit; same effect as `PATCH {watcherArmed:false}`, resolved by dirName instead |
**A commander's conversation is `GET /chat/sessions/:id`, unchanged.** This router deliberately does not **A commander's conversation is `GET /chat/sessions/:id`, unchanged.** This router deliberately does not
duplicate it — `GET /commanders/:sessionId` returns Command HQ's own metadata: `{sessionId, categoryId, duplicate it — `GET /commanders/:sessionId` returns Command HQ's own metadata: `{sessionId, categoryId,
@@ -463,8 +466,9 @@ occur.
`recordCommandHqTurn` in `queries.ts`; exposed read-only on `GET /commanders` and `recordCommandHqTurn` in `queries.ts`; exposed read-only on `GET /commanders` and
`GET /commanders/:sessionId` (`watcherLastFiredAt` survives as a name in the JSON response — it is just `GET /commanders/:sessionId` (`watcherLastFiredAt` survives as a name in the JSON response — it is just
computed there, `MAX(fired_at)`, not read off a column), and `PATCH /commanders/:sessionId` accepts computed there, `MAX(fired_at)`, not read off a column), and `PATCH /commanders/:sessionId` accepts
`{watcherArmed: false}` to disarm (arming itself has no route — it needs a live task id, which only a `{watcherArmed: false}` to disarm. Arming still has no route THERE — it needs a live task id, which only
running turn has). a running turn has — but see "Self-reported arm" below for the route that exists precisely because a
loop that outlives its turn doesn't have one.
- **`armWatcher` threaded end-to-end, client to server.** `BackgroundJobDialog` computes it (`watching && - **`armWatcher` threaded end-to-end, client to server.** `BackgroundJobDialog` computes it (`watching &&
onFire === 'handoff'`) → `sendWithHiddenPrefix` → `QueuedPrompt.armWatcher` → `sendPrompt`'s new last onFire === 'handoff'`) → `sendWithHiddenPrefix` → `QueuedPrompt.armWatcher` → `sendPrompt`'s new last
parameter → the chat websocket message → `ClientMessage`'s `'chat'` variant → `startClaudeTurn`'s parameter → the chat websocket message → `ClientMessage`'s `'chat'` variant → `startClaudeTurn`'s
@@ -537,6 +541,38 @@ OWN user id, not the row it belonged to). Found by re-reading every call site th
after the fact, not by any tool — worth remembering the next time a signature changes shape rather than after the fact, not by any tool — worth remembering the next time a signature changes shape rather than
just its types: a compiler that stays quiet is not the same as a diff that is correct. just its types: a compiler that stays quiet is not the same as a diff that is correct.
### Self-reported arm — 2026-09-14
The mechanism above has a gap it never surfaced until four real commanders on the owner's own machine were
checked against it: `pendingArmWatcher` is consumed on the harness's `task:started` for the live session
that submitted the arm dialog, and stays valid only as long as that session's own tracked-background-task
plumbing does. A loop that needs to outlive its turn — `setsid`/`nohup`/`disown`, or any tracked background
job that the harness itself drops when the turn ends, which multiple real commanders here do on purpose,
one of them (`docs`-adjacent `WATCHERS.md` for the OCR watcher) as an explicit standing decision favouring
job-panel visibility over survival — never produces that event. `watcherArmed` stayed `false` and
`command_hq_watcher_fires` stayed empty forever for every one of them, even while the loop genuinely polled
and genuinely fired, repeatedly, for weeks. Not cosmetic: `GET /commanders`'s card already papers over the
`watcherArmed` boolean with a live pid check (`row.watcherArmed || liveLoopPid(...) !== null`, `router.ts`),
so the badge was always right — but nothing papers over an empty `command_hq_watcher_fires`, so
`watcherLastFiredAt` read `null` forever on every card for a commander that had fired dozens of times.
Fixed by giving the loop a second, independent way to say the same thing the harness's own tracking says
for a session-scoped commander: three routes under `/commanders/dir/:dirName/{arm,fire,disarm}`,
authenticated with the commander's own long-lived `OFFICER_API_KEY` (the same credential its loop already
holds and already uses for `/api/notify` and `/api/tasks/*/run` — nothing new minted), keyed on `dirName`
because that is what a loop already knows about itself without needing its own transcript uuid. `arm` and
`disarm` call `armCommandHqWatcher`/`disarmCommandHqWatcher` directly — the same functions `websocket.ts`
calls, just reached from a second door. `fire` keeps the identical staleness guard `task:notification`
already has (`watcherTaskId` must match), so a superseded instance of a loop — an old pid a crash left
running, now replaced by a freshly-armed one — can't keep writing fires for a watch it no longer is. The
task id itself is no longer harness-issued for this path: the loop mints its own (pid + start time is
enough) at the top of the script and reuses it for every `fire` call until it restarts.
This does not change anything about the original harness-driven path — a commander whose loop genuinely is
a tracked background job on a live session still arms exactly as before, and most of what's written above
about `pendingArmWatcher` and the noticing in `createMessageHandler` is unchanged. It adds a second, parallel
way to reach the same two DB functions, for the class of commander the first path structurally can't reach.
## Later — explicitly deferred, not forgotten ## Later — explicitly deferred, not forgotten
Raised during spec and set aside for a second version, in the owner's words: _"that's not... that's out of Raised during spec and set aside for a second version, in the owner's words: _"that's not... that's out of
+88 -3
View File
@@ -41,9 +41,12 @@ import {
setCommanderSessionId, setCommanderSessionId,
listCommandHqSessions, listCommandHqSessions,
getCommandHqSession, getCommandHqSession,
getCommandHqSessionById,
setCommandHqSessionCategory, setCommandHqSessionCategory,
purgeCommandHqSessions, purgeCommandHqSessions,
deleteCommandHqSession, deleteCommandHqSession,
armCommandHqWatcher,
recordCommandHqWatcherFired,
disarmCommandHqWatcher, disarmCommandHqWatcher,
getCommandHqCostSummary, getCommandHqCostSummary,
listCommandHqTurns, listCommandHqTurns,
@@ -721,9 +724,9 @@ commandHqRouter.post('/commanders', async (ctx) => {
* `name: string` — rename. Proxies straight to `renameClaudeSession`, the SAME function `/chat`'s own * `name: string` — rename. Proxies straight to `renameClaudeSession`, the SAME function `/chat`'s own
* rename action calls — there is no separate name field on this table to disagree with it. * rename action calls — there is no separate name field on this table to disagree with it.
* `watcherArmed: false` — disarm a watcher, so the harness stops re-arming it on its next fire. Only * `watcherArmed: false` — disarm a watcher, so the harness stops re-arming it on its next fire. Only
* `false` is accepted here: arming requires a live task id (`armCommandHqWatcher`), which only the turn * `false` is accepted here: arming through THIS route would need a live task id, which only the turn
* that starts the watcher has — there is no product path to arm one from outside a running turn, and * that starts the watcher has. A loop reporting its own task id arms through `POST .../dir/:dirName/arm`
* this route is not it. * below instead — that route exists precisely because this one can't take it.
*/ */
commandHqRouter.patch('/commanders/:sessionId', async (ctx) => { commandHqRouter.patch('/commanders/:sessionId', async (ctx) => {
const user = ctx.get('user'); const user = ctx.get('user');
@@ -751,6 +754,88 @@ commandHqRouter.patch('/commanders/:sessionId', async (ctx) => {
return ctx.json({ ok: true }); return ctx.json({ ok: true });
}); });
/**
* The self-report counterpart to `armCommandHqWatcher`/`recordCommandHqWatcherFired`'s harness-driven
* calls in `websocket.ts`. Those fire off a `task:started`/`task:notification` event from the harness's
* OWN background-task tracking on the live session that submitted the arm dialog — which never happens
* for a loop that outlives its turn (`setsid`/`nohup`, or a tracked background job that dies with the
* turn anyway, which is most of them). `watcher_armed` stayed false and `command_hq_watcher_fires` stayed
* empty forever for exactly that class of commander — see `docs/command-hq.md` → "Self-reported arm".
*
* Authenticated with the commander's own `OFFICER_API_KEY` (minted per-commander in `credential.ts`), the
* same door its loop already uses for `/api/notify` and `/api/tasks/*​/run`. Keyed on `:dirName`, not
* `:sessionId` — a loop knows its own directory name (it's the folder it's running from; `dirName` is
* literally `String(id)`, see `createCommander`) and has no reason to also know its transcript's uuid.
*/
commandHqRouter.post('/commanders/dir/:dirName/arm', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row?.sessionId) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (!taskId) throw errors.BAD_REQUEST('taskId required');
await armCommandHqWatcher(user.id, row.sessionId, taskId);
return ctx.json({ ok: true });
});
/**
* `fire`'s own guard mirrors the harness-driven path exactly: only records a trigger when this row is
* CURRENTLY armed with a matching task id, so an old instance of a loop that a fresh restart superseded
* (new pid, new self-generated task id) can't keep writing fires for a watch it no longer is — the same
* protection `websocket.ts`'s `task:notification` handler gets from checking `watcherTaskId === taskId`.
*/
commandHqRouter.post('/commanders/dir/:dirName/fire', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (!taskId) throw errors.BAD_REQUEST('taskId required');
if (!row.watcherArmed || row.watcherTaskId !== taskId) {
return ctx.json({ ok: false, reason: 'not the current armed task' });
}
await recordCommandHqWatcherFired(row.id, taskId);
return ctx.json({ ok: true });
});
/**
* The loop's own clean-exit counterpart to `arm` — best-effort, same as every other disarm path: a loop
* killed without running its trap still leaves `watcher_armed: true` until the next arm or an explicit
* disarm, exactly as it already could before this route existed.
*
* Takes the SAME `taskId` guard as `fire`, and for the same reason: a restart replaces the old instance
* with a fresh arm before the old one's process necessarily notices it's been superseded (an inotify
* loop's kill can outrace the new one's `arm` call by seconds). An unguarded disarm from the OLD
* instance's exit trap would then clear the NEW instance's `watcher_armed` a moment after it was set —
* caught in exactly this shape while restarting a real duplicate on this machine, 2026-09-14. `taskId`
* is optional so an out-of-band disarm (nothing self-reported ever armed this row) still works — the
* guard only applies once there is a task id on the row to disagree with.
*/
commandHqRouter.post('/commanders/dir/:dirName/disarm', async (ctx) => {
const user = ctx.get('user');
const id = Number(ctx.req.param('dirName'));
if (!Number.isInteger(id)) throw errors.NOT_FOUND('not a commander');
const row = await getCommandHqSessionById(user.id, id);
if (!row?.sessionId) throw errors.NOT_FOUND('not a commander');
const body = await ctx.req.json<{ taskId?: string }>().catch(() => ({}) as { taskId?: string });
const taskId = body.taskId?.trim();
if (taskId && row.watcherTaskId && row.watcherTaskId !== taskId) {
return ctx.json({ ok: false, reason: 'not the current armed task' });
}
await disarmCommandHqWatcher(user.id, row.sessionId);
return ctx.json({ ok: true });
});
/** /**
* DELETE /command-hq/commanders/:sessionId * DELETE /command-hq/commanders/:sessionId
* *