diff --git a/todo/agent-coordination/a-commander-s-watcher-does-not-survive-a-reboot.md b/todo/agent-coordination/a-commander-s-watcher-does-not-survive-a-reboot.md new file mode 100644 index 00000000..c3153528 --- /dev/null +++ b/todo/agent-coordination/a-commander-s-watcher-does-not-survive-a-reboot.md @@ -0,0 +1,62 @@ +--- +id: agent-coordination/a-commander-s-watcher-does-not-survive-a-reboot +title: "A commander's watcher does not survive a reboot, and nothing says so" +theme: agent-coordination +status: open +priority: urgent +created: 2026-08-25 +date_source: stated +order: 3 +--- + +**A commander's watcher loop is a bare background process, so a reboot ends it and nothing brings it +back or mentions it.** Found 2026-08-25 while trying to deploy `2026.08.25.01`: the tag was pushed and +nothing picked it up. + +## What was verified + +- `commanders/40/state/loop-pid` held **291464, dead**. Its last state file was written 2026-08-24 + 03:50 — the `2026.08.24.02` rollout. +- `uptime -s` says the machine came up **2026-08-24 12:40:11**. +- `commanders/54` (Voice Memos) was armed after that boot and its pid is alive, which is why one card + reads "watching" and the other does not. +- Nothing else was watching either: no cron entry, and `data/deploy-watch.sh` was not running. + +So a release sat on the remote with nobody looking at it, and the only signal was the absence of a word +on a card. `watcher_armed` in `command_hq_sessions` was already `false` for both rows, so even the +database did not think it was armed — the UI is honest, it just says nothing. + +## Why it is worse than an ordinary crash + +The commander doctrine is that you arm it and walk away for months. A watcher that dies on reboot +breaks exactly the promise that makes a commander worth having, and it breaks it **quietly** — no +notification, no failed run, no error anywhere. The owner discovers it by noticing that something which +should have happened did not, which is the failure mode `docs/agent-coordination.md` exists to avoid. + +Note it is not only reboots. Anything that kills the process tree the loop was spawned under does the +same thing, and the loop's parentage is an accident of which agent turn armed it. + +## What to build, in order of how much it buys + +1. **Say it out loud.** The cheapest fix and most of the value: a commander whose row says it was armed + and whose pid is dead is a **detectable** state — `liveLoopPid()` already answers it, and + `router.ts` already computes `watcherArmed` and `watcherRunning` separately for exactly this reason. + Notify the owner once when a watcher that was armed is found not running. Nothing else on this list + matters if the owner is told. + +2. **Bring it back on boot.** A loop that records its pid can record its command line beside it, and + something at platform start can re-arm every commander whose row says armed and whose pid is gone. + The awkward part is that the loop is authored by the commander itself, per commander — so what gets + restarted has to be recorded when it is armed, not reconstructed afterwards. + +3. **Stop it being a bare process at all.** `docs/commander-directory.md` already argues a commander is + a directory rather than a session. A watcher belongs to that directory and could be supervised the + way sidecars are, rather than inheriting whichever agent turn happened to spawn it. + +## Interim, until any of it exists + +**After a reboot, re-arm every commander by hand.** `watcherRunning` on the Command HQ grid is the +truthful field — a card that does not say "watching" is not watching, whatever it did yesterday. + +Related: [[agent-coordination]] docs, and `data/deploy-watch.sh`, which is the deploy commander's own +loop script and equally unsupervised.