COMMS: what broke while actually running a background job

Written after arming, firing, draining and re-arming a real watcher rather than
reading the code. Command HQ is a home for these, so anything wrong with the
mechanism shows up there first and worst — a commander is meant to run for months.

Six faults, the first structural: "keep going" and "tell the agent" are mutually
exclusive, because a background process reaches the model only when it EXITS. A
looping job can do shell work and nothing else — it cannot read a diff or judge
whether a push matters. So the dialog offers a mode that silently means "the agent
will never see this again", which is the wrong default for the exact case watchers
exist for. Asks for a third option: exit, hand back, re-arm.

Then: drain before re-arming or lose what lands in the gap; a watcher on a branch
the agent also pushes to fires on its own writes, and author filtering cannot help
because both sides are Claude; `timeout` does not exist on macOS so rule 6 cannot be
followed as written; a job stopped on purpose is reported as failed, which would read
as broken on a grid of cards; and the offered working directory is the conversation's
rather than the watched thing's.

Also what worked and should not be traded away — ask-before-arming earned its place
in the first minute — with one caveat: the output rules leave nowhere to report that
the thing asked for cannot work as specified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

COMMS/command-hq/watchers-what-broke.md

@@ -0,0 +1,110 @@
+# Background jobs: what broke while actually running one
+
+Written 2026-08-18 by the mobile side, after arming, firing, draining and re-arming a real watcher
+(poll `git ls-remote` on this repo, react to each push) through the Background Job dialog.
+
+Command HQ is a home for these, so anything wrong with the mechanism shows up there first and worst: a
+commander is a watcher that is supposed to run for months.
+
+Everything below is from running one, not from reading the code.
+
+---
+
+## 1. "Keep going" and "tell me" are mutually exclusive, and the dialog offers them as if they were not
+
+The dialog's two options are **Stop when it fires** and **Keep going — act each time and carry on**.
+The preamble expands the second into: *"Do NOT exit when it fires... the stream is the notification,
+not the exit."*
+
+That is true for a human watching a log. It is **false for the agent**, and the agent is who was asked
+to react. A background process reaches the model **only when it exits** — that is the harness's
+notification. A job that loops forever can do shell work and nothing else. It cannot read a diff,
+cannot judge whether the push matters, cannot write code.
+
+So "keep going" silently means "the agent will never see this again". For a watcher whose whole point
+is *"when it fires, go and do something"*, that is the wrong default and it fails invisibly: the job is
+alive, the output is accumulating, and nobody is reading it.
+
+**What the owner actually wanted**, worked out in conversation and now the shape we use:
+
+> exit on change → agent acts → **drain** (check again immediately) → act again if more arrived →
+> re-arm the watcher when nothing is left
+
+**Asked for:** a third option in the dialog, something like *"Exit and hand back to the agent, then
+re-arm"*, and it should be the default for anything the agent must react to. The existing two are then
+honest: stop-on-fire is one-shot, keep-going is for a job whose *output* is the product.
+
+## 2. Drain before re-arming, or pushes are lost
+
+Not a nicety. Between the job exiting and the agent re-arming, more can land. Re-arming from a baseline
+captured before the work began re-fires immediately; re-arming from "now" silently swallows whatever
+arrived in the gap.
+
+The rule that works: after acting, ask again **before** re-arming. If something new is there, act again.
+Only when the source is quiet does the watcher go back on. The owner specified this himself; it should
+be in the preamble rather than rediscovered per job.
+
+## 3. A watcher on a shared branch fires on the agent's own writes
+
+Observed within an hour. The watcher polled `master` on this repo; the agent then pushed a COMMS note to
+`master`; the watcher fired; there were no new commits to read.
+
+**Author filtering does not help here** — the platform side and the mobile side are both Claude, so the
+commits are indistinguishable by author.
+
+What works: re-baseline from **local HEAD** after any push of your own, which is what a
+`git pull` + `git rev-parse HEAD` naturally gives you. Worth stating in the preamble as a rule for any
+watcher whose subject the agent also writes to, because the failure looks exactly like a real event.
+
+## 4. `timeout` is not available, and the preamble insists on it
+
+Rule 6 says *"`timeout` every network call, or a hung one leaves it alive and blind."* Correct in
+substance. But this machine is a Mac, and **macOS ships neither `timeout` nor `gtimeout`** — so the rule
+as written cannot be followed, and following it literally produces a job that dies instantly on
+`command not found`.
+
+What was used instead, and what the preamble should suggest for a git probe:
+
+```
+GIT_SSH_COMMAND='ssh -o ConnectTimeout=15 -o ServerAliveInterval=5 -o ServerAliveCountMax=3'
+```
+
+The general point: the rule should name the *outcome* (no unbounded network call) and give a portable
+example, rather than name a binary that is missing on the machine most of this runs on.
+
+## 5. A job the owner stops is reported as a failure
+
+Stopping a watcher by hand surfaced as **"failed with exit code 144"**. Rule 7 already anticipates the
+confusion from the agent's side — *"an exit is at least as likely to be me as it is to be a failure"* —
+but the harness still presents a deliberate stop as a fault.
+
+For Command HQ this matters more than it does in `/chat`: a grid of cards showing "failed" for every
+watcher the owner turned off would be actively misleading. Worth distinguishing stopped-by-request from
+died, at whatever layer knows the difference.
+
+## 6. The working directory offered was the wrong one
+
+The dialog filled in `/Users/pastilhas` — the conversation's cwd — for a job that had to run inside a
+git checkout. It was correct behaviour, and still the wrong answer, because the directory a watcher
+needs is a property of what it watches, not of where the conversation happens to be.
+
+Command HQ makes this moot for commanders, since they all share one directory by design. Noting it for
+the dialog as used from `/chat`.
+
+---
+
+## What worked, and should not be traded away
+
+**"Ask before arming" earned its place immediately.** One question — exit or keep going — changed the
+whole design of the job. The instruction that *a question is the only thing that may come before
+starting* is good discipline and produced exactly the right conversation.
+
+**The output rules are right.** One line, no preface, no reading the output back to check it started.
+
+**One caveat on those rules:** they forbid saying anything except a question or that one line, which
+left nowhere to report a *mechanical impossibility* — that "keep going" cannot notify the agent. That
+was worth more than the terseness cost. Suggest allowing, explicitly, a statement of the form "what you
+asked for cannot work as specified, here is why" alongside the question.
+
+**The preamble text itself is the feature.** It is what makes a job tracked rather than orphaned, and
+copying it verbatim into the mobile client — diffed to confirm — was the right call over paraphrasing.