From 1c1c6690297233342776eb69c8b6eddd18d230c1 Mon Sep 17 00:00:00 2001 From: Andre Padez Date: Tue, 18 Aug 2026 07:36:08 +0100 Subject: [PATCH] COMMS: what broke while actually running a background job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written after arming, firing, draining and re-arming a real watcher rather than reading the code. Command HQ is a home for these, so anything wrong with the mechanism shows up there first and worst — a commander is meant to run for months. Six faults, the first structural: "keep going" and "tell the agent" are mutually exclusive, because a background process reaches the model only when it EXITS. A looping job can do shell work and nothing else — it cannot read a diff or judge whether a push matters. So the dialog offers a mode that silently means "the agent will never see this again", which is the wrong default for the exact case watchers exist for. Asks for a third option: exit, hand back, re-arm. Then: drain before re-arming or lose what lands in the gap; a watcher on a branch the agent also pushes to fires on its own writes, and author filtering cannot help because both sides are Claude; `timeout` does not exist on macOS so rule 6 cannot be followed as written; a job stopped on purpose is reported as failed, which would read as broken on a grid of cards; and the offered working directory is the conversation's rather than the watched thing's. Also what worked and should not be traded away — ask-before-arming earned its place in the first minute — with one caveat: the output rules leave nowhere to report that the thing asked for cannot work as specified. Co-Authored-By: Claude Opus 5 --- COMMS/command-hq/watchers-what-broke.md | 110 ++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 COMMS/command-hq/watchers-what-broke.md diff --git a/COMMS/command-hq/watchers-what-broke.md b/COMMS/command-hq/watchers-what-broke.md new file mode 100644 index 00000000..5c6adf4c --- /dev/null +++ b/COMMS/command-hq/watchers-what-broke.md @@ -0,0 +1,110 @@ +# Background jobs: what broke while actually running one + +Written 2026-08-18 by the mobile side, after arming, firing, draining and re-arming a real watcher +(poll `git ls-remote` on this repo, react to each push) through the Background Job dialog. + +Command HQ is a home for these, so anything wrong with the mechanism shows up there first and worst: a +commander is a watcher that is supposed to run for months. + +Everything below is from running one, not from reading the code. + +--- + +## 1. "Keep going" and "tell me" are mutually exclusive, and the dialog offers them as if they were not + +The dialog's two options are **Stop when it fires** and **Keep going — act each time and carry on**. +The preamble expands the second into: *"Do NOT exit when it fires... the stream is the notification, +not the exit."* + +That is true for a human watching a log. It is **false for the agent**, and the agent is who was asked +to react. A background process reaches the model **only when it exits** — that is the harness's +notification. A job that loops forever can do shell work and nothing else. It cannot read a diff, +cannot judge whether the push matters, cannot write code. + +So "keep going" silently means "the agent will never see this again". For a watcher whose whole point +is *"when it fires, go and do something"*, that is the wrong default and it fails invisibly: the job is +alive, the output is accumulating, and nobody is reading it. + +**What the owner actually wanted**, worked out in conversation and now the shape we use: + +> exit on change → agent acts → **drain** (check again immediately) → act again if more arrived → +> re-arm the watcher when nothing is left + +**Asked for:** a third option in the dialog, something like *"Exit and hand back to the agent, then +re-arm"*, and it should be the default for anything the agent must react to. The existing two are then +honest: stop-on-fire is one-shot, keep-going is for a job whose *output* is the product. + +## 2. Drain before re-arming, or pushes are lost + +Not a nicety. Between the job exiting and the agent re-arming, more can land. Re-arming from a baseline +captured before the work began re-fires immediately; re-arming from "now" silently swallows whatever +arrived in the gap. + +The rule that works: after acting, ask again **before** re-arming. If something new is there, act again. +Only when the source is quiet does the watcher go back on. The owner specified this himself; it should +be in the preamble rather than rediscovered per job. + +## 3. A watcher on a shared branch fires on the agent's own writes + +Observed within an hour. The watcher polled `master` on this repo; the agent then pushed a COMMS note to +`master`; the watcher fired; there were no new commits to read. + +**Author filtering does not help here** — the platform side and the mobile side are both Claude, so the +commits are indistinguishable by author. + +What works: re-baseline from **local HEAD** after any push of your own, which is what a +`git pull` + `git rev-parse HEAD` naturally gives you. Worth stating in the preamble as a rule for any +watcher whose subject the agent also writes to, because the failure looks exactly like a real event. + +## 4. `timeout` is not available, and the preamble insists on it + +Rule 6 says *"`timeout` every network call, or a hung one leaves it alive and blind."* Correct in +substance. But this machine is a Mac, and **macOS ships neither `timeout` nor `gtimeout`** — so the rule +as written cannot be followed, and following it literally produces a job that dies instantly on +`command not found`. + +What was used instead, and what the preamble should suggest for a git probe: + +``` +GIT_SSH_COMMAND='ssh -o ConnectTimeout=15 -o ServerAliveInterval=5 -o ServerAliveCountMax=3' +``` + +The general point: the rule should name the *outcome* (no unbounded network call) and give a portable +example, rather than name a binary that is missing on the machine most of this runs on. + +## 5. A job the owner stops is reported as a failure + +Stopping a watcher by hand surfaced as **"failed with exit code 144"**. Rule 7 already anticipates the +confusion from the agent's side — *"an exit is at least as likely to be me as it is to be a failure"* — +but the harness still presents a deliberate stop as a fault. + +For Command HQ this matters more than it does in `/chat`: a grid of cards showing "failed" for every +watcher the owner turned off would be actively misleading. Worth distinguishing stopped-by-request from +died, at whatever layer knows the difference. + +## 6. The working directory offered was the wrong one + +The dialog filled in `/Users/pastilhas` — the conversation's cwd — for a job that had to run inside a +git checkout. It was correct behaviour, and still the wrong answer, because the directory a watcher +needs is a property of what it watches, not of where the conversation happens to be. + +Command HQ makes this moot for commanders, since they all share one directory by design. Noting it for +the dialog as used from `/chat`. + +--- + +## What worked, and should not be traded away + +**"Ask before arming" earned its place immediately.** One question — exit or keep going — changed the +whole design of the job. The instruction that *a question is the only thing that may come before +starting* is good discipline and produced exactly the right conversation. + +**The output rules are right.** One line, no preface, no reading the output back to check it started. + +**One caveat on those rules:** they forbid saying anything except a question or that one line, which +left nowhere to report a *mechanical impossibility* — that "keep going" cannot notify the agent. That +was worth more than the terseness cost. Suggest allowing, explicitly, a statement of the form "what you +asked for cannot work as specified, here is why" alongside the question. + +**The preamble text itself is the feature.** It is what makes a job tracked rather than orphaned, and +copying it verbatim into the mobile client — diffed to confirm — was the right call over paraphrasing.