platform/COMMS/command-hq/watchers-what-broke.md

# Background jobs: what broke while actually running one

Written 2026-08-18 by the mobile side, after arming, firing, draining and re-arming a real watcher
(poll `git ls-remote` on this repo, react to each push) through the Background Job dialog.

Command HQ is a home for these, so anything wrong with the mechanism shows up there first and worst: a
commander is a watcher that is supposed to run for months.

Everything below is from running one, not from reading the code.

---

## 1. "Keep going" and "tell me" are mutually exclusive, and the dialog offers them as if they were not

The dialog's two options are **Stop when it fires** and **Keep going — act each time and carry on**.
The preamble expands the second into: *"Do NOT exit when it fires... the stream is the notification,
not the exit."*

That is true for a human watching a log. It is **false for the agent**, and the agent is who was asked
to react. A background process reaches the model **only when it exits** — that is the harness's
notification. A job that loops forever can do shell work and nothing else. It cannot read a diff,
cannot judge whether the push matters, cannot write code.

So "keep going" silently means "the agent will never see this again". For a watcher whose whole point
is *"when it fires, go and do something"*, that is the wrong default and it fails invisibly: the job is
alive, the output is accumulating, and nobody is reading it.

**What the owner actually wanted**, worked out in conversation and now the shape we use:

> exit on change → agent acts → **drain** (check again immediately) → act again if more arrived →
> re-arm the watcher when nothing is left

**Asked for:** a third option in the dialog, something like *"Exit and hand back to the agent, then
re-arm"*, and it should be the default for anything the agent must react to. The existing two are then
honest: stop-on-fire is one-shot, keep-going is for a job whose *output* is the product.

## 2. Drain before re-arming, or pushes are lost

Not a nicety. Between the job exiting and the agent re-arming, more can land. Re-arming from a baseline
captured before the work began re-fires immediately; re-arming from "now" silently swallows whatever
arrived in the gap.

The rule that works: after acting, ask again **before** re-arming. If something new is there, act again.
Only when the source is quiet does the watcher go back on. The owner specified this himself; it should
be in the preamble rather than rediscovered per job.

## 3. A watcher on a shared branch fires on the agent's own writes

Observed within an hour. The watcher polled `master` on this repo; the agent then pushed a COMMS note to
`master`; the watcher fired; there were no new commits to read.

**Author filtering does not help here** — the platform side and the mobile side are both Claude, so the
commits are indistinguishable by author.

What works: re-baseline from **local HEAD** after any push of your own, which is what a
`git pull` + `git rev-parse HEAD` naturally gives you. Worth stating in the preamble as a rule for any
watcher whose subject the agent also writes to, because the failure looks exactly like a real event.

## 3b. Re-arming without stopping the old one leaves two watchers

Mine, not the mechanism's, and worth recording because the mechanism invites it. Having acted on a
fire, I armed a fresh watcher while the previous one was still alive — the old one had fired but its
process had not been reaped yet. Two watchers on the same branch then produce two notifications for one
event, and the second looks exactly like a real second push.

It compounds §3: the agent pushes, both watchers fire, and neither has anything to report.

The discipline that works is dull and reliable: **stop the watcher before you push, re-arm after.** Not
"re-arm when convenient". A `pgrep` for the job before arming a new one costs nothing and catches it.

For Command HQ this is a real hazard rather than a tidiness point — a commander that has been
accidentally armed twice is two processes doing the same work, and the grid would show one card.

## 4. `timeout` is not available, and the preamble insists on it

Rule 6 says *"`timeout` every network call, or a hung one leaves it alive and blind."* Correct in
substance. But this machine is a Mac, and **macOS ships neither `timeout` nor `gtimeout`** — so the rule
as written cannot be followed, and following it literally produces a job that dies instantly on
`command not found`.

What was used instead, and what the preamble should suggest for a git probe:

```
GIT_SSH_COMMAND='ssh -o ConnectTimeout=15 -o ServerAliveInterval=5 -o ServerAliveCountMax=3'
```

The general point: the rule should name the *outcome* (no unbounded network call) and give a portable
example, rather than name a binary that is missing on the machine most of this runs on.

## 5. A job the owner stops is reported as a failure

Stopping a watcher by hand surfaced as **"failed with exit code 144"**. Rule 7 already anticipates the
confusion from the agent's side — *"an exit is at least as likely to be me as it is to be a failure"* —
but the harness still presents a deliberate stop as a fault.

For Command HQ this matters more than it does in `/chat`: a grid of cards showing "failed" for every
watcher the owner turned off would be actively misleading. Worth distinguishing stopped-by-request from
died, at whatever layer knows the difference.

## 6. The working directory offered was the wrong one

The dialog filled in `/Users/pastilhas` — the conversation's cwd — for a job that had to run inside a
git checkout. It was correct behaviour, and still the wrong answer, because the directory a watcher
needs is a property of what it watches, not of where the conversation happens to be.

Command HQ makes this moot for commanders, since they all share one directory by design. Noting it for
the dialog as used from `/chat`.

---

## What worked, and should not be traded away

**"Ask before arming" earned its place immediately.** One question — exit or keep going — changed the
whole design of the job. The instruction that *a question is the only thing that may come before
starting* is good discipline and produced exactly the right conversation.

**The output rules are right.** One line, no preface, no reading the output back to check it started.

**One caveat on those rules:** they forbid saying anything except a question or that one line, which
left nowhere to report a *mechanical impossibility* — that "keep going" cannot notify the agent. That
was worth more than the terseness cost. Suggest allowing, explicitly, a statement of the form "what you
asked for cannot work as specified, here is why" alongside the question.

**The preamble text itself is the feature.** It is what makes a job tracked rather than orphaned, and
copying it verbatim into the mobile client — diffed to confirm — was the right call over paraphrasing.