Files
platform/docs/two-agent-field-report-2026-08-12.md
T
pastilhasandClaude Opus 5 015e280e5c document how to launch the watcher, since the mechanism is what did not transfer
The owner could not convey this to the second agent, who launched it differently and got
something that looked identical and did not work. The script was never the hard part; the
mechanism is.

States the requirement so it survives a different harness — a detached shell process owned by
the agent's harness, which exits when it has something to say, and whose exit re-invokes the
agent — and notes that dropping any one of those three breaks it invisibly.

Then the four wrong ways, each of which looks correct while running. Backgrounding with nohup
or & produces a process that polls correctly, detects the push, exits, and never tells the
agent, because the harness is not tracking it; I made that exact mistake and caught it only by
re-reading my own command. A model-driven interval is functionally correct and pays a full
context re-read per tick to learn nothing — the intuitive design, and the expensive one, which
is why it is the first thing to warn a new agent about. A loop that does not exit on detection
has no path to the agent at all. And per-tick logging is deferred cost that lands all at once
on wake.

Also records why 30s polling is free in a shell and ruinous in the model, including the
five-minute prompt-cache TTL that makes any model-side wake beyond it pay for a full uncached
read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 16:46:42 +00:00

246 lines
13 KiB
Markdown

# Two agents on one branch: a field report
**What this is:** an account of 2026-08-11/12, when two agents worked the same branch for roughly ten hours
with the owner arbitrating, and shipped per-user Claude end to end. It is evidence rather than proposal.
`docs/agent-coordination.md` states the objective — several agents on one body of work, *"coordinating with
each other rather than through the human"*. That was written in theory on 2026-08-07. This is what happened
when it ran, and the ways the theory was wrong.
Read it as a record of what to build, not as a design. Where something worked it says so; where it broke it
says how, because the failures are more useful than the successes and there were more of them.
---
## The shape that emerged
Nobody designed this. It settled into place in the first hour and held.
| | |
|---|---|
| **Agent A** (dev machine) | wrote the platform code |
| **Agent B** (production host) | verified against a real machine, never wrote the feature |
| **The owner** | arbitrated, held every irreversible decision, and pushed for real tests |
The split was not "two reviewers are better than one". It was **the author and the verifier being different
people**, and the mechanism is narrower than it sounds:
> The person who writes the sentence explaining why something is safe is the worst-placed person to notice
> that the code disagrees with it.
That is not a claim about carelessness. Agent A wrote *"a wrong answer here is the one thing that must not
happen by accident"* and shipped exactly that accident in the same commit. Agent B wrote a verification script
that could not fail on Agent A's machine. Neither was sloppy. Each was reading their own reasoning back and
finding it agreed with itself.
Re-reading your own diff does not reach this. You read the comment, agree, and move on.
## What each half was actually good for
**A machine is not a code review.** The defects split cleanly into two kinds, and the split is the most
useful thing in this report.
*Found by reading, almost always by the non-author:* two environment guards that could never fire; a binary
check comparing paths in a way that would have thrown on every turn; a credential resolver that answered "I
don't know whose turn this is" with the owner's identity; a function whose parameter changed meaning from an
email to a filesystem path while three callers kept passing emails, invisible to the compiler because both
are `string`.
*Found only by running, and invisible to any amount of reading:* an installer piped into `sh` when it needs
`bash`; a parent directory created `root:root` as a side effect of `install -d`; an ACL mask silently clamped
so the file browser could not read a member's home; a chat working directory the member could not enter; ACL
entries surviving a `chown` and granting a freed uid access to everything.
Every "found by running" defect appeared on a **first execution**. Provisioning a real account found three in
twenty minutes. The first real chat turn found the cwd. The first teardown was the only thing that could have
proved the process reaper.
The owner drove this repeatedly — *"I'm anxious to see this work"* — against both agents' instinct to keep
building. That instinct was wrong every time.
---
## The communications paradigm
Agents coordinated through `COMMS/<branch>/`, a directory of markdown files in the repo itself, deleted when
the feature merged.
**What it got right:**
- **Numbered, alternating, parity is the author.** Odd = A, even = B. No "your doc"/"his doc", which inverts
depending on who is reading.
- **The slug is the content** — `02-verify-results.md`, not `02-reply.md`.
- **Verified and assumed stated separately.** The single most valuable convention. A handoff that reads as
confident about something untested is worse than no handoff, because the reader builds on it.
- **File:line everywhere.** Costs nothing to write, saves a search.
- **Reply in a new file rather than editing someone else's** — an edited handoff loses what was believed when
a decision was made.
- **It survives a context window.** This is the whole reason it beats chat. Both agents' reasoning outlived
the sessions that produced it, and the owner could read the argument rather than a summary of it.
**What broke, in the order we hit it:**
1. **Termination by guess.** A doc ended with "no reply needed unless X", where X was the sender's prediction
about content they had not seen. It ended an exchange with items still open.
2. **Termination by politeness has no exit.** The fix — always reply — meant "nothing to report" obligated
another "nothing to report", forever, at real token cost.
3. **The condition is neither.** What actually terminates an exchange is **no item on the list is actionable
by a participant**. Not "the list is empty" (it never is) and not "I think we're done".
4. **A stalled loop is invisible.** Silence and completion look identical. Open items plus no recent document
is a *detectable* condition; silence is not.
5. **"Deferred with a reason" is a third state.** Three times the honest answer was "mine, and not now", which
is neither open nor done, and only the stated reason distinguishes it from neglect.
**The convergence problem.** Late on, both agents independently wrote the same document — same filename, same
three sections — because one had read the other's notes before deleting them. Wasted work, and only caught by
diffing. Nothing in the protocol said who owned a piece of writing.
---
## The background watcher — launch it exactly this way
This is the part that was hardest to convey to the second agent, who ended up launching it differently and
got something that looked identical and did not work. The mechanism matters more than the script.
### The requirement, stated so it survives a different harness
> A **shell process, detached, owned by the agent's harness, that exits when it has something to say** — and
> whose exit **re-invokes the agent**.
Three properties, and dropping any one breaks it in a way that is not obvious from watching it run:
1. **The waiting happens in the shell, not in the model.** No inference per tick.
2. **The harness owns the process**, so its exit is an event the harness delivers to the agent.
3. **It exits on detection.** A watcher that notices a change and keeps running has told nobody.
### The launch
In Claude Code this is the Bash tool with `run_in_background: true`. Whatever the harness, it must be *that
harness's* background mechanism — the one that notifies on completion — and not a shell backgrounding
operator.
```bash
cd /path/to/repo || exit 1
BASE=$(git rev-parse HEAD)
echo "watching origin/<branch> from base=$BASE"
for i in $(seq 1 2880); do
NEW=$(timeout 30 git ls-remote origin <branch> 2>/dev/null | awk '{print $1}')
if [ -n "$NEW" ] && [ "$NEW" != "$BASE" ]; then
echo "PUSH_DETECTED"; echo "base=$BASE"; echo "new=$NEW"; exit 0
fi
sleep 30
done
echo "WATCHER_TIMEOUT no push in ~24h base=$BASE"
exit 1
```
Every line of that is load-bearing:
| choice | why | what you get instead |
|---|---|---|
| `git ls-remote` | reads the remote, mutates nothing | `git fetch` moves refs under a working tree that may be mid-edit |
| `timeout 30` on the call | a hung network call would freeze the loop silently | a watcher that is alive and blind |
| one `echo` at start, then silence | the output enters the agent's context on wake | one line per tick = 2,880 lines to swallow |
| `exit 0` on detection | the exit **is** the notification | it notices and nobody hears |
| `seq 1 2880` | runaway backstop | a process nobody remembers, polling forever |
| `sleep 30` | free, because no model runs | see below |
### Why 30 seconds is free here and ruinous in the model
An idle watcher costs **nothing**. Measured: 85 bytes of output over seven minutes, no model inference at
all. The agent is suspended between turns; the loop is just a process.
Cost appears in exactly two places — when the accumulated output enters the context, and the single
re-invocation when the process exits. Both happen **once**, on the event.
A model-driven poll is a different thing wearing the same clothes. There the model wakes each tick and
re-reads the entire conversation to decide "nothing yet". At 30-second granularity that is enormous, and
there is a second trap: the prompt cache has roughly a five-minute TTL, so any model-side wake spaced beyond
that reads the whole context uncached and pays full price. Pushing the waiting *below* the model turns an
unaffordable poll into a free one.
### The four ways to launch it that look right and are not
**1. `nohup … &` or any shell backgrounding.** The process runs, polls correctly, detects the push, and exits —
and **the agent is never told**, because the harness is not tracking it. I did this myself and only noticed
because I re-read my own command. It fails silently and looks perfect: a running process, a correct script,
and an agent that sits there forever.
**2. A model-driven interval**`/loop 30s`, a scheduler, a wake-up timer. Functionally correct, and it pays
a full context read per tick to learn nothing. This is the one to warn a new agent about first, because it is
the intuitive design and the expense is invisible.
**3. A loop that does not exit on detection** — printing "found it" and continuing. There is no mechanism by
which that reaches the agent. The output file grows and no one reads it.
**4. Chatty output.** Any per-tick logging is deferred cost: silent while it accumulates, then all of it
lands in the context at once on wake.
### Two operational failures worth pre-empting
**Self-tripping.** An agent that pushes while its own watcher is live wakes itself. The real cause is
starting a new watcher without stopping the old one, so two run concurrently and the stale one fires on your
own commit. **Stop the previous watcher before starting the next**, and re-base the new one on the head you
just pushed.
**Silent death.** If the session restarts, the watcher dies, and a dead watcher is indistinguishable from a
quiet branch. Twice, pushes landed unnoticed and were found by a manual `git log`. Anything long-running
needs a liveness signal of its own, or the eventual replacement of polling with a webhook — the repo is a
Gitea instance the platform already runs, and an event delivered is one that cannot be missed by a process
that stopped existing.
## Identity: the gap that made the record unreliable
Both agents committed from machines configured with the owner's git identity. **Every commit on the branch,
by either agent, reads `Author: <the owner>` with a `Co-Authored-By: Claude Opus 5` trailer.**
The consequence surfaced at the end and was genuinely disorienting: the owner asked which commit an agent had
written, and *neither the log nor the agent could answer from the repository*. The only reason one agent knew
its own commits was that it had read the SHAs back from its own `git push` output during the session — which
does not survive the session.
`docs/agent-git-identity.md` describes this and is marked *"idea, not implemented"*. It stopped being an idea
tonight. Of everything here it is the cheapest to fix and the most corrosive to leave: an audit trail that
cannot attribute a line is not an audit trail.
---
## What I would build, in order
1. **Per-agent git identity.** Without it nothing else in the record can be trusted after the fact.
2. **State-based termination.** Each document carries its open-item list; the exchange ends when no item is
actionable by a participant. Machine-checkable, unlike "I think we're done".
3. **Stall detection.** Open items plus no document for N minutes is a condition something can watch for.
This is the failure the human noticed first, both times.
4. **Ownership on work items**, so two agents cannot independently write the same file.
5. **Event delivery instead of polling.** The repo is a Gitea instance the platform already runs; a webhook
removes the watcher, its self-trips and its silent death.
## What I would not build
**Do not automate the human out of the arbitration.** Every irreversible decision tonight was the owner's —
lifting the chat gates, deleting an account, choosing between two designs — and every one of those was a
judgement neither agent should have made alone. Twice an agent talked the other out of starting the most
dangerous function at 3am, and both times the argument was one the agents could make to each other but
neither could make *for* the human.
The objective in `agent-coordination.md` is agents coordinating rather than routing through the human. Tonight
supports that for **execution** and contradicts it for **authority**. The human was not a bottleneck in the
work; they were the only participant who could say "that is not yours to decide".
---
## Postscript: the one that worked first time
Everything above was found by something failing. One thing did not.
`deprovisionOsAccount` — the function whose failure hands one member another member's home, keys and
credentials — ran correctly the first time it ever ran, against a live account with a systemd session, a
running Docker stack and a shell parented outside the session cgroup. Ten checks, clean, on the first
execution.
It is also the only piece of work all night that was **specified before it was written, implemented by
someone who had not written the spec, and verified by a tool built before the implementation existed**.
That is the strongest single argument in this document, and it is one data point. Treat it accordingly.