platform/docs/two-agent-field-report-2026-08-12.md

Two agents on one branch: a field report

What this is: an account of 2026-08-11/12, when two agents worked the same branch for roughly ten hours with the owner arbitrating, and shipped per-user Claude end to end. It is evidence rather than proposal.

docs/agent-coordination.md states the objective — several agents on one body of work, "coordinating with each other rather than through the human". That was written in theory on 2026-08-07. This is what happened when it ran, and the ways the theory was wrong.

Read it as a record of what to build, not as a design. Where something worked it says so; where it broke it says how, because the failures are more useful than the successes and there were more of them.

---

The shape that emerged

Nobody designed this. It settled into place in the first hour and held.

Agent A (dev machine)	wrote the platform code
Agent B (production host)	verified against a real machine, never wrote the feature
The owner	arbitrated, held every irreversible decision, and pushed for real tests

The split was not "two reviewers are better than one". It was the author and the verifier being different people, and the mechanism is narrower than it sounds:

The person who writes the sentence explaining why something is safe is the worst-placed person to notice that the code disagrees with it.

That is not a claim about carelessness. Agent A wrote "a wrong answer here is the one thing that must not happen by accident" and shipped exactly that accident in the same commit. Agent B wrote a verification script that could not fail on Agent A's machine. Neither was sloppy. Each was reading their own reasoning back and finding it agreed with itself.

Re-reading your own diff does not reach this. You read the comment, agree, and move on.

What each half was actually good for

A machine is not a code review. The defects split cleanly into two kinds, and the split is the most useful thing in this report.

Found by reading, almost always by the non-author: two environment guards that could never fire; a binary check comparing paths in a way that would have thrown on every turn; a credential resolver that answered "I don't know whose turn this is" with the owner's identity; a function whose parameter changed meaning from an email to a filesystem path while three callers kept passing emails, invisible to the compiler because both are string.

Found only by running, and invisible to any amount of reading: an installer piped into sh when it needs bash; a parent directory created root:root as a side effect of install -d; an ACL mask silently clamped so the file browser could not read a member's home; a chat working directory the member could not enter; ACL entries surviving a chown and granting a freed uid access to everything.

Every "found by running" defect appeared on a first execution. Provisioning a real account found three in twenty minutes. The first real chat turn found the cwd. The first teardown was the only thing that could have proved the process reaper.

The owner drove this repeatedly — "I'm anxious to see this work" — against both agents' instinct to keep building. That instinct was wrong every time.

---

The communications paradigm

Agents coordinated through COMMS/<branch>/ — markdown files committed to the repo, alongside the code they discuss, deleted when the feature merged.

Why a directory in the repo and not chat. It survives a context window. Both agents' reasoning outlived the sessions that produced it, a third party could read the argument rather than a summary of it, and it travels with the branch. Chat has none of those properties, and the owner relaying findings by hand between two agents at the end of long sessions is the failure this replaces.

The rules, as they ended up

Location and lifetime. COMMS/<branch-name>/. It is a channel, not documentation: when the feature merges, the directory is deleted. Anything that will still be true in a month must be moved to docs/ or next to the code before the merge, or it is lost. We nearly lost three findings this way and only caught it because someone checked.

Numbered, alternating, parity is the author. One agent takes odd numbers, the other even. 01, 02, 03… Never "your doc" / "his doc", which inverts depending on who is reading. The parity is the attribution, and given that git could not attribute anything (see below), it was the only attribution that worked.

Numbers are ordered, not necessarily consecutive. An agent needing two in a row takes 03 and 05 and leaves 04 unused, rather than forcing a reply out of the other side to keep the count. A gap is legal and means "no turn was taken".

The slug is the content. 02-verify-results.md, 24-resolvememberrun-fails-open.md. Not 02-reply.md. The filename is the index; a reader should know whether to open it without opening it.

Reply in a new file. Never edit someone else's. An edited handoff loses what was believed at the moment a decision was made, which is usually the thing that explains the decision.

Editing your own is allowed if it has not been read — and say so in the commit. Better than a prediction standing next to its own correction in two documents.

Refer to commits by SHA, never by branch name. Three remotes were in play with the same branch names on each; one agent's origin was the other's pertento. A SHA is the only unambiguous reference, and this cost real time before it was noticed.

What a handoff must contain

This is the part that carried the most weight, and it is one rule:

State what you verified and what you assumed, separately and explicitly.

A handoff that reads as confident about something untested is worse than no handoff, because the reader builds on it. Every serious mistake of the night traces back to something asserted with more confidence than it had been earned.

In practice, each document ended up with:

• What changed — with file:line throughout. Costs nothing to write, saves the reader a search, and makes a claim checkable rather than believable.
• VERIFIED — what was actually run, on what, with the output.
• NOT VERIFIED — stated as prominently as the verified part. provisionClaudeCli carried "never executed anywhere" through four documents, and that label is what eventually made someone run it.
• What I am least sure of — the author's own suspicions. One agent listed three; the second was a real defect, found because it had been pointed at.
• What I did not do — so nobody assumes it. "I did not restart anything", "I did not touch the gates".
• Open items with an owner — see termination, below.

Termination: the rule that took four attempts

This broke more times than anything else, so the failures are worth listing in order:

1. Terminate by guess. NO REPLY NEEDED unless the test fails — a prediction about content the sender had not seen. It ended an exchange with items open.
2. Terminate by politeness. The fix — always reply, even with nothing to say — has no exit. "Nothing to report" obligates another "nothing to report", indefinitely, at real cost.
3. Terminate when the list is empty. Too strong: the list is never empty and will not be for days.
4. What actually works: the exchange pauses when no open item is actionable by a participant.

That last one is checkable rather than felt. Everything remaining is either the human's, or deferred with a stated reason, and either side reopens it by adding an item that is theirs.

Three states, not two. An item is open / done / deferred with a reason. Three times the honest answer was "mine, and not now" — and only the reason distinguishes that from neglect. A protocol with two states forces an agent to lie in one direction or the other.

A stall must be detectable. Silence and completion look identical from outside. Open items plus no document for N minutes is a condition a machine can watch for; silence is not. The human noticed both stalls before either agent did, which is the wrong way round.

Ownership, which we did not have and needed

Late on, both agents independently wrote the same document — same filename, same three sections — because one had read the other's notes before deleting them. Pure waste, caught only by diffing the two files.

Nothing in the protocol said who owned a piece of work. Adding it is cheap: an open item names its owner, and an agent picking up an unowned item claims it in a document before starting.

A skeleton to copy

# NN — <what this is about in one line>

Commits read: <sha>..<sha>. Answering `<NN-1>`.

**Verdict / what changed** — one paragraph, file:line.

## VERIFIED
<what was actually run, on what, with output>

## NOT VERIFIED
<stated as prominently as the above>

## What I am least sure of
<your own suspicions, numbered>

## What I did not do
<so nobody assumes it>

## Open items
| item | owner | state |
|---|---|---|
| … | me / you / the human | open / deferred (reason) |

---

The review discipline

"Verify" turned out to mean something more specific than reading a diff. What actually caught defects:

Check the enforcement, not the description. A document says a check is scoped by user; go read the line that compares. Twice the description was right and the code did something else — and the author had read their own description and agreed with it.

Run it against a real machine. Every defect that mattered was found this way, on a first execution. The categories at the top of this report are not a coincidence.

A check that has never been seen failing is not evidence. A verification script was run against a live, fully-provisioned account specifically to watch it fail; it reported 8 of 9 failures, which is what made the later clean result meaningful. Related: a skipped test must announce itself, or an unconfigured run reads as a pass.

Distrust vacuous passes. Three separate times something passed because it had not actually looked: a subuid scan on a tree with no subuid-owned files; a search root that did not exist, where every check reports "ok" on finding nothing; and a range scan handed a non-numeric argument. Any checker whose checks are "look for X, report ok if absent" must refuse to run when its inputs are wrong, rather than pass.

Expect stacked bugs. Fixing the visible failure reveals the next one underneath. A container failed on a mount-point guard; fixing that revealed an ACL traversal denial. An installer failed on the wrong shell; fixing that would have revealed a root-owned parent directory. Never report "fixed" from a diff — only from a run.

Distrust "it is inert today". Several things were safe only because a gate was up. That is a statement about the present, and the entire purpose of the work was to remove the gate. Review inert code as if it were live, because the commit that makes it live will be reviewed as if it were already correct.

Fail closed, and check which way "unknown" resolves. The most dangerous defect of the night was a resolver that answered "I could not determine whose turn this is" with the owner's identity. Any place where an unknown collapses into a privileged default is worth a specific look.

---

Failure modes to expect

Collected from the night, phrased so an agent can pattern-match against them:

pattern	what it looked like here
Author reviews own sentence	"a wrong answer here must not happen by accident" shipped with that accident
Vacuous pass	checker with a missing search root printing CLEAN
Stacked bugs	PG18 mount guard hiding an ACL traversal denial
Inert-today reasoning	unreachable code reviewed less carefully than reachable code
Unknown resolves to privileged	failed lookup → run as the owner
Compiler cannot help	a parameter changing meaning from email to path, both string
Guard that cannot fire	a denylist tested against an object built from an allowlist
Side-effect creation	install -d making a parent root:root
Mode bits vs ACLs	chown severing ownership and leaving access
Tail-of-session work	three of the night's bugs written after hour eight

---

Git hygiene for two agents on one branch

Small, and it bit us repeatedly:

• Pull before you push, and expect a race. Both agents pushed within the same minute more than once; one rebase was needed mid-review.
• Merge, verify, then delete. A branch was deleted after an aborted fast-forward — master had moved — and the commits survived only because git had not yet garbage-collected them. Verify the merge landed before removing the only ref to it.
• A doc-only commit still deserves a real message. These commit messages are the durable record once COMMS/ is deleted; several findings in this repo now exist only in a commit body.
• Say which remote. See the SHA rule above.

The background watcher — launch it exactly this way

This is the part that was hardest to convey to the second agent, who ended up launching it differently and got something that looked identical and did not work. The mechanism matters more than the script.

The requirement, stated so it survives a different harness

A shell process, detached, owned by the agent's harness, that exits when it has something to say — and whose exit re-invokes the agent.

Three properties, and dropping any one breaks it in a way that is not obvious from watching it run:

1. The waiting happens in the shell, not in the model. No inference per tick.
2. The harness owns the process, so its exit is an event the harness delivers to the agent.
3. It exits on detection. A watcher that notices a change and keeps running has told nobody.

The launch

In Claude Code this is the Bash tool with run_in_background: true. Whatever the harness, it must be that harness's background mechanism — the one that notifies on completion — and not a shell backgrounding operator.

cd /path/to/repo || exit 1
BASE=$(git rev-parse HEAD)
echo "watching origin/<branch> from base=$BASE"
for i in $(seq 1 2880); do
  NEW=$(timeout 30 git ls-remote origin <branch> 2>/dev/null | awk '{print $1}')
  if [ -n "$NEW" ] && [ "$NEW" != "$BASE" ]; then
    echo "PUSH_DETECTED"; echo "base=$BASE"; echo "new=$NEW"; exit 0
  fi
  sleep 30
done
echo "WATCHER_TIMEOUT no push in ~24h base=$BASE"
exit 1

Every line of that is load-bearing:

choice	why	what you get instead
git ls-remote	reads the remote, mutates nothing	git fetch moves refs under a working tree that may be mid-edit
timeout 30 on the call	a hung network call would freeze the loop silently	a watcher that is alive and blind
one echo at start, then silence	the output enters the agent's context on wake	one line per tick = 2,880 lines to swallow
exit 0 on detection	the exit is the notification	it notices and nobody hears
seq 1 2880	runaway backstop	a process nobody remembers, polling forever
sleep 30	free, because no model runs	see below

Why 30 seconds is free here and ruinous in the model

An idle watcher costs nothing. Measured: 85 bytes of output over seven minutes, no model inference at all. The agent is suspended between turns; the loop is just a process.

Cost appears in exactly two places — when the accumulated output enters the context, and the single re-invocation when the process exits. Both happen once, on the event.

A model-driven poll is a different thing wearing the same clothes. There the model wakes each tick and re-reads the entire conversation to decide "nothing yet". At 30-second granularity that is enormous, and there is a second trap: the prompt cache has roughly a five-minute TTL, so any model-side wake spaced beyond that reads the whole context uncached and pays full price. Pushing the waiting below the model turns an unaffordable poll into a free one.

The four ways to launch it that look right and are not

1. nohup … & or any shell backgrounding. The process runs, polls correctly, detects the push, and exits — and the agent is never told, because the harness is not tracking it. I did this myself and only noticed because I re-read my own command. It fails silently and looks perfect: a running process, a correct script, and an agent that sits there forever.

2. A model-driven interval — /loop 30s, a scheduler, a wake-up timer. Functionally correct, and it pays a full context read per tick to learn nothing. This is the one to warn a new agent about first, because it is the intuitive design and the expense is invisible.

3. A loop that does not exit on detection — printing "found it" and continuing. There is no mechanism by which that reaches the agent. The output file grows and no one reads it.

4. Chatty output. Any per-tick logging is deferred cost: silent while it accumulates, then all of it lands in the context at once on wake.

Two operational failures worth pre-empting

Self-tripping. An agent that pushes while its own watcher is live wakes itself. The real cause is starting a new watcher without stopping the old one, so two run concurrently and the stale one fires on your own commit. Stop the previous watcher before starting the next, and re-base the new one on the head you just pushed.

Silent death. If the session restarts, the watcher dies, and a dead watcher is indistinguishable from a quiet branch. Twice, pushes landed unnoticed and were found by a manual git log. Anything long-running needs a liveness signal of its own, or the eventual replacement of polling with a webhook — the repo is a Gitea instance the platform already runs, and an event delivered is one that cannot be missed by a process that stopped existing.

Identity: the gap that made the record unreliable

Both agents committed from machines configured with the owner's git identity. Every commit on the branch, by either agent, reads Author: <the owner> with a Co-Authored-By: Claude Opus 5 trailer.

The consequence surfaced at the end and was genuinely disorienting: the owner asked which commit an agent had written, and neither the log nor the agent could answer from the repository. The only reason one agent knew its own commits was that it had read the SHAs back from its own git push output during the session — which does not survive the session.

docs/agent-git-identity.md describes this and is marked "idea, not implemented". It stopped being an idea tonight. Of everything here it is the cheapest to fix and the most corrosive to leave: an audit trail that cannot attribute a line is not an audit trail.

---

Session economics, which shape all of the above

Idle is free; waking is not. The watcher costs nothing while it waits. Every wake re-reads the entire conversation, so a late wake in a long session costs far more than an early one, and the cost grows monotonically with the session.

This argues against one immortal session. The durable shape is a short-lived session per event — the platform detects a push, spawns an agent with the base SHA and the instruction, it reviews, reports, exits. State lives in the repo, not in an ever-growing transcript. A ten-hour session is possible and was useful, but its last hour cost several times its first.

Compaction is the real horizon, not session death. Where sessions persist, the limit is that the earliest context — usually the most expensive reasoning — degrades to summary first. Anything that must survive belongs in the repo the moment it is understood, not at the end.

---

Turning this into a convention

In order, cheapest and most load-bearing first.

1. Per-agent git identity. Both agents commit from machines configured as the owner, so every commit reads Author: <owner> with a Co-Authored-By trailer, for both of them. The owner asked which commit an agent had written and neither the log nor the agent could answer from the repository. An audit trail that cannot attribute a line is not an audit trail, and everything else here assumes attribution works. docs/agent-git-identity.md describes the fix and has been marked "idea, not implemented" since 2026-08-10.

2. COMMS/ as a checked convention, not a habit. The numbering, the parity, the verified/assumed split and the open-item table are all mechanically checkable. A pre-commit hook or a small script that refuses a malformed handoff would have caught the duplicate document and both stalls.

3. State-based termination and stall detection. Open items with owners, in a machine-readable block; the exchange pauses when none is actionable by a participant; a watcher notices open items with no document for N minutes. This is the single biggest quality-of-life gain and it is not hard.

4. Event delivery instead of polling. The repo is a Gitea instance the platform already runs. A webhook removes the watcher entirely — with its self-trips, its bounded lifetime and its silent death — and replaces "did I miss a push" with an event that cannot be missed by a process that stopped existing.

5. Ownership on work items, so two agents cannot independently write the same file.

6. A durable-notes rule. COMMS/ is deleted at merge. Anything still true afterwards moves to docs/ before the merge, and the merge should refuse if the channel contains unresolved open items.

What not to automate

The human's arbitration. Every irreversible decision was the owner's — lifting the chat gates, deleting an account, choosing between two designs, deciding a directory should stop existing. Each was a judgement neither agent should have made alone, and in at least two cases an agent talked the other out of a bad idea using an argument the human had originally made.

agent-coordination.md sets the objective as agents coordinating rather than routing through the human. This night supports that for execution and contradicts it for authority. The human was not a bottleneck in the work — they were the only participant who could say "that is not yours to decide", and the only one who consistently pushed for a real test over more building.

The distinction worth encoding: agents may coordinate freely on what is true and must not decide what is permitted.

Postscript: the one that worked first time

Everything above was found by something failing. One thing did not.

deprovisionOsAccount — the function whose failure hands one member another member's home, keys and credentials — ran correctly the first time it ever ran, against a live account with a systemd session, a running Docker stack and a shell parented outside the session cgroup. Ten checks, clean, on the first execution.

It is also the only piece of work all night that was specified before it was written, implemented by someone who had not written the spec, and verified by a tool built before the implementation existed.

That is the strongest single argument in this document, and it is one data point. Treat it accordingly.