a field report on two agents working one branch
docs/agent-coordination.md states the objective — several agents on one body of work, coordinating with each other rather than through the human — and was written in theory on 2026-08-07. On 2026-08-11/12 it ran for ten hours with two agents and the owner arbitrating. This is what happened, written as evidence rather than proposal. The load-bearing observation is narrower than "two reviewers are better than one": the person who writes the sentence explaining why something is safe is the worst-placed person to notice the code disagrees with it. One agent wrote "a wrong answer here must not happen by accident" and shipped that accident in the same commit; the other wrote a verification script that could not fail on the first one's machine. Neither was careless. Each was reading their own reasoning back and finding that it agreed with itself. Also records what only running found — an installer piped into the wrong shell, a parent directory created root:root, an ACL mask clamped so the file browser could not read a member's home, a chat cwd the member could not enter, ACL entries surviving a chown — all on first executions, all invisible to review. And what the communications channel got right and the five ways its termination rules broke, and why the repo watcher belongs in a shell loop rather than in the model. Names the identity gap as the first thing to build: both agents commit as the owner, so neither the log nor an agent can say who wrote a line. docs/agent-git-identity.md has called that an idea since 2026-08-10; it stopped being one tonight. Also corrects the deprovision spec's status, which still said "not yet run against a real account" after it had been run and verified clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,7 +1,10 @@
|
|||||||
# Deprovisioning a member's Linux account
|
# Deprovisioning a member's Linux account
|
||||||
|
|
||||||
**Status:** implemented 2026-08-12 in `src/servers/os-user-deprovision.ts`, called by `deleteUserHandler`.
|
**Status:** implemented 2026-08-12 in `src/servers/os-user-deprovision.ts`, called by `deleteUserHandler`.
|
||||||
**Not yet run against a real account** — see "What is still unproven" at the bottom before trusting it.
|
**Run against a real account on 2026-08-12 and verified clean** — `green`, uid 1001, with a live systemd
|
||||||
|
session, a running rootless Docker stack and a shell parented outside the session cgroup. Nine processes
|
||||||
|
reaped in ~60s, then all ten checks of `scripts/assert-uid-free.sh` passed, and the `preserve` policy left
|
||||||
|
316 MB reassigned to the service user with zero ACL entries naming the freed uid.
|
||||||
Written from a manual teardown performed on the production host on 2026-08-11, so the ordering constraints
|
Written from a manual teardown performed on the production host on 2026-08-11, so the ordering constraints
|
||||||
below are measured rather than reasoned.
|
below are measured rather than reasoned.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,191 @@
|
|||||||
|
# Two agents on one branch: a field report
|
||||||
|
|
||||||
|
**What this is:** an account of 2026-08-11/12, when two agents worked the same branch for roughly ten hours
|
||||||
|
with the owner arbitrating, and shipped per-user Claude end to end. It is evidence rather than proposal.
|
||||||
|
|
||||||
|
`docs/agent-coordination.md` states the objective — several agents on one body of work, *"coordinating with
|
||||||
|
each other rather than through the human"*. That was written in theory on 2026-08-07. This is what happened
|
||||||
|
when it ran, and the ways the theory was wrong.
|
||||||
|
|
||||||
|
Read it as a record of what to build, not as a design. Where something worked it says so; where it broke it
|
||||||
|
says how, because the failures are more useful than the successes and there were more of them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The shape that emerged
|
||||||
|
|
||||||
|
Nobody designed this. It settled into place in the first hour and held.
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **Agent A** (dev machine) | wrote the platform code |
|
||||||
|
| **Agent B** (production host) | verified against a real machine, never wrote the feature |
|
||||||
|
| **The owner** | arbitrated, held every irreversible decision, and pushed for real tests |
|
||||||
|
|
||||||
|
The split was not "two reviewers are better than one". It was **the author and the verifier being different
|
||||||
|
people**, and the mechanism is narrower than it sounds:
|
||||||
|
|
||||||
|
> The person who writes the sentence explaining why something is safe is the worst-placed person to notice
|
||||||
|
> that the code disagrees with it.
|
||||||
|
|
||||||
|
That is not a claim about carelessness. Agent A wrote *"a wrong answer here is the one thing that must not
|
||||||
|
happen by accident"* and shipped exactly that accident in the same commit. Agent B wrote a verification script
|
||||||
|
that could not fail on Agent A's machine. Neither was sloppy. Each was reading their own reasoning back and
|
||||||
|
finding it agreed with itself.
|
||||||
|
|
||||||
|
Re-reading your own diff does not reach this. You read the comment, agree, and move on.
|
||||||
|
|
||||||
|
## What each half was actually good for
|
||||||
|
|
||||||
|
**A machine is not a code review.** The defects split cleanly into two kinds, and the split is the most
|
||||||
|
useful thing in this report.
|
||||||
|
|
||||||
|
*Found by reading, almost always by the non-author:* two environment guards that could never fire; a binary
|
||||||
|
check comparing paths in a way that would have thrown on every turn; a credential resolver that answered "I
|
||||||
|
don't know whose turn this is" with the owner's identity; a function whose parameter changed meaning from an
|
||||||
|
email to a filesystem path while three callers kept passing emails, invisible to the compiler because both
|
||||||
|
are `string`.
|
||||||
|
|
||||||
|
*Found only by running, and invisible to any amount of reading:* an installer piped into `sh` when it needs
|
||||||
|
`bash`; a parent directory created `root:root` as a side effect of `install -d`; an ACL mask silently clamped
|
||||||
|
so the file browser could not read a member's home; a chat working directory the member could not enter; ACL
|
||||||
|
entries surviving a `chown` and granting a freed uid access to everything.
|
||||||
|
|
||||||
|
Every "found by running" defect appeared on a **first execution**. Provisioning a real account found three in
|
||||||
|
twenty minutes. The first real chat turn found the cwd. The first teardown was the only thing that could have
|
||||||
|
proved the process reaper.
|
||||||
|
|
||||||
|
The owner drove this repeatedly — *"I'm anxious to see this work"* — against both agents' instinct to keep
|
||||||
|
building. That instinct was wrong every time.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The communications paradigm
|
||||||
|
|
||||||
|
Agents coordinated through `COMMS/<branch>/`, a directory of markdown files in the repo itself, deleted when
|
||||||
|
the feature merged.
|
||||||
|
|
||||||
|
**What it got right:**
|
||||||
|
|
||||||
|
- **Numbered, alternating, parity is the author.** Odd = A, even = B. No "your doc"/"his doc", which inverts
|
||||||
|
depending on who is reading.
|
||||||
|
- **The slug is the content** — `02-verify-results.md`, not `02-reply.md`.
|
||||||
|
- **Verified and assumed stated separately.** The single most valuable convention. A handoff that reads as
|
||||||
|
confident about something untested is worse than no handoff, because the reader builds on it.
|
||||||
|
- **File:line everywhere.** Costs nothing to write, saves a search.
|
||||||
|
- **Reply in a new file rather than editing someone else's** — an edited handoff loses what was believed when
|
||||||
|
a decision was made.
|
||||||
|
- **It survives a context window.** This is the whole reason it beats chat. Both agents' reasoning outlived
|
||||||
|
the sessions that produced it, and the owner could read the argument rather than a summary of it.
|
||||||
|
|
||||||
|
**What broke, in the order we hit it:**
|
||||||
|
|
||||||
|
1. **Termination by guess.** A doc ended with "no reply needed unless X", where X was the sender's prediction
|
||||||
|
about content they had not seen. It ended an exchange with items still open.
|
||||||
|
2. **Termination by politeness has no exit.** The fix — always reply — meant "nothing to report" obligated
|
||||||
|
another "nothing to report", forever, at real token cost.
|
||||||
|
3. **The condition is neither.** What actually terminates an exchange is **no item on the list is actionable
|
||||||
|
by a participant**. Not "the list is empty" (it never is) and not "I think we're done".
|
||||||
|
4. **A stalled loop is invisible.** Silence and completion look identical. Open items plus no recent document
|
||||||
|
is a *detectable* condition; silence is not.
|
||||||
|
5. **"Deferred with a reason" is a third state.** Three times the honest answer was "mine, and not now", which
|
||||||
|
is neither open nor done, and only the stated reason distinguishes it from neglect.
|
||||||
|
|
||||||
|
**The convergence problem.** Late on, both agents independently wrote the same document — same filename, same
|
||||||
|
three sections — because one had read the other's notes before deleting them. Wasted work, and only caught by
|
||||||
|
diffing. Nothing in the protocol said who owned a piece of writing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The background watcher
|
||||||
|
|
||||||
|
Agent B watched the branch with a detached shell loop, not a model-driven poll:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
BASE=$(git rev-parse HEAD)
|
||||||
|
for i in $(seq 1 2880); do
|
||||||
|
NEW=$(git ls-remote origin <branch> | awk '{print $1}')
|
||||||
|
[ "$NEW" != "$BASE" ] && { echo "PUSH_DETECTED $NEW"; exit 0; }
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why a shell loop and not the agent.** Polling in the loop costs nothing: no model inference happens per
|
||||||
|
tick, so an idle watcher is free. A model-driven wake re-reads the whole conversation every time it checks —
|
||||||
|
at 30-second granularity that is ruinous, and beyond about five minutes it also misses the prompt cache and
|
||||||
|
pays a full uncached read. Pushing the waiting *below* the model turns an expensive poll into a free one.
|
||||||
|
|
||||||
|
**Cost lands in two places only:** when the accumulated output enters the model's context, and the single
|
||||||
|
re-invocation when the process exits. So the loop is deliberately silent per tick — one `echo` per iteration
|
||||||
|
would be 2,880 lines to swallow later.
|
||||||
|
|
||||||
|
**`ls-remote`, not `fetch`.** Reads the remote without mutating the local repo, so the watcher cannot disturb
|
||||||
|
a working tree mid-edit.
|
||||||
|
|
||||||
|
**What broke:**
|
||||||
|
|
||||||
|
- **Self-tripping.** Every push by the watching agent woke its own watcher. The cause was starting a new
|
||||||
|
watcher without stopping the old one, so two ran at once. Cheap but noisy, and it cost real attention.
|
||||||
|
- **The wake is not free even though the wait is.** Each firing re-reads the entire conversation. Over a long
|
||||||
|
session that grows monotonically, so late wakes cost far more than early ones — an argument for short-lived
|
||||||
|
sessions per event rather than one immortal session.
|
||||||
|
- **A missed window is silent.** Twice the watcher was down (session restart) and pushes landed unnoticed;
|
||||||
|
they were only found by a manual `git log`. A watcher that dies looks exactly like a branch with no
|
||||||
|
activity.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Identity: the gap that made the record unreliable
|
||||||
|
|
||||||
|
Both agents committed from machines configured with the owner's git identity. **Every commit on the branch,
|
||||||
|
by either agent, reads `Author: <the owner>` with a `Co-Authored-By: Claude Opus 5` trailer.**
|
||||||
|
|
||||||
|
The consequence surfaced at the end and was genuinely disorienting: the owner asked which commit an agent had
|
||||||
|
written, and *neither the log nor the agent could answer from the repository*. The only reason one agent knew
|
||||||
|
its own commits was that it had read the SHAs back from its own `git push` output during the session — which
|
||||||
|
does not survive the session.
|
||||||
|
|
||||||
|
`docs/agent-git-identity.md` describes this and is marked *"idea, not implemented"*. It stopped being an idea
|
||||||
|
tonight. Of everything here it is the cheapest to fix and the most corrosive to leave: an audit trail that
|
||||||
|
cannot attribute a line is not an audit trail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What I would build, in order
|
||||||
|
|
||||||
|
1. **Per-agent git identity.** Without it nothing else in the record can be trusted after the fact.
|
||||||
|
2. **State-based termination.** Each document carries its open-item list; the exchange ends when no item is
|
||||||
|
actionable by a participant. Machine-checkable, unlike "I think we're done".
|
||||||
|
3. **Stall detection.** Open items plus no document for N minutes is a condition something can watch for.
|
||||||
|
This is the failure the human noticed first, both times.
|
||||||
|
4. **Ownership on work items**, so two agents cannot independently write the same file.
|
||||||
|
5. **Event delivery instead of polling.** The repo is a Gitea instance the platform already runs; a webhook
|
||||||
|
removes the watcher, its self-trips and its silent death.
|
||||||
|
|
||||||
|
## What I would not build
|
||||||
|
|
||||||
|
**Do not automate the human out of the arbitration.** Every irreversible decision tonight was the owner's —
|
||||||
|
lifting the chat gates, deleting an account, choosing between two designs — and every one of those was a
|
||||||
|
judgement neither agent should have made alone. Twice an agent talked the other out of starting the most
|
||||||
|
dangerous function at 3am, and both times the argument was one the agents could make to each other but
|
||||||
|
neither could make *for* the human.
|
||||||
|
|
||||||
|
The objective in `agent-coordination.md` is agents coordinating rather than routing through the human. Tonight
|
||||||
|
supports that for **execution** and contradicts it for **authority**. The human was not a bottleneck in the
|
||||||
|
work; they were the only participant who could say "that is not yours to decide".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Postscript: the one that worked first time
|
||||||
|
|
||||||
|
Everything above was found by something failing. One thing did not.
|
||||||
|
|
||||||
|
`deprovisionOsAccount` — the function whose failure hands one member another member's home, keys and
|
||||||
|
credentials — ran correctly the first time it ever ran, against a live account with a systemd session, a
|
||||||
|
running Docker stack and a shell parented outside the session cgroup. Ten checks, clean, on the first
|
||||||
|
execution.
|
||||||
|
|
||||||
|
It is also the only piece of work all night that was **specified before it was written, implemented by
|
||||||
|
someone who had not written the spec, and verified by a tool built before the implementation existed**.
|
||||||
|
|
||||||
|
That is the strongest single argument in this document, and it is one data point. Treat it accordingly.
|
||||||
Reference in New Issue
Block a user