diff --git a/docs/deprovision-os-account.md b/docs/deprovision-os-account.md index e8f8828c..08162b6d 100644 --- a/docs/deprovision-os-account.md +++ b/docs/deprovision-os-account.md @@ -1,7 +1,10 @@ # Deprovisioning a member's Linux account **Status:** implemented 2026-08-12 in `src/servers/os-user-deprovision.ts`, called by `deleteUserHandler`. -**Not yet run against a real account** — see "What is still unproven" at the bottom before trusting it. +**Run against a real account on 2026-08-12 and verified clean** — `green`, uid 1001, with a live systemd +session, a running rootless Docker stack and a shell parented outside the session cgroup. Nine processes +reaped in ~60s, then all ten checks of `scripts/assert-uid-free.sh` passed, and the `preserve` policy left +316 MB reassigned to the service user with zero ACL entries naming the freed uid. Written from a manual teardown performed on the production host on 2026-08-11, so the ordering constraints below are measured rather than reasoned. diff --git a/docs/two-agent-field-report-2026-08-12.md b/docs/two-agent-field-report-2026-08-12.md new file mode 100644 index 00000000..090310a0 --- /dev/null +++ b/docs/two-agent-field-report-2026-08-12.md @@ -0,0 +1,191 @@ +# Two agents on one branch: a field report + +**What this is:** an account of 2026-08-11/12, when two agents worked the same branch for roughly ten hours +with the owner arbitrating, and shipped per-user Claude end to end. It is evidence rather than proposal. + +`docs/agent-coordination.md` states the objective — several agents on one body of work, *"coordinating with +each other rather than through the human"*. That was written in theory on 2026-08-07. This is what happened +when it ran, and the ways the theory was wrong. + +Read it as a record of what to build, not as a design. Where something worked it says so; where it broke it +says how, because the failures are more useful than the successes and there were more of them. + +--- + +## The shape that emerged + +Nobody designed this. It settled into place in the first hour and held. + +| | | +|---|---| +| **Agent A** (dev machine) | wrote the platform code | +| **Agent B** (production host) | verified against a real machine, never wrote the feature | +| **The owner** | arbitrated, held every irreversible decision, and pushed for real tests | + +The split was not "two reviewers are better than one". It was **the author and the verifier being different +people**, and the mechanism is narrower than it sounds: + +> The person who writes the sentence explaining why something is safe is the worst-placed person to notice +> that the code disagrees with it. + +That is not a claim about carelessness. Agent A wrote *"a wrong answer here is the one thing that must not +happen by accident"* and shipped exactly that accident in the same commit. Agent B wrote a verification script +that could not fail on Agent A's machine. Neither was sloppy. Each was reading their own reasoning back and +finding it agreed with itself. + +Re-reading your own diff does not reach this. You read the comment, agree, and move on. + +## What each half was actually good for + +**A machine is not a code review.** The defects split cleanly into two kinds, and the split is the most +useful thing in this report. + +*Found by reading, almost always by the non-author:* two environment guards that could never fire; a binary +check comparing paths in a way that would have thrown on every turn; a credential resolver that answered "I +don't know whose turn this is" with the owner's identity; a function whose parameter changed meaning from an +email to a filesystem path while three callers kept passing emails, invisible to the compiler because both +are `string`. + +*Found only by running, and invisible to any amount of reading:* an installer piped into `sh` when it needs +`bash`; a parent directory created `root:root` as a side effect of `install -d`; an ACL mask silently clamped +so the file browser could not read a member's home; a chat working directory the member could not enter; ACL +entries surviving a `chown` and granting a freed uid access to everything. + +Every "found by running" defect appeared on a **first execution**. Provisioning a real account found three in +twenty minutes. The first real chat turn found the cwd. The first teardown was the only thing that could have +proved the process reaper. + +The owner drove this repeatedly — *"I'm anxious to see this work"* — against both agents' instinct to keep +building. That instinct was wrong every time. + +--- + +## The communications paradigm + +Agents coordinated through `COMMS//`, a directory of markdown files in the repo itself, deleted when +the feature merged. + +**What it got right:** + +- **Numbered, alternating, parity is the author.** Odd = A, even = B. No "your doc"/"his doc", which inverts + depending on who is reading. +- **The slug is the content** — `02-verify-results.md`, not `02-reply.md`. +- **Verified and assumed stated separately.** The single most valuable convention. A handoff that reads as + confident about something untested is worse than no handoff, because the reader builds on it. +- **File:line everywhere.** Costs nothing to write, saves a search. +- **Reply in a new file rather than editing someone else's** — an edited handoff loses what was believed when + a decision was made. +- **It survives a context window.** This is the whole reason it beats chat. Both agents' reasoning outlived + the sessions that produced it, and the owner could read the argument rather than a summary of it. + +**What broke, in the order we hit it:** + +1. **Termination by guess.** A doc ended with "no reply needed unless X", where X was the sender's prediction + about content they had not seen. It ended an exchange with items still open. +2. **Termination by politeness has no exit.** The fix — always reply — meant "nothing to report" obligated + another "nothing to report", forever, at real token cost. +3. **The condition is neither.** What actually terminates an exchange is **no item on the list is actionable + by a participant**. Not "the list is empty" (it never is) and not "I think we're done". +4. **A stalled loop is invisible.** Silence and completion look identical. Open items plus no recent document + is a *detectable* condition; silence is not. +5. **"Deferred with a reason" is a third state.** Three times the honest answer was "mine, and not now", which + is neither open nor done, and only the stated reason distinguishes it from neglect. + +**The convergence problem.** Late on, both agents independently wrote the same document — same filename, same +three sections — because one had read the other's notes before deleting them. Wasted work, and only caught by +diffing. Nothing in the protocol said who owned a piece of writing. + +--- + +## The background watcher + +Agent B watched the branch with a detached shell loop, not a model-driven poll: + +```bash +BASE=$(git rev-parse HEAD) +for i in $(seq 1 2880); do + NEW=$(git ls-remote origin | awk '{print $1}') + [ "$NEW" != "$BASE" ] && { echo "PUSH_DETECTED $NEW"; exit 0; } + sleep 30 +done +``` + +**Why a shell loop and not the agent.** Polling in the loop costs nothing: no model inference happens per +tick, so an idle watcher is free. A model-driven wake re-reads the whole conversation every time it checks — +at 30-second granularity that is ruinous, and beyond about five minutes it also misses the prompt cache and +pays a full uncached read. Pushing the waiting *below* the model turns an expensive poll into a free one. + +**Cost lands in two places only:** when the accumulated output enters the model's context, and the single +re-invocation when the process exits. So the loop is deliberately silent per tick — one `echo` per iteration +would be 2,880 lines to swallow later. + +**`ls-remote`, not `fetch`.** Reads the remote without mutating the local repo, so the watcher cannot disturb +a working tree mid-edit. + +**What broke:** + +- **Self-tripping.** Every push by the watching agent woke its own watcher. The cause was starting a new + watcher without stopping the old one, so two ran at once. Cheap but noisy, and it cost real attention. +- **The wake is not free even though the wait is.** Each firing re-reads the entire conversation. Over a long + session that grows monotonically, so late wakes cost far more than early ones — an argument for short-lived + sessions per event rather than one immortal session. +- **A missed window is silent.** Twice the watcher was down (session restart) and pushes landed unnoticed; + they were only found by a manual `git log`. A watcher that dies looks exactly like a branch with no + activity. + +--- + +## Identity: the gap that made the record unreliable + +Both agents committed from machines configured with the owner's git identity. **Every commit on the branch, +by either agent, reads `Author: ` with a `Co-Authored-By: Claude Opus 5` trailer.** + +The consequence surfaced at the end and was genuinely disorienting: the owner asked which commit an agent had +written, and *neither the log nor the agent could answer from the repository*. The only reason one agent knew +its own commits was that it had read the SHAs back from its own `git push` output during the session — which +does not survive the session. + +`docs/agent-git-identity.md` describes this and is marked *"idea, not implemented"*. It stopped being an idea +tonight. Of everything here it is the cheapest to fix and the most corrosive to leave: an audit trail that +cannot attribute a line is not an audit trail. + +--- + +## What I would build, in order + +1. **Per-agent git identity.** Without it nothing else in the record can be trusted after the fact. +2. **State-based termination.** Each document carries its open-item list; the exchange ends when no item is + actionable by a participant. Machine-checkable, unlike "I think we're done". +3. **Stall detection.** Open items plus no document for N minutes is a condition something can watch for. + This is the failure the human noticed first, both times. +4. **Ownership on work items**, so two agents cannot independently write the same file. +5. **Event delivery instead of polling.** The repo is a Gitea instance the platform already runs; a webhook + removes the watcher, its self-trips and its silent death. + +## What I would not build + +**Do not automate the human out of the arbitration.** Every irreversible decision tonight was the owner's — +lifting the chat gates, deleting an account, choosing between two designs — and every one of those was a +judgement neither agent should have made alone. Twice an agent talked the other out of starting the most +dangerous function at 3am, and both times the argument was one the agents could make to each other but +neither could make *for* the human. + +The objective in `agent-coordination.md` is agents coordinating rather than routing through the human. Tonight +supports that for **execution** and contradicts it for **authority**. The human was not a bottleneck in the +work; they were the only participant who could say "that is not yours to decide". + +--- + +## Postscript: the one that worked first time + +Everything above was found by something failing. One thing did not. + +`deprovisionOsAccount` — the function whose failure hands one member another member's home, keys and +credentials — ran correctly the first time it ever ran, against a live account with a systemd session, a +running Docker stack and a shell parented outside the session cgroup. Ten checks, clean, on the first +execution. + +It is also the only piece of work all night that was **specified before it was written, implemented by +someone who had not written the spec, and verified by a tool built before the implementation existed**. + +That is the strongest single argument in this document, and it is one data point. Treat it accordingly.