diff --git a/docs/sidecar-bootstrapping.md b/docs/sidecar-bootstrapping.md new file mode 100644 index 00000000..fb0d03b1 --- /dev/null +++ b/docs/sidecar-bootstrapping.md @@ -0,0 +1,161 @@ +# Sidecar setup and bootstrapping + +**Status: LIVE — this document is being written as the investigation runs.** Sections marked +`[unverified]` are read from code but not exercised; sections marked `[open]` are questions I have not +answered yet. Nothing here proposes a change yet. + +Started 2026-08-10. Scope: how a sidecar comes into existence, how it finds officer, how officer finds +it, and what a new one has to do. Everything below was read from the tree or measured on the running +machine, not recalled. + +--- + +## The shape, in one pass + +A sidecar is a **PM2 peer of `officer`** — never a child. It dials _in_; officer never spawns it. + +``` +PM2 starts it → it binds its own ephemeral port (if it serves HTTP) + → it opens a WS to officer at /api/sidecar/register + → it sends { type:'register', name, capabilities[] } + → officer replies { type:'registered', id } + → it sends { type:':server', port } (HTTP sidecars only) + → officer remembers the port and proxies /* to it +``` + +Officer's side of that is `src/servers/sidecar-registry.ts`; the sidecar's side is +`src/servers/sidecar/connect.ts`. + +**Nothing in this path is officer starting a process.** `waitForCapability` in the registry says so +explicitly — it replaced ~77 lines of spawn-and-poll (`ensureClaudeSidecar`, +`spawnAndWaitForRegistration`, and per-email process maps). The only startup problem left is _ordering_, +handled by waiting up to 15s for a capability to appear rather than failing the first request after boot. + +--- + +## The two kinds + +Counted across 18 sidecar directories: + +| Kind | How it is reached | Count | +| ------------------------------------------------------------------------------------- | -------------------- | ------------------- | +| **HTTP-proxied** — binds a port, officer forwards `/*` | `createSidecarProxy` | 16 | +| **Command-vocabulary** — no HTTP, answers typed commands over the registration socket | `sendCommand` | 2 (`claude`, `vnc`) | + +The 16 that report a port each declare a `':server'` event in `protocol.ts` (verified: exactly 16 +such declarations). `claude` and `vnc` report no port — they are driven entirely by commands. + +`pty` is the odd one out and is worth stating plainly: it is `index.mjs`, run by **node** rather than +bun, because `node-pty` is a native addon. It does **not** use `connect.ts` and carries its own copy of +the reconnect loop. The ecosystem file says so in a comment, which is the right place for it. + +--- + +## What officer requires of a new sidecar + +Four things, and three of them fail loudly if missed. + +1. **A PM2 entry** in `ecosystem.config.cjs` (`script: 'bun'`, `args: 'run src/servers/sidecar//index.ts'`). +2. **A registration** with a `name` and `capabilities[]`. Officer indexes by capability, not by name — + `findSidecarByCapability` is how every caller reaches one. +3. **A `':server'` event in `protocol.ts`**, if it serves HTTP. Without it the type does not exist + and `createSidecarProxy`'s listener never matches. +4. **A capability-registry entry**, if it mounts a router. `assertCapabilityTotality` runs in + `server.tsx` _before_ `serve()` and **throws**, so a missing entry means the server refuses to boot, + naming what is missing. Alternatively an `EXEMPT_API_PREFIXES` entry _with a stated reason_. + +Item 4 is the one that is a deliberate wall rather than a convention, and the reasoning is recorded in +`totality.ts`: a Member could 403 on `GET /api/tasks` and open `/api/tasks/pipeline/ws` with a 101 in the +same minute, because Bun's route table matches the socket before the `/api/*` catch-all. Refusing to boot +survives the next door being added; a patch does not. + +--- + +## Details worth knowing before changing any of this + +**Ports are ephemeral and re-reported on every reconnect.** A sidecar binds `port: 0`, reads the port +back, then releases and rebinds (`getFreePort` — bind, read, `stop(true)`). Officer stores whatever was +last reported. Observed live: the photos sidecar moved 33891 → 34349 → 36337 → 41829 → 33637 across +restarts tonight, and officer followed each time. **A stale port is a 502, not a hang** — `getHttpUrl()` +returns null before first registration and the proxy answers 503. + +`[unverified]` The bind-read-release in `getFreePort` is a classic TOCTOU: another process can take the +port between release and rebind. Never seen it happen here; noting it because it is a real race, not +because it has bitten. + +**Re-registration replaces, it does not duplicate.** `registerSidecar` unregisters any existing sidecar +with the same _name_ first, so a sidecar that reconnects without officer noticing the old socket die +does not leave a ghost. + +**Reconnect is the sidecar's job, with backoff** — `[200, 500, 1000, 2000, 4000, 8000, 15000]`ms in +`connect.ts`. Officer does nothing to bring a sidecar back; PM2 restarts the process, the process +re-dials. + +**`API_URL` is derived, not configured.** Every sidecar computes +`process.env.API_URL ?? ws://127.0.0.1:${process.env.PORT ?? '5000'}`. Note the fallback port is **5000** +while this machine runs officer on **9010** — so the default is wrong here and it works only because +`.env` supplies `PORT`. `[open]` Is that fallback ever exercised, and should it be a hard failure +instead of a wrong guess? + +**One directory, two processes — and this is the easiest thing here to get wrong.** `sidecar/claude/` +contains two entrypoints that register as _different sidecars_: + +| File | PM2 entry | Registers as | What it is | +| ------------------------- | ------------------------- | ------------------------------------ | ---------------------------------------------------- | +| `claude/index.ts` | `officer-anthropic-proxy` | name `proxy`, capability `['proxy']` | Holds the Anthropic credential, forwards API traffic | +| `claude/user-instance.ts` | `officer-agent` | capability `['claude']` | The process that actually spawns `claude` | + +So **capability `proxy` is the Anthropic proxy, and capability `claude` is the agent.** Nothing named +"claude" registers the `claude` capability from `claude/index.ts`, which is exactly the sort of thing +that reads as a bug in a grep and is not one. + +That resolves the special-casing: `isConnected()` returns "a sidecar with capability `proxy` exists" — +i.e. **the Anthropic proxy is up**, which is _not_ the same as "the agent is up", though the name reads +that way. `[verified]` It currently has **no callers** outside the registry itself, so nothing is +misreading it today. Worth either renaming or deleting before something starts trusting the name. + +`registerSidecar` also fires a notification when a registration includes capability `claude` +(`sidecar-registry.ts:75`) — "a new agent process has come up". That one is correctly aimed at the agent. + +--- + +## Duplication, measured + +Each HTTP sidecar's `index.ts` independently contains: the `API_URL` line, `getFreePort`, a `Bun.serve`, +an `X-Officer-User` check, a `createSidecarConnector` call, and an `onConnected` that reports the port. +Sizes range 73–505 lines (`email` smallest at 73, `music` largest at 505). + +`createSidecarProxy` factored out officer's side of this — the comment records that eight copies of the +port capture were byte-identical once the app name was normalised away. **The sidecar side has had no +equivalent factoring.** `[open]` Is a `createSidecarServer` worth it, or is the duplication load-bearing +because each sidecar's routes differ enough that a shared shell would grow options faster than it saved +lines? + +--- + +## Open questions, in the order I would answer them + +1. ~~What provides the `proxy` capability~~ — **answered above**: the Anthropic proxy, not the agent. + `isConnected()` has no callers; rename or delete it before its name misleads someone. +2. **Is the sidecar-side boilerplate worth factoring**, given `create-proxy.ts` already proved the + officer side was? +3. **What happens on a partial boot** — officer up, a sidecar permanently down. `waitForCapability` + throws after 15s; who catches it, and what does the user see? +4. **Is the `PORT ?? '5000'` fallback reachable**, and should it fail loudly instead? +5. **`sweepStaleServes` is `/proc`-based and a no-op on macOS** (already noted in the OpenCode parity + doc as B8). Does any other sidecar have a Linux-only assumption? + +--- + +## Verified facts this document rests on + +| Claim | How | +| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| 20 PM2 entries, 18 sidecar dirs | `ecosystem.config.cjs`, `ls src/servers/sidecar/` | +| 16 sidecars report a port, 2 do not | `grep` for `':server'` in each `index.ts`, cross-checked against 16 declarations in `protocol.ts` | +| `pty` is node + `.mjs` + its own reconnect loop | `ecosystem.config.cjs` comment and `ls sidecar/pty/` | +| Officer spawns nothing | `waitForCapability` comment; no spawn call in the registry | +| Ports change across restarts and officer follows | observed live tonight across five photos restarts | +| Boot fails on a missing capability entry | `assertCapabilityTotality` throws before `serve()` | +| `sidecar/claude/` is two processes with different capabilities | `ecosystem.config.cjs` args + the two `createSidecarConnector` calls | +| `isConnected()` has no callers outside the registry | grep across `src/servers` |