# Sidecar setup and bootstrapping **Status: LIVE — this document is being written as the investigation runs.** Sections marked `[unverified]` are read from code but not exercised; sections marked `[open]` are questions I have not answered yet. Nothing here proposes a change yet. Started 2026-08-10. Scope: how a sidecar comes into existence, how it finds officer, how officer finds it, and what a new one has to do. Everything below was read from the tree or measured on the running machine, not recalled. --- ## The shape, in one pass A sidecar is a **PM2 peer of `officer`** — never a child. It dials _in_; officer never spawns it. ``` PM2 starts it → it binds its own ephemeral port (if it serves HTTP) → it opens a WS to officer at /api/sidecar/register → it sends { type:'register', name, capabilities[] } → officer replies { type:'registered', id } → it sends { type:':server', port } (HTTP sidecars only) → officer remembers the port and proxies /* to it ``` Officer's side of that is `src/servers/sidecar-registry.ts`; the sidecar's side is `src/servers/sidecar/connect.ts`. **Nothing in this path is officer starting a process.** `waitForCapability` in the registry says so explicitly — it replaced ~77 lines of spawn-and-poll (`ensureClaudeSidecar`, `spawnAndWaitForRegistration`, and per-email process maps). The only startup problem left is _ordering_, handled by waiting up to 15s for a capability to appear rather than failing the first request after boot. --- ## The two kinds Counted across 18 sidecar directories: | Kind | How it is reached | Count | | ------------------------------------------------------------------------------------- | -------------------- | ------------------- | | **HTTP-proxied** — binds a port, officer forwards `/*` | `createSidecarProxy` | 16 | | **Command-vocabulary** — no HTTP, answers typed commands over the registration socket | `sendCommand` | 2 (`claude`, `vnc`) | The 16 that report a port each declare a `':server'` event in `protocol.ts` (verified: exactly 16 such declarations). `claude` and `vnc` report no port — they are driven entirely by commands. `pty` is the odd one out and is worth stating plainly: it is `index.mjs`, run by **node** rather than bun, because `node-pty` is a native addon. It does **not** use `connect.ts` and carries its own copy of the reconnect loop. The ecosystem file says so in a comment, which is the right place for it. --- ## What officer requires of a new sidecar Four things, and three of them fail loudly if missed. 1. **A PM2 entry** in `ecosystem.config.cjs` (`script: 'bun'`, `args: 'run src/servers/sidecar//index.ts'`). 2. **A registration** with a `name` and `capabilities[]`. Officer indexes by capability, not by name — `findSidecarByCapability` is how every caller reaches one. 3. **A `':server'` event in `protocol.ts`**, if it serves HTTP. Without it the type does not exist and `createSidecarProxy`'s listener never matches. 4. **A capability-registry entry**, if it mounts a router. `assertCapabilityTotality` runs in `server.tsx` _before_ `serve()` and **throws**, so a missing entry means the server refuses to boot, naming what is missing. Alternatively an `EXEMPT_API_PREFIXES` entry _with a stated reason_. Item 4 is the one that is a deliberate wall rather than a convention, and the reasoning is recorded in `totality.ts`: a Member could 403 on `GET /api/tasks` and open `/api/tasks/pipeline/ws` with a 101 in the same minute, because Bun's route table matches the socket before the `/api/*` catch-all. Refusing to boot survives the next door being added; a patch does not. --- ## Details worth knowing before changing any of this **Ports are ephemeral and re-reported on every reconnect.** A sidecar binds `port: 0`, reads the port back, then releases and rebinds (`getFreePort` — bind, read, `stop(true)`). Officer stores whatever was last reported. Observed live: the photos sidecar moved 33891 → 34349 → 36337 → 41829 → 33637 across restarts tonight, and officer followed each time. **A stale port is a 502, not a hang** — `getHttpUrl()` returns null before first registration and the proxy answers 503. `[unverified]` The bind-read-release in `getFreePort` is a classic TOCTOU: another process can take the port between release and rebind. Never seen it happen here; noting it because it is a real race, not because it has bitten. **Re-registration replaces, it does not duplicate.** `registerSidecar` unregisters any existing sidecar with the same _name_ first, so a sidecar that reconnects without officer noticing the old socket die does not leave a ghost. **Reconnect is the sidecar's job, with backoff** — `[200, 500, 1000, 2000, 4000, 8000, 15000]`ms in `connect.ts`. Officer does nothing to bring a sidecar back; PM2 restarts the process, the process re-dials. **`API_URL` is derived, not configured.** Every sidecar computes `process.env.API_URL ?? ws://127.0.0.1:${process.env.PORT ?? '5000'}`. Note the fallback port is **5000** while this machine runs officer on **9010** — so the default is wrong here and it works only because `.env` supplies `PORT`. `[open]` Is that fallback ever exercised, and should it be a hard failure instead of a wrong guess? **One directory, two processes — and this is the easiest thing here to get wrong.** `sidecar/claude/` contains two entrypoints that register as _different sidecars_: | File | PM2 entry | Registers as | What it is | | ------------------------- | ------------------------- | ------------------------------------ | ---------------------------------------------------- | | `claude/index.ts` | `officer-anthropic-proxy` | name `proxy`, capability `['proxy']` | Holds the Anthropic credential, forwards API traffic | | `claude/user-instance.ts` | `officer-agent` | capability `['claude']` | The process that actually spawns `claude` | So **capability `proxy` is the Anthropic proxy, and capability `claude` is the agent.** Nothing named "claude" registers the `claude` capability from `claude/index.ts`, which is exactly the sort of thing that reads as a bug in a grep and is not one. That resolves the special-casing: `isConnected()` returns "a sidecar with capability `proxy` exists" — i.e. **the Anthropic proxy is up**, which is _not_ the same as "the agent is up", though the name reads that way. `[verified]` It currently has **no callers** outside the registry itself, so nothing is misreading it today. Worth either renaming or deleting before something starts trusting the name. `registerSidecar` also fires a notification when a registration includes capability `claude` (`sidecar-registry.ts:75`) — "a new agent process has come up". That one is correctly aimed at the agent. --- ## Duplication, measured Each HTTP sidecar's `index.ts` independently contains: the `API_URL` line, `getFreePort`, a `Bun.serve`, an `X-Officer-User` check, a `createSidecarConnector` call, and an `onConnected` that reports the port. Sizes range 73–505 lines (`email` smallest at 73, `music` largest at 505). `createSidecarProxy` factored out officer's side of this — the comment records that eight copies of the port capture were byte-identical once the app name was normalised away. **The sidecar side has had no equivalent factoring.** `[open]` Is a `createSidecarServer` worth it, or is the duplication load-bearing because each sidecar's routes differ enough that a shared shell would grow options faster than it saved lines? --- ## Open questions, in the order I would answer them 1. ~~What provides the `proxy` capability~~ — **answered above**: the Anthropic proxy, not the agent. `isConnected()` has no callers; rename or delete it before its name misleads someone. 2. **Is the sidecar-side boilerplate worth factoring**, given `create-proxy.ts` already proved the officer side was? 3. **What happens on a partial boot** — officer up, a sidecar permanently down. `waitForCapability` throws after 15s; who catches it, and what does the user see? 4. **Is the `PORT ?? '5000'` fallback reachable**, and should it fail loudly instead? 5. **`sweepStaleServes` is `/proc`-based and a no-op on macOS** (already noted in the OpenCode parity doc as B8). Does any other sidecar have a Linux-only assumption? --- ## Verified facts this document rests on | Claim | How | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | 20 PM2 entries, 18 sidecar dirs | `ecosystem.config.cjs`, `ls src/servers/sidecar/` | | 16 sidecars report a port, 2 do not | `grep` for `':server'` in each `index.ts`, cross-checked against 16 declarations in `protocol.ts` | | `pty` is node + `.mjs` + its own reconnect loop | `ecosystem.config.cjs` comment and `ls sidecar/pty/` | | Officer spawns nothing | `waitForCapability` comment; no spawn call in the registry | | Ports change across restarts and officer follows | observed live tonight across five photos restarts | | Boot fails on a missing capability entry | `assertCapabilityTotality` throws before `serve()` | | `sidecar/claude/` is two processes with different capabilities | `ecosystem.config.cjs` args + the two `createSidecarConnector` calls | | `isConnected()` has no callers outside the registry | grep across `src/servers` |