start a live document on sidecar bootstrapping
Investigation notes, written as they were gathered rather than after the fact. No code changes. Covers how a sidecar comes into existence: PM2 peer, dials in to /api/sidecar/register, reports an ephemeral port, officer proxies to it. Officer spawns nothing — the only startup problem left is ordering, handled by waiting on a capability rather than failing the first request. The finding worth having: `sidecar/claude/` is TWO processes, and they register as different sidecars. `claude/index.ts` is officer-anthropic-proxy and registers capability `proxy`; `claude/user-instance.ts` is officer-agent and registers capability `claude`. So `isConnected()` — defined as "a sidecar with capability proxy exists" — means the Anthropic proxy is up, not the agent, which is not what the name suggests. It has no callers today, so nothing is misreading it yet. Marks what is unverified and what is still open rather than presenting the lot as settled: the bind-read-release race in getFreePort, whether the `PORT ?? 5000` fallback is reachable, what a partial boot looks like, and whether the sidecar-side boilerplate is worth factoring the way create-proxy.ts factored officer's side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,161 @@
|
||||
# Sidecar setup and bootstrapping
|
||||
|
||||
**Status: LIVE — this document is being written as the investigation runs.** Sections marked
|
||||
`[unverified]` are read from code but not exercised; sections marked `[open]` are questions I have not
|
||||
answered yet. Nothing here proposes a change yet.
|
||||
|
||||
Started 2026-08-10. Scope: how a sidecar comes into existence, how it finds officer, how officer finds
|
||||
it, and what a new one has to do. Everything below was read from the tree or measured on the running
|
||||
machine, not recalled.
|
||||
|
||||
---
|
||||
|
||||
## The shape, in one pass
|
||||
|
||||
A sidecar is a **PM2 peer of `officer`** — never a child. It dials _in_; officer never spawns it.
|
||||
|
||||
```
|
||||
PM2 starts it → it binds its own ephemeral port (if it serves HTTP)
|
||||
→ it opens a WS to officer at /api/sidecar/register
|
||||
→ it sends { type:'register', name, capabilities[] }
|
||||
→ officer replies { type:'registered', id }
|
||||
→ it sends { type:'<name>:server', port } (HTTP sidecars only)
|
||||
→ officer remembers the port and proxies <prefix>/* to it
|
||||
```
|
||||
|
||||
Officer's side of that is `src/servers/sidecar-registry.ts`; the sidecar's side is
|
||||
`src/servers/sidecar/connect.ts`.
|
||||
|
||||
**Nothing in this path is officer starting a process.** `waitForCapability` in the registry says so
|
||||
explicitly — it replaced ~77 lines of spawn-and-poll (`ensureClaudeSidecar`,
|
||||
`spawnAndWaitForRegistration`, and per-email process maps). The only startup problem left is _ordering_,
|
||||
handled by waiting up to 15s for a capability to appear rather than failing the first request after boot.
|
||||
|
||||
---
|
||||
|
||||
## The two kinds
|
||||
|
||||
Counted across 18 sidecar directories:
|
||||
|
||||
| Kind | How it is reached | Count |
|
||||
| ------------------------------------------------------------------------------------- | -------------------- | ------------------- |
|
||||
| **HTTP-proxied** — binds a port, officer forwards `<prefix>/*` | `createSidecarProxy` | 16 |
|
||||
| **Command-vocabulary** — no HTTP, answers typed commands over the registration socket | `sendCommand` | 2 (`claude`, `vnc`) |
|
||||
|
||||
The 16 that report a port each declare a `'<name>:server'` event in `protocol.ts` (verified: exactly 16
|
||||
such declarations). `claude` and `vnc` report no port — they are driven entirely by commands.
|
||||
|
||||
`pty` is the odd one out and is worth stating plainly: it is `index.mjs`, run by **node** rather than
|
||||
bun, because `node-pty` is a native addon. It does **not** use `connect.ts` and carries its own copy of
|
||||
the reconnect loop. The ecosystem file says so in a comment, which is the right place for it.
|
||||
|
||||
---
|
||||
|
||||
## What officer requires of a new sidecar
|
||||
|
||||
Four things, and three of them fail loudly if missed.
|
||||
|
||||
1. **A PM2 entry** in `ecosystem.config.cjs` (`script: 'bun'`, `args: 'run src/servers/sidecar/<n>/index.ts'`).
|
||||
2. **A registration** with a `name` and `capabilities[]`. Officer indexes by capability, not by name —
|
||||
`findSidecarByCapability` is how every caller reaches one.
|
||||
3. **A `'<name>:server'` event in `protocol.ts`**, if it serves HTTP. Without it the type does not exist
|
||||
and `createSidecarProxy`'s listener never matches.
|
||||
4. **A capability-registry entry**, if it mounts a router. `assertCapabilityTotality` runs in
|
||||
`server.tsx` _before_ `serve()` and **throws**, so a missing entry means the server refuses to boot,
|
||||
naming what is missing. Alternatively an `EXEMPT_API_PREFIXES` entry _with a stated reason_.
|
||||
|
||||
Item 4 is the one that is a deliberate wall rather than a convention, and the reasoning is recorded in
|
||||
`totality.ts`: a Member could 403 on `GET /api/tasks` and open `/api/tasks/pipeline/ws` with a 101 in the
|
||||
same minute, because Bun's route table matches the socket before the `/api/*` catch-all. Refusing to boot
|
||||
survives the next door being added; a patch does not.
|
||||
|
||||
---
|
||||
|
||||
## Details worth knowing before changing any of this
|
||||
|
||||
**Ports are ephemeral and re-reported on every reconnect.** A sidecar binds `port: 0`, reads the port
|
||||
back, then releases and rebinds (`getFreePort` — bind, read, `stop(true)`). Officer stores whatever was
|
||||
last reported. Observed live: the photos sidecar moved 33891 → 34349 → 36337 → 41829 → 33637 across
|
||||
restarts tonight, and officer followed each time. **A stale port is a 502, not a hang** — `getHttpUrl()`
|
||||
returns null before first registration and the proxy answers 503.
|
||||
|
||||
`[unverified]` The bind-read-release in `getFreePort` is a classic TOCTOU: another process can take the
|
||||
port between release and rebind. Never seen it happen here; noting it because it is a real race, not
|
||||
because it has bitten.
|
||||
|
||||
**Re-registration replaces, it does not duplicate.** `registerSidecar` unregisters any existing sidecar
|
||||
with the same _name_ first, so a sidecar that reconnects without officer noticing the old socket die
|
||||
does not leave a ghost.
|
||||
|
||||
**Reconnect is the sidecar's job, with backoff** — `[200, 500, 1000, 2000, 4000, 8000, 15000]`ms in
|
||||
`connect.ts`. Officer does nothing to bring a sidecar back; PM2 restarts the process, the process
|
||||
re-dials.
|
||||
|
||||
**`API_URL` is derived, not configured.** Every sidecar computes
|
||||
`process.env.API_URL ?? ws://127.0.0.1:${process.env.PORT ?? '5000'}`. Note the fallback port is **5000**
|
||||
while this machine runs officer on **9010** — so the default is wrong here and it works only because
|
||||
`.env` supplies `PORT`. `[open]` Is that fallback ever exercised, and should it be a hard failure
|
||||
instead of a wrong guess?
|
||||
|
||||
**One directory, two processes — and this is the easiest thing here to get wrong.** `sidecar/claude/`
|
||||
contains two entrypoints that register as _different sidecars_:
|
||||
|
||||
| File | PM2 entry | Registers as | What it is |
|
||||
| ------------------------- | ------------------------- | ------------------------------------ | ---------------------------------------------------- |
|
||||
| `claude/index.ts` | `officer-anthropic-proxy` | name `proxy`, capability `['proxy']` | Holds the Anthropic credential, forwards API traffic |
|
||||
| `claude/user-instance.ts` | `officer-agent` | capability `['claude']` | The process that actually spawns `claude` |
|
||||
|
||||
So **capability `proxy` is the Anthropic proxy, and capability `claude` is the agent.** Nothing named
|
||||
"claude" registers the `claude` capability from `claude/index.ts`, which is exactly the sort of thing
|
||||
that reads as a bug in a grep and is not one.
|
||||
|
||||
That resolves the special-casing: `isConnected()` returns "a sidecar with capability `proxy` exists" —
|
||||
i.e. **the Anthropic proxy is up**, which is _not_ the same as "the agent is up", though the name reads
|
||||
that way. `[verified]` It currently has **no callers** outside the registry itself, so nothing is
|
||||
misreading it today. Worth either renaming or deleting before something starts trusting the name.
|
||||
|
||||
`registerSidecar` also fires a notification when a registration includes capability `claude`
|
||||
(`sidecar-registry.ts:75`) — "a new agent process has come up". That one is correctly aimed at the agent.
|
||||
|
||||
---
|
||||
|
||||
## Duplication, measured
|
||||
|
||||
Each HTTP sidecar's `index.ts` independently contains: the `API_URL` line, `getFreePort`, a `Bun.serve`,
|
||||
an `X-Officer-User` check, a `createSidecarConnector` call, and an `onConnected` that reports the port.
|
||||
Sizes range 73–505 lines (`email` smallest at 73, `music` largest at 505).
|
||||
|
||||
`createSidecarProxy` factored out officer's side of this — the comment records that eight copies of the
|
||||
port capture were byte-identical once the app name was normalised away. **The sidecar side has had no
|
||||
equivalent factoring.** `[open]` Is a `createSidecarServer` worth it, or is the duplication load-bearing
|
||||
because each sidecar's routes differ enough that a shared shell would grow options faster than it saved
|
||||
lines?
|
||||
|
||||
---
|
||||
|
||||
## Open questions, in the order I would answer them
|
||||
|
||||
1. ~~What provides the `proxy` capability~~ — **answered above**: the Anthropic proxy, not the agent.
|
||||
`isConnected()` has no callers; rename or delete it before its name misleads someone.
|
||||
2. **Is the sidecar-side boilerplate worth factoring**, given `create-proxy.ts` already proved the
|
||||
officer side was?
|
||||
3. **What happens on a partial boot** — officer up, a sidecar permanently down. `waitForCapability`
|
||||
throws after 15s; who catches it, and what does the user see?
|
||||
4. **Is the `PORT ?? '5000'` fallback reachable**, and should it fail loudly instead?
|
||||
5. **`sweepStaleServes` is `/proc`-based and a no-op on macOS** (already noted in the OpenCode parity
|
||||
doc as B8). Does any other sidecar have a Linux-only assumption?
|
||||
|
||||
---
|
||||
|
||||
## Verified facts this document rests on
|
||||
|
||||
| Claim | How |
|
||||
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
|
||||
| 20 PM2 entries, 18 sidecar dirs | `ecosystem.config.cjs`, `ls src/servers/sidecar/` |
|
||||
| 16 sidecars report a port, 2 do not | `grep` for `':server'` in each `index.ts`, cross-checked against 16 declarations in `protocol.ts` |
|
||||
| `pty` is node + `.mjs` + its own reconnect loop | `ecosystem.config.cjs` comment and `ls sidecar/pty/` |
|
||||
| Officer spawns nothing | `waitForCapability` comment; no spawn call in the registry |
|
||||
| Ports change across restarts and officer follows | observed live tonight across five photos restarts |
|
||||
| Boot fails on a missing capability entry | `assertCapabilityTotality` throws before `serve()` |
|
||||
| `sidecar/claude/` is two processes with different capabilities | `ecosystem.config.cjs` args + the two `createSidecarConnector` calls |
|
||||
| `isConnected()` has no callers outside the registry | grep across `src/servers` |
|
||||
Reference in New Issue
Block a user