platform/src/servers/sidecar/SIDECAR.md

Process Sidecar

Independent Bun process that owns all long-running work so the API server can restart without disrupting active sessions.

Problem

The API server (port 5000) previously owned all spawned processes: Pi agents, Claude Code sessions, the Anthropic proxy, and the job queue. Restarting the API server would:

• Kill active Pi and Claude Code conversations mid-response
• Regenerate the Anthropic proxy secret, breaking any Claude Code sessions using it
• Lose in-flight job progress (queue engine ran in-process)

Solution

A separate Bun process ("process sidecar") on port 5100 that owns all spawned processes. The API server communicates with it over a single WebSocket connection. The sidecar is managed by pm2 and starts before the API server.

Architecture

┌─────────────────────────┐         ┌──────────────────────────────┐
│   API Server (:5000)    │   WS    │    Process Sidecar (:5100)   │
│                         │◄───────►│                              │
│  - Browser WS clients   │         │  - Anthropic Proxy (:5051)   │
│  - REST API routes      │         │  - Claude Code processes     │
│  - Channel bots         │         │  - Pi agent processes        │
│  - sidecar-client.ts    │         │  - Job queue engine          │
│    (auto-reconnect)     │         │  - State persistence         │
└─────────────────────────┘         └──────────────────────────────┘

What the sidecar owns

Concern	Previous location	Sidecar module
Anthropic proxy (port 5051)	api/anthropic-proxy.ts	sidecar/proxy.ts
Claude Code spawn + session tracking	channels/send-claude-code.ts	sidecar/claude-manager.ts
Pi agent spawn + RPC commands	api/pi/pi-bridge.ts	sidecar/pi-manager.ts
Job queue engine + handlers	queue/engine.ts	sidecar/queue-runner.ts

What stays in the API server

• Browser WebSocket connections (ephemeral by nature)
• REST API routes (now thin proxies to sidecar)
• Channel bots (Discord/Telegram/WhatsApp — already reconnect gracefully)
• Browser relay (CDP state is ephemeral)
• PTY sidecar (already its own process, unchanged)

Communication Protocol

Single WebSocket between API server and sidecar. JSON messages with { type, id?, ... } envelopes.

Request/response: Commands include an id field. The sidecar responds with a message carrying the same id. The client correlates responses via this ID with configurable timeouts.

Streaming events: Pi and Claude Code output events are broadcast to all connected clients without a correlation ID. They carry a sessionId or sessionKey so the API server can route them to the correct browser WS.

Command categories

ping / pong                         — health check
state:sync                          — full state snapshot on connect

proxy:secret                        — get persisted proxy secret

claude:spawn / claude:result        — blocking Claude Code exec
claude:spawn-streaming / claude:event — streaming Claude Code exec
claude:kill / claude:clear-session  — session management

pi:spawn / pi:prompt / pi:abort     — Pi agent lifecycle
pi:kill / pi:set-thinking           — Pi session control

queue:enqueue / queue:cancel        — job management
queue:list / queue:get              — job queries

See protocol.ts for the full type definitions.

State Persistence

File: data/sidecar/state.json

Written every 30 seconds (debounced) and on graceful shutdown (SIGTERM/SIGINT). Contains:

• proxySecret — generated once on first boot, reused forever. This is the key fix: the Anthropic proxy secret no longer changes on restart.
• claudeSessions — map of sessionKey → Claude Code session_id for --resume support across restarts.
• piSessions — session metadata with PIDs for liveness checking on restart.

Lockfile

data/sidecar/sidecar.lock — contains the PID of the running sidecar. On startup, if the lock exists and the PID is alive, the sidecar exits. Stale locks (dead PID) are cleaned up automatically.

File Structure

src/servers/sidecar/
  index.ts           — entry point: Bun.serve on :5100, WS dispatch, shutdown
  protocol.ts        — shared message types (imported by both sides)
  proxy.ts           — Anthropic proxy server (moved from anthropic-proxy.ts)
  claude-manager.ts  — Claude Code blocking + streaming spawn, session map
  pi-manager.ts      — Pi agent spawn, RPC (prompt/abort/thinking), event parsing
  queue-runner.ts    — Job queue engine (lanes, retries, notifications)
  state.ts           — File-backed state persistence + lockfile

src/servers/sidecar-client.ts  — API server's WebSocket client (singleton)

API Server Integration

The API server connects to the sidecar on startup via initSidecarClient() in server.tsx. The client auto-reconnects with exponential backoff (200ms → 15s).

On connect, it sends state:sync to get the current proxy secret and live session info.

Modified API server files

File	Change
server.tsx	startAnthropicProxy() → initSidecarClient()
bootstrap.ts	Removed initQueue() (sidecar owns it)
channels/send-claude-code.ts	550 lines → 73 lines thin client
api/pi/websocket.ts	Spawn/prompt/abort go through sidecar
api/pi/session-manager.ts	Cleanup sidecar subscriptions on delete
api/queue/queue.ts	Routes use sidecar client
channels/discord/handler.ts	enqueue → enqueueJob via sidecar
channels/telegram/handler.ts	Same
channels/whatsapp/handler.ts	Same

PM2 Configuration

// ecosystem.config.cjs
module.exports = {
  apps: [
    {
      name: 'officer-sidecar',   // starts first
      script: 'bun',
      args: 'run src/servers/sidecar/index.ts',
      watch: false,
    },
    {
      name: 'officer',
      script: 'bun',
      args: 'start',
      watch: false,
    },
  ],
};

The sidecar is listed first so pm2 starts it before the API server. The API server's sidecar client handles the case where the sidecar isn't ready yet (auto-reconnect with backoff).

Environment Variables

Variable	Default	Description
SIDECAR_PORT	5100	Sidecar HTTP/WS port
ANTHROPIC_PROXY_PORT	5051	Anthropic proxy port (owned by sidecar)
DATA_PATH	./data	Shared data directory

Manual Testing

1. Start the sidecar standalone

# From monorepo root
bun run src/servers/sidecar/index.ts

Expected output:

[sidecar:proxy] listening on 127.0.0.1:5051
[sidecar:queue] initialized
[sidecar] listening on 127.0.0.1:5100

2. Health check

# Basic liveness
curl http://127.0.0.1:5100/
# → "process-sidecar"

# Detailed health
curl http://127.0.0.1:5100/health
# → {"status":"ok","uptime":12345,"piSessions":0,"claudeSessions":0}

3. Verify Anthropic proxy is running

# Should reject without valid secret
curl -s http://127.0.0.1:5051/v1/messages
# → {"error":"Unauthorized"}

# Check state file was created
cat data/sidecar/state.json
# → should show proxySecret, empty claudeSessions, empty piSessions

4. Verify proxy secret persistence

# Note the proxySecret from state.json
cat data/sidecar/state.json | jq .proxySecret

# Stop and restart the sidecar
# Kill the sidecar (Ctrl+C or kill)
bun run src/servers/sidecar/index.ts

# Check the secret is the same
cat data/sidecar/state.json | jq .proxySecret
# → should be identical to before

5. WebSocket communication test

# In one terminal, start the sidecar
bun run src/servers/sidecar/index.ts

# In another terminal, connect with websocat (or wscat)
# Install: cargo install websocat  OR  npm install -g wscat
websocat ws://127.0.0.1:5100

# Send a ping
{"type":"ping","id":"test1"}
# → should receive: {"type":"pong","id":"test1"}

# Send state sync
{"type":"state:sync","id":"test2"}
# → should receive: {"type":"state:sync","id":"test2","state":{...}}

# Get proxy secret
{"type":"proxy:secret","id":"test3"}
# → should receive: {"type":"proxy:secret","id":"test3","secret":"sk-ant-api03-..."}

6. Test lockfile protection

# Start sidecar in one terminal
bun run src/servers/sidecar/index.ts

# Try starting another in a second terminal
bun run src/servers/sidecar/index.ts
# → should exit with: "[sidecar] another instance is already running"

# Check lockfile
cat data/sidecar/sidecar.lock
# → PID of the running sidecar

7. Full integration test (sidecar + API server)

# Start sidecar first
bun run src/servers/sidecar/index.ts &

# Start API server
bun start
# → should see "[sidecar-client] connected" in logs

# Test via the dashboard:
# 1. Open a chat panel, send a message with claude-code model
#    → should see streaming response (routed through sidecar)
# 2. Open a chat with a Pi model
#    → should see streaming response (routed through sidecar)
# 3. Restart the API server (kill + bun start)
#    → active Claude Code processes should NOT die
#    → sidecar should show "client disconnected" then "client connected"
#    → proxy secret should remain the same

8. Test API server restart resilience

This is the key scenario that motivated the sidecar:

# 1. Start sidecar + API server
bun run src/servers/sidecar/index.ts &
bun start &

# 2. Start a Claude Code streaming session in the dashboard

# 3. While Claude Code is running, kill the API server
kill $(pgrep -f "bun start")

# 4. Restart the API server
bun start

# 5. Check:
#    - The Claude Code process should still be running (check with ps)
#    - The proxy secret should be the same (check data/sidecar/state.json)
#    - The sidecar should show the reconnection in logs

9. Test with pm2

pm2 start ecosystem.config.cjs

# Check both are running
pm2 status
# → officer-sidecar: online
# → officer: online

# Restart API server only
pm2 restart officer

# Check sidecar is still running
pm2 status
curl http://127.0.0.1:5100/health

# Stop everything
pm2 stop all

10. Queue test

# With sidecar running, queue a job via the API
curl -X POST http://localhost:5000/api/queue/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-token>" \
  -d '{"lane":"test","type":"gmail-sync"}'

# List jobs
curl http://localhost:5000/api/queue/jobs \
  -H "Authorization: Bearer <your-token>"

Graceful Shutdown

On SIGTERM or SIGINT, the sidecar:

1. Flushes pending state to data/sidecar/state.json
2. Releases the lockfile
3. Does NOT kill spawned processes — they are independent OS processes

On restart, the sidecar reads the persisted state and checks which PIDs are still alive.

Risks & Mitigations

Risk	Mitigation
Sidecar dies with live sessions	pm2 auto-restart + state.json + processes are independent OS processes
API↔sidecar WebSocket drops	Auto-reconnect with exponential backoff (200ms → 15s)
Two sidecar instances running	Lockfile with PID liveness check on startup
Sidecar needs DB access	Imports DB modules directly (same user, same filesystem)
Queue handlers need server context	Handlers are self-contained modules imported by the sidecar

Future Possibilities

The current architecture is one API server → one sidecar. But the protocol is sidecar-agnostic and could evolve in the opposite direction: one API server → many sidecars.

              ┌── sidecar-A (machine 1, user X's files)
API server ───┼── sidecar-B (machine 2, heavy compute)
              └── sidecar-C (localhost, default)

In this model the API server becomes the orchestrator — it decides which sidecar to dispatch to based on the user, the task type, or available capacity. A user's Claude Code session runs on the machine where their project files live. A long-running Pi agent gets dispatched to a box with more resources. Quick jobs stay local.

What would need to change:

• Sidecar identity: Each sidecar needs a name/id. The current hardcoded ws://127.0.0.1:5100 becomes a registry of endpoints (config file or DB table).
• Connection pool: sidecar-client.ts becomes a pool of connections rather than a singleton.
• Routing: The API server needs a session→sidecar mapping so it knows where to forward browser messages.
• The protocol itself: Doesn't change. A pi:spawn command works the same whether it goes to a local or remote sidecar.