# Process Sidecar

Independent Bun process that owns all long-running work so the API server can restart without disrupting active sessions.

## Problem

The API server (port 5000) previously owned all spawned processes: Pi agents, Claude Code sessions, the Anthropic proxy, and the job queue. Restarting the API server would:

- Kill active Pi and Claude Code conversations mid-response
- Regenerate the Anthropic proxy secret, breaking any Claude Code sessions using it
- Lose in-flight job progress (queue engine ran in-process)

## Solution

A separate Bun process ("process sidecar") on port 5100 that owns all spawned processes. The API server communicates with it over a single WebSocket connection. The sidecar is managed by pm2 and starts before the API server.

## Architecture

```
┌─────────────────────────┐         ┌──────────────────────────────┐
│   API Server (:5000)    │   WS    │    Process Sidecar (:5100)   │
│                         │◄───────►│                              │
│  - Browser WS clients   │         │  - Anthropic Proxy (:5051)   │
│  - REST API routes      │         │  - Claude Code processes     │
│  - Channel bots         │         │  - Pi agent processes        │
│  - sidecar-client.ts    │         │  - Job queue engine          │
│    (auto-reconnect)     │         │  - State persistence         │
└─────────────────────────┘         └──────────────────────────────┘
```

### What the sidecar owns

| Concern | Previous location | Sidecar module |
|---------|-------------------|----------------|
| Anthropic proxy (port 5051) | `api/anthropic-proxy.ts` | `sidecar/proxy.ts` |
| Claude Code spawn + session tracking | `channels/send-claude-code.ts` | `sidecar/claude-manager.ts` |
| Pi agent spawn + RPC commands | `api/pi/pi-bridge.ts` | `sidecar/pi-manager.ts` |
| Job queue engine + handlers | `queue/engine.ts` | `sidecar/queue-runner.ts` |

### What stays in the API server

- Browser WebSocket connections (ephemeral by nature)
- REST API routes (now thin proxies to sidecar)
- Channel bots (Discord/Telegram/WhatsApp — already reconnect gracefully)
- Browser relay (CDP state is ephemeral)
- PTY sidecar (already its own process, unchanged)

## Communication Protocol

Single WebSocket between API server and sidecar. JSON messages with `{ type, id?, ... }` envelopes.

**Request/response**: Commands include an `id` field. The sidecar responds with a message carrying the same `id`. The client correlates responses via this ID with configurable timeouts.

**Streaming events**: Pi and Claude Code output events are broadcast to all connected clients without a correlation ID. They carry a `sessionId` or `sessionKey` so the API server can route them to the correct browser WS.

### Command categories

```
ping / pong                         — health check
state:sync                          — full state snapshot on connect

proxy:secret                        — get persisted proxy secret

claude:spawn / claude:result        — blocking Claude Code exec
claude:spawn-streaming / claude:event — streaming Claude Code exec
claude:kill / claude:clear-session  — session management

pi:spawn / pi:prompt / pi:abort     — Pi agent lifecycle
pi:kill / pi:set-thinking           — Pi session control

queue:enqueue / queue:cancel        — job management
queue:list / queue:get              — job queries
```

See `protocol.ts` for the full type definitions.

## State Persistence

File: `data/sidecar/state.json`

Written every 30 seconds (debounced) and on graceful shutdown (SIGTERM/SIGINT). Contains:

- **proxySecret** — generated once on first boot, reused forever. This is the key fix: the Anthropic proxy secret no longer changes on restart.
- **claudeSessions** — map of `sessionKey → Claude Code session_id` for `--resume` support across restarts.
- **piSessions** — session metadata with PIDs for liveness checking on restart.

### Lockfile

`data/sidecar/sidecar.lock` — contains the PID of the running sidecar. On startup, if the lock exists and the PID is alive, the sidecar exits. Stale locks (dead PID) are cleaned up automatically.

## File Structure

```
src/servers/sidecar/
  index.ts           — entry point: Bun.serve on :5100, WS dispatch, shutdown
  protocol.ts        — shared message types (imported by both sides)
  proxy.ts           — Anthropic proxy server (moved from anthropic-proxy.ts)
  claude-manager.ts  — Claude Code blocking + streaming spawn, session map
  pi-manager.ts      — Pi agent spawn, RPC (prompt/abort/thinking), event parsing
  queue-runner.ts    — Job queue engine (lanes, retries, notifications)
  state.ts           — File-backed state persistence + lockfile

src/servers/sidecar-client.ts  — API server's WebSocket client (singleton)
```

## API Server Integration

The API server connects to the sidecar on startup via `initSidecarClient()` in `server.tsx`. The client auto-reconnects with exponential backoff (200ms → 15s).

On connect, it sends `state:sync` to get the current proxy secret and live session info.

### Modified API server files

| File | Change |
|------|--------|
| `server.tsx` | `startAnthropicProxy()` → `initSidecarClient()` |
| `bootstrap.ts` | Removed `initQueue()` (sidecar owns it) |
| `channels/send-claude-code.ts` | 550 lines → 73 lines thin client |
| `api/pi/websocket.ts` | Spawn/prompt/abort go through sidecar |
| `api/pi/session-manager.ts` | Cleanup sidecar subscriptions on delete |
| `api/queue/queue.ts` | Routes use sidecar client |
| `channels/discord/handler.ts` | `enqueue` → `enqueueJob` via sidecar |
| `channels/telegram/handler.ts` | Same |
| `channels/whatsapp/handler.ts` | Same |

## PM2 Configuration

```js
// ecosystem.config.cjs
module.exports = {
  apps: [
    {
      name: 'officer-sidecar',   // starts first
      script: 'bun',
      args: 'run src/servers/sidecar/index.ts',
      watch: false,
    },
    {
      name: 'officer',
      script: 'bun',
      args: 'start',
      watch: false,
    },
  ],
};
```

The sidecar is listed first so pm2 starts it before the API server. The API server's sidecar client handles the case where the sidecar isn't ready yet (auto-reconnect with backoff).

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `SIDECAR_PORT` | `5100` | Sidecar HTTP/WS port |
| `ANTHROPIC_PROXY_PORT` | `5051` | Anthropic proxy port (owned by sidecar) |
| `DATA_PATH` | `./data` | Shared data directory |

## Manual Testing

### 1. Start the sidecar standalone

```bash
# From monorepo root
bun run src/servers/sidecar/index.ts
```

Expected output:
```
[sidecar:proxy] listening on 127.0.0.1:5051
[sidecar:queue] initialized
[sidecar] listening on 127.0.0.1:5100
```

### 2. Health check

```bash
# Basic liveness
curl http://127.0.0.1:5100/
# → "process-sidecar"

# Detailed health
curl http://127.0.0.1:5100/health
# → {"status":"ok","uptime":12345,"piSessions":0,"claudeSessions":0}
```

### 3. Verify Anthropic proxy is running

```bash
# Should reject without valid secret
curl -s http://127.0.0.1:5051/v1/messages
# → {"error":"Unauthorized"}

# Check state file was created
cat data/sidecar/state.json
# → should show proxySecret, empty claudeSessions, empty piSessions
```

### 4. Verify proxy secret persistence

```bash
# Note the proxySecret from state.json
cat data/sidecar/state.json | jq .proxySecret

# Stop and restart the sidecar
# Kill the sidecar (Ctrl+C or kill)
bun run src/servers/sidecar/index.ts

# Check the secret is the same
cat data/sidecar/state.json | jq .proxySecret
# → should be identical to before
```

### 5. WebSocket communication test

```bash
# In one terminal, start the sidecar
bun run src/servers/sidecar/index.ts

# In another terminal, connect with websocat (or wscat)
# Install: cargo install websocat  OR  npm install -g wscat
websocat ws://127.0.0.1:5100

# Send a ping
{"type":"ping","id":"test1"}
# → should receive: {"type":"pong","id":"test1"}

# Send state sync
{"type":"state:sync","id":"test2"}
# → should receive: {"type":"state:sync","id":"test2","state":{...}}

# Get proxy secret
{"type":"proxy:secret","id":"test3"}
# → should receive: {"type":"proxy:secret","id":"test3","secret":"sk-ant-api03-..."}
```

### 6. Test lockfile protection

```bash
# Start sidecar in one terminal
bun run src/servers/sidecar/index.ts

# Try starting another in a second terminal
bun run src/servers/sidecar/index.ts
# → should exit with: "[sidecar] another instance is already running"

# Check lockfile
cat data/sidecar/sidecar.lock
# → PID of the running sidecar
```

### 7. Full integration test (sidecar + API server)

```bash
# Start sidecar first
bun run src/servers/sidecar/index.ts &

# Start API server
bun start
# → should see "[sidecar-client] connected" in logs

# Test via the dashboard:
# 1. Open a chat panel, send a message with claude-code model
#    → should see streaming response (routed through sidecar)
# 2. Open a chat with a Pi model
#    → should see streaming response (routed through sidecar)
# 3. Restart the API server (kill + bun start)
#    → active Claude Code processes should NOT die
#    → sidecar should show "client disconnected" then "client connected"
#    → proxy secret should remain the same
```

### 8. Test API server restart resilience

This is the key scenario that motivated the sidecar:

```bash
# 1. Start sidecar + API server
bun run src/servers/sidecar/index.ts &
bun start &

# 2. Start a Claude Code streaming session in the dashboard

# 3. While Claude Code is running, kill the API server
kill $(pgrep -f "bun start")

# 4. Restart the API server
bun start

# 5. Check:
#    - The Claude Code process should still be running (check with ps)
#    - The proxy secret should be the same (check data/sidecar/state.json)
#    - The sidecar should show the reconnection in logs
```

### 9. Test with pm2

```bash
pm2 start ecosystem.config.cjs

# Check both are running
pm2 status
# → officer-sidecar: online
# → officer: online

# Restart API server only
pm2 restart officer

# Check sidecar is still running
pm2 status
curl http://127.0.0.1:5100/health

# Stop everything
pm2 stop all
```

### 10. Queue test

```bash
# With sidecar running, queue a job via the API
curl -X POST http://localhost:5000/api/queue/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-token>" \
  -d '{"lane":"test","type":"gmail-sync"}'

# List jobs
curl http://localhost:5000/api/queue/jobs \
  -H "Authorization: Bearer <your-token>"
```

## Graceful Shutdown

On SIGTERM or SIGINT, the sidecar:
1. Flushes pending state to `data/sidecar/state.json`
2. Releases the lockfile
3. Does **NOT** kill spawned processes — they are independent OS processes

On restart, the sidecar reads the persisted state and checks which PIDs are still alive.

## Risks & Mitigations

| Risk | Mitigation |
|------|------------|
| Sidecar dies with live sessions | pm2 auto-restart + state.json + processes are independent OS processes |
| API↔sidecar WebSocket drops | Auto-reconnect with exponential backoff (200ms → 15s) |
| Two sidecar instances running | Lockfile with PID liveness check on startup |
| Sidecar needs DB access | Imports DB modules directly (same user, same filesystem) |
| Queue handlers need server context | Handlers are self-contained modules imported by the sidecar |

## Future Possibilities

The current architecture is one API server → one sidecar. But the protocol is sidecar-agnostic and could evolve in the opposite direction: one API server → many sidecars.

```
              ┌── sidecar-A (machine 1, user X's files)
API server ───┼── sidecar-B (machine 2, heavy compute)
              └── sidecar-C (localhost, default)
```

In this model the API server becomes the orchestrator — it decides which sidecar to dispatch to based on the user, the task type, or available capacity. A user's Claude Code session runs on the machine where their project files live. A long-running Pi agent gets dispatched to a box with more resources. Quick jobs stay local.

What would need to change:

- **Sidecar identity**: Each sidecar needs a name/id. The current hardcoded `ws://127.0.0.1:5100` becomes a registry of endpoints (config file or DB table).
- **Connection pool**: `sidecar-client.ts` becomes a pool of connections rather than a singleton.
- **Routing**: The API server needs a session→sidecar mapping so it knows where to forward browser messages.
- **The protocol itself**: Doesn't change. A `pi:spawn` command works the same whether it goes to a local or remote sidecar.