officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

process sidecar: independent process manager for long-running work

Browse Source 
Introduces a separate Bun process (port 5100) that owns all spawned
processes and long-running work, so the API server can restart freely
without disrupting active sessions.

The sidecar owns:
- Anthropic proxy (port 5051) with persisted secret across restarts
- Claude Code process spawning and session tracking (--resume support)
- Pi agent spawning and RPC lifecycle (prompt/abort/thinking)
- Job queue engine (lane processing, retries, notifications)

The API server becomes a thin client that forwards commands over a
single WebSocket connection with auto-reconnect. send-claude-code.ts
goes from 550 lines of spawn logic to 73 lines of sidecar delegation.

State persisted to data/sidecar/state.json every 30s and on shutdown.
Lockfile prevents duplicate instances. See SIDECAR.md for full docs
and manual testing procedures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-03-05 07:36:47 +00:00
co-authored by Claude Opus 4.6
parent 726c77e346
commit 86c2d6333a
21 changed files with 2415 additions and 610 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/servers/sidecar/SIDECAR.md
+347
View File
@@ -0,0 +1,347 @@
# Process Sidecar
Independent Bun process that owns all long-running work so the API server can restart without disrupting active sessions.
## Problem
The API server (port 5000) previously owned all spawned processes: Pi agents, Claude Code sessions, the Anthropic proxy, and the job queue. Restarting the API server would:
- Kill active Pi and Claude Code conversations mid-response
- Regenerate the Anthropic proxy secret, breaking any Claude Code sessions using it
- Lose in-flight job progress (queue engine ran in-process)
## Solution
A separate Bun process ("process sidecar") on port 5100 that owns all spawned processes. The API server communicates with it over a single WebSocket connection. The sidecar is managed by pm2 and starts before the API server.
## Architecture
```
┌─────────────────────────┐ ┌──────────────────────────────┐
│ API Server (:5000) │ WS │ Process Sidecar (:5100) │
│ │◄───────►│ │
│ - Browser WS clients │ │ - Anthropic Proxy (:5051) │
│ - REST API routes │ │ - Claude Code processes │
│ - Channel bots │ │ - Pi agent processes │
│ - sidecar-client.ts │ │ - Job queue engine │
│ (auto-reconnect) │ │ - State persistence │
└─────────────────────────┘ └──────────────────────────────┘
```
### What the sidecar owns
| Concern | Previous location | Sidecar module |
|---------|-------------------|----------------|
| Anthropic proxy (port 5051) | `api/anthropic-proxy.ts` | `sidecar/proxy.ts` |
| Claude Code spawn + session tracking | `channels/send-claude-code.ts` | `sidecar/claude-manager.ts` |
| Pi agent spawn + RPC commands | `api/pi/pi-bridge.ts` | `sidecar/pi-manager.ts` |
| Job queue engine + handlers | `queue/engine.ts` | `sidecar/queue-runner.ts` |
### What stays in the API server
- Browser WebSocket connections (ephemeral by nature)
- REST API routes (now thin proxies to sidecar)
- Channel bots (Discord/Telegram/WhatsApp — already reconnect gracefully)
- Browser relay (CDP state is ephemeral)
- PTY sidecar (already its own process, unchanged)
## Communication Protocol
Single WebSocket between API server and sidecar. JSON messages with `{ type, id?, ... }` envelopes.
**Request/response**: Commands include an `id` field. The sidecar responds with a message carrying the same `id`. The client correlates responses via this ID with configurable timeouts.
**Streaming events**: Pi and Claude Code output events are broadcast to all connected clients without a correlation ID. They carry a `sessionId` or `sessionKey` so the API server can route them to the correct browser WS.
### Command categories
```
ping / pong — health check
state:sync — full state snapshot on connect
proxy:secret — get persisted proxy secret
claude:spawn / claude:result — blocking Claude Code exec
claude:spawn-streaming / claude:event — streaming Claude Code exec
claude:kill / claude:clear-session — session management
pi:spawn / pi:prompt / pi:abort — Pi agent lifecycle
pi:kill / pi:set-thinking — Pi session control
queue:enqueue / queue:cancel — job management
queue:list / queue:get — job queries
```
See `protocol.ts` for the full type definitions.
## State Persistence
File: `data/sidecar/state.json`
Written every 30 seconds (debounced) and on graceful shutdown (SIGTERM/SIGINT). Contains:
- **proxySecret** — generated once on first boot, reused forever. This is the key fix: the Anthropic proxy secret no longer changes on restart.
- **claudeSessions** — map of `sessionKey → Claude Code session_id` for `--resume` support across restarts.
- **piSessions** — session metadata with PIDs for liveness checking on restart.
### Lockfile
`data/sidecar/sidecar.lock` — contains the PID of the running sidecar. On startup, if the lock exists and the PID is alive, the sidecar exits. Stale locks (dead PID) are cleaned up automatically.
## File Structure
```
src/servers/sidecar/
index.ts — entry point: Bun.serve on :5100, WS dispatch, shutdown
protocol.ts — shared message types (imported by both sides)
proxy.ts — Anthropic proxy server (moved from anthropic-proxy.ts)
claude-manager.ts — Claude Code blocking + streaming spawn, session map
pi-manager.ts — Pi agent spawn, RPC (prompt/abort/thinking), event parsing
queue-runner.ts — Job queue engine (lanes, retries, notifications)
state.ts — File-backed state persistence + lockfile
src/servers/sidecar-client.ts — API server's WebSocket client (singleton)
```
## API Server Integration
The API server connects to the sidecar on startup via `initSidecarClient()` in `server.tsx`. The client auto-reconnects with exponential backoff (200ms → 15s).
On connect, it sends `state:sync` to get the current proxy secret and live session info.
### Modified API server files
| File | Change |
|------|--------|
| `server.tsx` | `startAnthropicProxy()` → `initSidecarClient()` |
| `bootstrap.ts` | Removed `initQueue()` (sidecar owns it) |
| `channels/send-claude-code.ts` | 550 lines → 73 lines thin client |
| `api/pi/websocket.ts` | Spawn/prompt/abort go through sidecar |
| `api/pi/session-manager.ts` | Cleanup sidecar subscriptions on delete |
| `api/queue/queue.ts` | Routes use sidecar client |
| `channels/discord/handler.ts` | `enqueue` → `enqueueJob` via sidecar |
| `channels/telegram/handler.ts` | Same |
| `channels/whatsapp/handler.ts` | Same |
## PM2 Configuration
```js
// ecosystem.config.cjs
module.exports = {
apps: [
{
name: 'officer-sidecar', // starts first
script: 'bun',
args: 'run src/servers/sidecar/index.ts',
watch: false,
},
{
name: 'officer',
script: 'bun',
args: 'start',
watch: false,
},
],
};
```
The sidecar is listed first so pm2 starts it before the API server. The API server's sidecar client handles the case where the sidecar isn't ready yet (auto-reconnect with backoff).
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `SIDECAR_PORT` | `5100` | Sidecar HTTP/WS port |
| `ANTHROPIC_PROXY_PORT` | `5051` | Anthropic proxy port (owned by sidecar) |
| `DATA_PATH` | `./data` | Shared data directory |
## Manual Testing
### 1. Start the sidecar standalone
```bash
# From monorepo root
bun run src/servers/sidecar/index.ts
```
Expected output:
```
[sidecar:proxy] listening on 127.0.0.1:5051
[sidecar:queue] initialized
[sidecar] listening on 127.0.0.1:5100
```
### 2. Health check
```bash
# Basic liveness
curl http://127.0.0.1:5100/
# → "process-sidecar"
# Detailed health
curl http://127.0.0.1:5100/health
# → {"status":"ok","uptime":12345,"piSessions":0,"claudeSessions":0}
```
### 3. Verify Anthropic proxy is running
```bash
# Should reject without valid secret
curl -s http://127.0.0.1:5051/v1/messages
# → {"error":"Unauthorized"}
# Check state file was created
cat data/sidecar/state.json
# → should show proxySecret, empty claudeSessions, empty piSessions
```
### 4. Verify proxy secret persistence
```bash
# Note the proxySecret from state.json
cat data/sidecar/state.json | jq .proxySecret
# Stop and restart the sidecar
# Kill the sidecar (Ctrl+C or kill)
bun run src/servers/sidecar/index.ts
# Check the secret is the same
cat data/sidecar/state.json | jq .proxySecret
# → should be identical to before
```
### 5. WebSocket communication test
```bash
# In one terminal, start the sidecar
bun run src/servers/sidecar/index.ts
# In another terminal, connect with websocat (or wscat)
# Install: cargo install websocat OR npm install -g wscat
websocat ws://127.0.0.1:5100
# Send a ping
{"type":"ping","id":"test1"}
# → should receive: {"type":"pong","id":"test1"}
# Send state sync
{"type":"state:sync","id":"test2"}
# → should receive: {"type":"state:sync","id":"test2","state":{...}}
# Get proxy secret
{"type":"proxy:secret","id":"test3"}
# → should receive: {"type":"proxy:secret","id":"test3","secret":"sk-ant-api03-..."}
```
### 6. Test lockfile protection
```bash
# Start sidecar in one terminal
bun run src/servers/sidecar/index.ts
# Try starting another in a second terminal
bun run src/servers/sidecar/index.ts
# → should exit with: "[sidecar] another instance is already running"
# Check lockfile
cat data/sidecar/sidecar.lock
# → PID of the running sidecar
```
### 7. Full integration test (sidecar + API server)
```bash
# Start sidecar first
bun run src/servers/sidecar/index.ts &
# Start API server
bun start
# → should see "[sidecar-client] connected" in logs
# Test via the dashboard:
# 1. Open a chat panel, send a message with claude-code model
# → should see streaming response (routed through sidecar)
# 2. Open a chat with a Pi model
# → should see streaming response (routed through sidecar)
# 3. Restart the API server (kill + bun start)
# → active Claude Code processes should NOT die
# → sidecar should show "client disconnected" then "client connected"
# → proxy secret should remain the same
```
### 8. Test API server restart resilience
This is the key scenario that motivated the sidecar:
```bash
# 1. Start sidecar + API server
bun run src/servers/sidecar/index.ts &
bun start &
# 2. Start a Claude Code streaming session in the dashboard
# 3. While Claude Code is running, kill the API server
kill $(pgrep -f "bun start")
# 4. Restart the API server
bun start
# 5. Check:
# - The Claude Code process should still be running (check with ps)
# - The proxy secret should be the same (check data/sidecar/state.json)
# - The sidecar should show the reconnection in logs
```
### 9. Test with pm2
```bash
pm2 start ecosystem.config.cjs
# Check both are running
pm2 status
# → officer-sidecar: online
# → officer: online
# Restart API server only
pm2 restart officer
# Check sidecar is still running
pm2 status
curl http://127.0.0.1:5100/health
# Stop everything
pm2 stop all
```
### 10. Queue test
```bash
# With sidecar running, queue a job via the API
curl -X POST http://localhost:5000/api/queue/jobs \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-token>" \
-d '{"lane":"test","type":"gmail-sync"}'
# List jobs
curl http://localhost:5000/api/queue/jobs \
-H "Authorization: Bearer <your-token>"
```
## Graceful Shutdown
On SIGTERM or SIGINT, the sidecar:
1. Flushes pending state to `data/sidecar/state.json`
2. Releases the lockfile
3. Does **NOT** kill spawned processes — they are independent OS processes
On restart, the sidecar reads the persisted state and checks which PIDs are still alive.
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Sidecar dies with live sessions | pm2 auto-restart + state.json + processes are independent OS processes |
| API↔sidecar WebSocket drops | Auto-reconnect with exponential backoff (200ms → 15s) |
| Two sidecar instances running | Lockfile with PID liveness check on startup |
| Sidecar needs DB access | Imports DB modules directly (same user, same filesystem) |
| Queue handlers need server context | Handlers are self-contained modules imported by the sidecar |