From a82ce9ecb09378a2b46c080f908b6962fd956d64 Mon Sep 17 00:00:00 2001 From: Andre Padez Date: Wed, 22 Jul 2026 20:25:44 +0100 Subject: [PATCH] drop inherited architecture docs that no longer match the code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AUTOMATION_CONTEXT.md and SIDECAR.md described the multi-tenant scope model, the seed/ tree, the marketplace, and a single sidecar owning the queue — all superseded. Fix the stale doc pointers in opencode.json and CLAUDE.md, and point at TODO.md as the source of truth on direction. Co-Authored-By: Claude Opus 4.8 --- AUTOMATION_CONTEXT.md | 293 -------------------------- CLAUDE.md | 6 +- opencode.json | 4 - src/servers/sidecar/SIDECAR.md | 366 --------------------------------- 4 files changed, 3 insertions(+), 666 deletions(-) delete mode 100644 AUTOMATION_CONTEXT.md delete mode 100644 src/servers/sidecar/SIDECAR.md diff --git a/AUTOMATION_CONTEXT.md b/AUTOMATION_CONTEXT.md deleted file mode 100644 index 99e51352..00000000 --- a/AUTOMATION_CONTEXT.md +++ /dev/null @@ -1,293 +0,0 @@ -# Automation System — Context Document - -This document is written for AI assistants working on this codebase. It captures architectural decisions, the full vision, and the current state of the automation/tools system built during a long design and implementation session. Read this before touching anything in `src/servers/api/pi/`, `src/servers/api/skills/`, `src/servers/api/tasks/`, `src/servers/api/processes/`, `seed/tools/`, `seed/extensions/`, or `src/servers/sync-*.ts`. - ---- - -## What Officer Is - -Officer is a self-hosted AI-native intranet for small and medium businesses. Every user gets a personal AI assistant, terminal, file browser, code editor, workspaces, and automation tools. The platform is multi-user with role-based access: `Member → Admin → Owner → Super Admin`. - -The AI backend was originally Claude SDK / OpenCode. It has been migrated to **Pi** (`@mariozechner/pi-coding-agent`) as the primary agent harness, running in RPC mode as a subprocess managed by `src/servers/api/pi/pi-bridge.ts`. - ---- - -## The Automation Ladder - -The owner has a clear conceptual model for automation capabilities, ordered from atomic to orchestrated: - -| Rung | Description | Status | -|------|-------------|--------| -| **Skills** | Passive markdown documentation the agent reads as context. Reference for CLIs, APIs, services. | ✅ Implemented | -| **Tools** | Executable TypeScript functions Pi can call directly during agent execution. Registered via Pi extensions. | ✅ Implemented | -| **Tasks** | Structured instructions (TASK.md) to accomplish an atomic goal. Uses skills + tools. Has inputs, triggers, steps. | ✅ Skeleton exists | -| **Pipelines** | A linear sequence of tasks, run one after the other (think pipe). | 🔲 Phase 2 | -| **Processes** | A sequence of tasks that forks based on the result of the previous one (conditional branching). | 🔲 Phase 2 | -| **Workflows** | Arbitrary task graphs, like n8n. | 🔲 Phase 3 | -| **Services** | Event-driven execution — watches directories or events, triggers tasks based on configuration. | 🔲 Phase 3 | -| **Crons** | Scheduled execution of a single task, pipeline, process, or workflow. | 🔲 Phase 3 | - -**When the owner says "tool", they mean any rung of this ladder**, not just the Pi tool concept. - -### Key distinction: agent-interpreted vs headless execution - -Currently all tasks are **agent-interpreted** — the agent reads the TASK.md and reasons about how to execute it. In phase 2, well-defined tasks (deterministic inputs → tool call → result) should be executable **headlessly** without an LLM, directly by a task runner. The tool implementations are already standalone TypeScript functions that support this future — they have no dependency on Pi or the agent. - ---- - -## How Pi Is Integrated - -Pi runs as a subprocess in **RPC mode** (`pi --mode rpc`). The server communicates via stdin/stdout JSON. See `src/servers/api/pi/pi-bridge.ts` and `src/servers/api/pi/websocket.ts`. - -### Spawning Pi (host, non-sandboxed) - -``` -pi --mode rpc - --no-skills --no-prompt-templates --no-themes - --skill /data/skills/{name} (one per global skill) - --skill /data/{email}/skills/{name} (one per user skill) - --extension /data/extensions/{name}/index.ts - --model {model} - -env: - PI_CODING_AGENT_DIR = DATA_PATH/pi-config - PI_TOOLS_DIRS = DATA_PATH/tools:DATA_PATH/{email}/tools - PI_SEARXNG_URL = https://searxng.home.pastilhas.eu - + all stored API keys -``` - -Note: `--no-extensions` was intentionally removed to allow our tool-loader extension to work. `--no-skills` is kept because we pass skills explicitly via `--skill` flags to control scope correctly. - -### Spawning Pi (sandboxed, Docker container) - -Same pattern but via `docker exec`, using container-side paths: - -``` -docker exec -i -w {workdir} \ - -e PI_CODING_AGENT_DIR=/home/{username}/.pi/agent \ - -e PI_TOOLS_DIRS=/officer/tools:/officer/user/tools \ - -e PI_SEARXNG_URL=... \ - -e {API_KEYS} \ - {containerId} \ - pi --mode rpc --no-skills --no-prompt-templates --no-themes \ - --skill /officer/skills/{name} \ - --skill /officer/user/skills/{name} \ - --extension /officer/extensions/tool-loader/index.ts -``` - ---- - -## Directory Structure - -``` -DATA_PATH/ # Default: ./data, override with DATA_PATH env -├── pi-config/ # Pi's global config (models.json, settings.json) -├── skills/ # Global skills (org-wide) -├── tools/ # Global tools (org-wide) -├── extensions/ # Global extensions (managed by server, not users) -├── {email}/ -│ ├── home/ # Mounted as /home/{username} in Docker container -│ ├── skills/ # User-specific skills -│ ├── tools/ # User-specific tools -│ └── extensions/ # User-specific extensions (future) -└── searxng.json # SearXNG instance URL config - -seed/ # Bundled with the app (read-only source of truth) -├── skills/ # Native/officerdev skills -├── tools/ # Native/officerdev tools -├── extensions/ # Native/officerdev extensions -│ └── tool-loader/index.ts # The Pi extension that registers tools -└── tasks/ # Native/officerdev tasks -``` - -### Sync on server start (`src/servers/bootstrap.ts`) - -- `syncSeedSkills()` → copies `seed/skills/*` → `DATA_PATH/skills/` (skip if exists, preserves user edits) -- `syncSeedTools()` → copies `seed/tools/*` → `DATA_PATH/tools/` (skip if exists) -- `syncSeedExtensions()` → copies `seed/extensions/*` → `DATA_PATH/extensions/` (**always overwrites** — extensions are server-managed code, not user-editable) - ---- - -## The Tool System - -### How tools are defined - -Each tool lives in a directory with two files: - -``` -seed/tools/web-fetch/ -├── TOOL.md # Metadata + input schema (read by tool-loader, registered with Pi) -└── index.ts # Implementation (lazy-loaded when the tool is actually called) -``` - -**TOOL.md frontmatter format:** -```yaml ---- -name: tool_name # Pi tool name (snake_case) -label: Tool Label # Human-readable label -description: ... # What Pi sees in its context (keep concise — this is in every session prompt) -language: typescript # typescript | bash | python -inputs: - param_name: - type: string # string | number | boolean | enum - description: ... - optional: true # omit if required - mode: - type: enum - values: single,batch # comma-separated for enum (parser limitation) - description: ... ---- -``` - -**index.ts export signature:** -```typescript -export async function execute( - toolCallId: string, - params: { [key: string]: any }, - signal: AbortSignal | undefined, - onUpdate?: (partial: { content: Array<{ type: string; text: string }> }) => void, -): Promise<{ content: Array<{ type: string; text: string }>; details?: object; isError?: boolean }> -``` - -The `onUpdate` callback streams progress to the agent. Use it liberally — the frontend shows it in real time. - -### How the tool-loader extension works (`seed/extensions/tool-loader/index.ts`) - -- Loaded by Pi as an extension via `--extension` flag -- Reads `PI_TOOLS_DIRS` env var (colon-separated list of directories) -- Discovers `TOOL.md` in each directory, parses frontmatter, builds TypeBox schema -- Registers each tool synchronously in the extension factory function (so tools appear in the system prompt) -- Lazy-loads `index.ts` implementation only when the tool is actually called -- User dirs come after global — later registration wins (user tools override global by name) - -### Existing tools - -| Tool | Location | Description | -|------|----------|-------------| -| `web_fetch` | `seed/tools/web-fetch/` | Fetch URL content. Tries `.md` suffix → `/llms.txt` → plain text → HTML strip | -| `web_search` | `seed/tools/web-search/` | Search via SearXNG. Returns titles, URLs, snippets. Pair with web_fetch | -| `convert_audio_to_mp3` | `seed/tools/convert-audio-to-mp3/` | Convert audio to MP3 via ffmpeg. Single file (% progress) or batch directory (per-file progress) | - -### Adding a new tool - -1. Create `seed/tools/{name}/TOOL.md` and `seed/tools/{name}/index.ts` -2. Restart server — `syncSeedTools()` copies it to `DATA_PATH/tools/` -3. New Pi sessions automatically get the tool registered - ---- - -## Docker Container Setup - -Each user has a persistent Docker container (`officer-terminal-{userId}`). Containers are managed by `src/servers/api/terminal/websocket.ts`. - -### Read-only resource mounts - -Added to every container at creation time: - -``` -/officer/skills → DATA_PATH/skills (global skills, ro) -/officer/tools → DATA_PATH/tools (global tools, ro) -/officer/extensions → DATA_PATH/extensions (global extensions, ro) -/officer/user/skills → DATA_PATH/{email}/skills (user skills, ro) -/officer/user/tools → DATA_PATH/{email}/tools (user tools, ro) -``` - -The user's home directory (`DATA_PATH/{email}/home`) is mounted read-write as `/home/{username}`. - -### Mount migration - -`ensureDockerContainer` checks `containerHasResourceMounts()` before reusing an existing container. If the mounts are missing (old container created before this feature), the container is removed and recreated automatically. This is a one-time migration. - ---- - -## Scope & Permission Model - -All automation entities (skills, tasks, processes, and future rungs) follow the same three-tier scope model: - -| Scope | Location | Who creates | Who can see | -|-------|----------|-------------|-------------| -| `native` | `seed/` (read-only) | officerdev (us) | Everyone | -| `global` | `DATA_PATH/{type}/` | Admins+ | Everyone | -| `user` | `DATA_PATH/{email}/{type}/` | Anyone | Owner + SAs (future) | - -### Current API behavior (skills, tasks, processes) - -```typescript -function isPrivileged(role: string) { - return role === 'Admin' || role === 'Owner' || role === 'Super Admin'; -} -``` - -- `POST /` (create): Members → user scope. Admins+ → global scope. -- `DELETE /:name`: Members can only delete their own (user scope). Admins+ can delete global. -- `PUT /:name/chat` / `DELETE /:name/chat`: Members blocked from writing to native/global entries. - -### Full vision (phase 2/3 — not yet implemented) - -- **Super Admins** can see all users' personal items (currently everyone only sees their own) -- **Members can submit** their personal item for SA review (`status: draft | submitted | approved | rejected` — a flag in frontmatter, not a new scope) -- **SAs can promote** any user's item to global — this is a **move** (not a copy), item leaves user scope -- **SAs can see and use** everyone's tools -- The `CapabilitySummary` type already has `scope` surfaced in the frontend list with a badge - ---- - -## SearXNG Integration - -- Config stored in `DATA_PATH/searxng.json` -- Default URL: `https://searxng.home.pastilhas.eu` (hardcoded default in `src/servers/api/server-settings/searxng.ts`) -- API: `GET /api/server-settings/searxng`, `PUT /api/server-settings/searxng` -- Injected into Pi processes as `PI_SEARXNG_URL` env var (both host and container) -- UI configuration is a future task — the API is already there - ---- - -## Attribution & Marketplace Vision (future) - -The owner has a clear attribution model: - -- **Native/officerdev**: `officerdev/` — built-in, shipped with the product -- **User-created org tools**: attribution via `author: ` in frontmatter -- **Marketplace — officerdev official**: `officerdev/` -- **Marketplace — third party**: `/` or `/` - -Future marketplace will cover: Skills, Tools, Tasks, Pipelines, Processes, Workflows, Services, Crons, **Widgets**, **Apps**, and **Themes**. Widgets and Apps are already bootstrapped in the platform. Each self-hosted instance will connect to a central webstore. - -The `native` scope covers both built-in and future marketplace-installed items. When marketplace is implemented, `native` may split into `native` (local seed) and `marketplace` (installed from store, with version + source metadata). - ---- - -## Files Changed / Created - -### New files -- `src/servers/sync-tools.ts` — mirrors sync-skills pattern for tools -- `src/servers/sync-extensions.ts` — same for extensions (always overwrites) -- `src/servers/api/server-settings/searxng.ts` — SearXNG config router -- `seed/tools/web-fetch/TOOL.md` + `index.ts` -- `seed/tools/web-search/TOOL.md` + `index.ts` -- `seed/tools/convert-audio-to-mp3/TOOL.md` + `index.ts` -- `seed/extensions/tool-loader/index.ts` - -### Modified files -- `src/servers/data-path.ts` — added tool/extension dir helpers -- `src/servers/bootstrap.ts` — calls syncSeedTools, syncSeedExtensions -- `src/servers/api/pi/pi-bridge.ts` — removed `--no-extensions`, added extension/tool flags, PI_TOOLS_DIRS, PI_SEARXNG_URL, container-aware path support -- `src/servers/api/terminal/websocket.ts` — added 5 read-only resource mounts to containers, mount migration check -- `src/servers/api/skills/skills.ts` — fixed scope-aware POST/DELETE/chat endpoints -- `src/servers/api/tasks/tasks.ts` — same fixes -- `src/servers/api/processes/processes.ts` — same fixes -- `src/servers/api/server-settings/server-settings.ts` — registered searxng router -- `seed/skills/convert-audio-to-mp3/SKILL.md` — removed hardcoded paths, references tool instead -- `seed/tasks/convert-to-mp3/TASK.md` — updated to use tool, added `tools:` frontmatter field -- `src/apps/officer-web/Screens/Dashboard/CapabilityPage.tsx` — fixed `scope` type to include `'native'` - ---- - -## What To Do Next (Rough Priority) - -1. **Wire `tool_execution_update` events** through `pi-bridge.ts` → websocket → frontend so `onUpdate` progress actually shows in the chat UI -2. **Container support testing** — verify tools and extensions work correctly inside Docker after the read-only mount changes -3. **SA cross-user visibility** — API + UI for Super Admins to see all users' personal items -4. **Submit/approve workflow** — `status` flag in frontmatter, submission UI for members, review UI for SAs -5. **SearXNG UI** — settings panel to configure the URL (API is already done) -6. **Headless task runner** — execute deterministic tasks without an agent (phase 2) diff --git a/CLAUDE.md b/CLAUDE.md index 75dca301..aee5f43a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -258,11 +258,11 @@ Detailed patterns for each area live in their respective directories: **Frontend:** - `src/apps/CLAUDE.md` - Shared frontend patterns (components, hooks, state) -- `src/apps/dashboard/CLAUDE.md` - Dashboard specifics -- `src/apps/editor/CLAUDE.md` - Editor specifics +- `src/apps/officer-web/Screens/Dashboard/Automation/CLAUDE.md` - Automation screen specifics + **Backend:** -- `src/servers/CLAUDE.md` - API and tracking servers - `src/databases/CLAUDE.md` - Database schemas and patterns **Project-wide:** - `CONVENTIONS.md` - Detailed code patterns with rationale (component organization, state management, React patterns) +- `TODO.md` - Current direction and deferred work. Takes precedence over this file where they disagree. diff --git a/opencode.json b/opencode.json index f5682119..1f10d072 100644 --- a/opencode.json +++ b/opencode.json @@ -4,10 +4,6 @@ "CLAUDE.md", "CONVENTIONS.md", "src/apps/CLAUDE.md", - "src/apps/dashboard/CLAUDE.md", - "src/apps/editor/CLAUDE.md", - "src/apps/runtime/CLAUDE.md", - "src/servers/CLAUDE.md", "src/databases/CLAUDE.md" ] } diff --git a/src/servers/sidecar/SIDECAR.md b/src/servers/sidecar/SIDECAR.md deleted file mode 100644 index 891ccb30..00000000 --- a/src/servers/sidecar/SIDECAR.md +++ /dev/null @@ -1,366 +0,0 @@ -# Process Sidecar - -Independent Bun process that owns all long-running work so the API server can restart without disrupting active sessions. - -## Problem - -The API server (port 5000) previously owned all spawned processes: Pi agents, Claude Code sessions, the Anthropic proxy, and the job queue. Restarting the API server would: - -- Kill active Pi and Claude Code conversations mid-response -- Regenerate the Anthropic proxy secret, breaking any Claude Code sessions using it -- Lose in-flight job progress (queue engine ran in-process) - -## Solution - -A separate Bun process ("process sidecar") on port 5100 that owns all spawned processes. The API server communicates with it over a single WebSocket connection. The sidecar is managed by pm2 and starts before the API server. - -## Architecture - -``` -┌─────────────────────────┐ ┌──────────────────────────────┐ -│ API Server (:5000) │ WS │ Process Sidecar (:5100) │ -│ │◄───────►│ │ -│ - Browser WS clients │ │ - Anthropic Proxy (:5051) │ -│ - REST API routes │ │ - Claude Code processes │ -│ - Channel bots │ │ - Pi agent processes │ -│ - sidecar-client.ts │ │ - Job queue engine │ -│ (auto-reconnect) │ │ - State persistence │ -└─────────────────────────┘ └──────────────────────────────┘ -``` - -### What the sidecar owns - -| Concern | Previous location | Sidecar module | -|---------|-------------------|----------------| -| Anthropic proxy (port 5051) | `api/anthropic-proxy.ts` | `sidecar/proxy.ts` | -| Claude Code spawn + session tracking | `channels/send-claude-code.ts` | `sidecar/claude-manager.ts` | -| Pi agent spawn + RPC commands | `api/pi/pi-bridge.ts` | `sidecar/pi-manager.ts` | -| Job queue engine + handlers | `queue/engine.ts` | `sidecar/queue-runner.ts` | - -### What stays in the API server - -- Browser WebSocket connections (ephemeral by nature) -- REST API routes (now thin proxies to sidecar) -- Channel bots (Discord/Telegram/WhatsApp — already reconnect gracefully) -- Browser relay (CDP state is ephemeral) -- PTY sidecar (already its own process, unchanged) - -## Communication Protocol - -Single WebSocket between API server and sidecar. JSON messages with `{ type, id?, ... }` envelopes. - -**Request/response**: Commands include an `id` field. The sidecar responds with a message carrying the same `id`. The client correlates responses via this ID with configurable timeouts. - -**Streaming events**: Pi and Claude Code output events are broadcast to all connected clients without a correlation ID. They carry a `sessionId` or `sessionKey` so the API server can route them to the correct browser WS. - -### Command categories - -``` -ping / pong — health check -state:sync — full state snapshot on connect - -proxy:secret — get persisted proxy secret - -claude:spawn / claude:result — blocking Claude Code exec -claude:spawn-streaming / claude:event — streaming Claude Code exec -claude:kill / claude:clear-session — session management - -pi:spawn / pi:prompt / pi:abort — Pi agent lifecycle -pi:kill / pi:set-thinking — Pi session control - -queue:enqueue / queue:cancel — job management -queue:list / queue:get — job queries -``` - -See `protocol.ts` for the full type definitions. - -## State Persistence - -File: `data/sidecar/state.json` - -Written every 30 seconds (debounced) and on graceful shutdown (SIGTERM/SIGINT). Contains: - -- **proxySecret** — generated once on first boot, reused forever. This is the key fix: the Anthropic proxy secret no longer changes on restart. -- **claudeSessions** — map of `sessionKey → Claude Code session_id` for `--resume` support across restarts. -- **piSessions** — session metadata with PIDs for liveness checking on restart. - -### Lockfile - -`data/sidecar/sidecar.lock` — contains the PID of the running sidecar. On startup, if the lock exists and the PID is alive, the sidecar exits. Stale locks (dead PID) are cleaned up automatically. - -## File Structure - -``` -src/servers/sidecar/ - index.ts — entry point: Bun.serve on :5100, WS dispatch, shutdown - protocol.ts — shared message types (imported by both sides) - proxy.ts — Anthropic proxy server (moved from anthropic-proxy.ts) - claude-manager.ts — Claude Code blocking + streaming spawn, session map - pi-manager.ts — Pi agent spawn, RPC (prompt/abort/thinking), event parsing - queue-runner.ts — Job queue engine (lanes, retries, notifications) - state.ts — File-backed state persistence + lockfile - -src/servers/sidecar-client.ts — API server's WebSocket client (singleton) -``` - -## API Server Integration - -The API server connects to the sidecar on startup via `initSidecarClient()` in `server.tsx`. The client auto-reconnects with exponential backoff (200ms → 15s). - -On connect, it sends `state:sync` to get the current proxy secret and live session info. - -### Modified API server files - -| File | Change | -|------|--------| -| `server.tsx` | `startAnthropicProxy()` → `initSidecarClient()` | -| `bootstrap.ts` | Removed `initQueue()` (sidecar owns it) | -| `channels/send-claude-code.ts` | 550 lines → 73 lines thin client | -| `api/pi/websocket.ts` | Spawn/prompt/abort go through sidecar | -| `api/pi/session-manager.ts` | Cleanup sidecar subscriptions on delete | -| `api/queue/queue.ts` | Routes use sidecar client | -| `channels/discord/handler.ts` | `enqueue` → `enqueueJob` via sidecar | -| `channels/telegram/handler.ts` | Same | -| `channels/whatsapp/handler.ts` | Same | - -## PM2 Configuration - -```js -// ecosystem.config.cjs -module.exports = { - apps: [ - { - name: 'officer-sidecar', // starts first - script: 'bun', - args: 'run src/servers/sidecar/index.ts', - watch: false, - }, - { - name: 'officer', - script: 'bun', - args: 'start', - watch: false, - }, - ], -}; -``` - -The sidecar is listed first so pm2 starts it before the API server. The API server's sidecar client handles the case where the sidecar isn't ready yet (auto-reconnect with backoff). - -## Environment Variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `SIDECAR_PORT` | `5100` | Sidecar HTTP/WS port | -| `ANTHROPIC_PROXY_PORT` | `5051` | Anthropic proxy port (owned by sidecar) | -| `DATA_PATH` | `./data` | Shared data directory | - -## Manual Testing - -### 1. Start the sidecar standalone - -```bash -# From monorepo root -bun run src/servers/sidecar/index.ts -``` - -Expected output: -``` -[sidecar:proxy] listening on 127.0.0.1:5051 -[sidecar:queue] initialized -[sidecar] listening on 127.0.0.1:5100 -``` - -### 2. Health check - -```bash -# Basic liveness -curl http://127.0.0.1:5100/ -# → "process-sidecar" - -# Detailed health -curl http://127.0.0.1:5100/health -# → {"status":"ok","uptime":12345,"piSessions":0,"claudeSessions":0} -``` - -### 3. Verify Anthropic proxy is running - -```bash -# Should reject without valid secret -curl -s http://127.0.0.1:5051/v1/messages -# → {"error":"Unauthorized"} - -# Check state file was created -cat data/sidecar/state.json -# → should show proxySecret, empty claudeSessions, empty piSessions -``` - -### 4. Verify proxy secret persistence - -```bash -# Note the proxySecret from state.json -cat data/sidecar/state.json | jq .proxySecret - -# Stop and restart the sidecar -# Kill the sidecar (Ctrl+C or kill) -bun run src/servers/sidecar/index.ts - -# Check the secret is the same -cat data/sidecar/state.json | jq .proxySecret -# → should be identical to before -``` - -### 5. WebSocket communication test - -```bash -# In one terminal, start the sidecar -bun run src/servers/sidecar/index.ts - -# In another terminal, connect with websocat (or wscat) -# Install: cargo install websocat OR npm install -g wscat -websocat ws://127.0.0.1:5100 - -# Send a ping -{"type":"ping","id":"test1"} -# → should receive: {"type":"pong","id":"test1"} - -# Send state sync -{"type":"state:sync","id":"test2"} -# → should receive: {"type":"state:sync","id":"test2","state":{...}} - -# Get proxy secret -{"type":"proxy:secret","id":"test3"} -# → should receive: {"type":"proxy:secret","id":"test3","secret":"sk-ant-api03-..."} -``` - -### 6. Test lockfile protection - -```bash -# Start sidecar in one terminal -bun run src/servers/sidecar/index.ts - -# Try starting another in a second terminal -bun run src/servers/sidecar/index.ts -# → should exit with: "[sidecar] another instance is already running" - -# Check lockfile -cat data/sidecar/sidecar.lock -# → PID of the running sidecar -``` - -### 7. Full integration test (sidecar + API server) - -```bash -# Start sidecar first -bun run src/servers/sidecar/index.ts & - -# Start API server -bun start -# → should see "[sidecar-client] connected" in logs - -# Test via the dashboard: -# 1. Open a chat panel, send a message with claude-code model -# → should see streaming response (routed through sidecar) -# 2. Open a chat with a Pi model -# → should see streaming response (routed through sidecar) -# 3. Restart the API server (kill + bun start) -# → active Claude Code processes should NOT die -# → sidecar should show "client disconnected" then "client connected" -# → proxy secret should remain the same -``` - -### 8. Test API server restart resilience - -This is the key scenario that motivated the sidecar: - -```bash -# 1. Start sidecar + API server -bun run src/servers/sidecar/index.ts & -bun start & - -# 2. Start a Claude Code streaming session in the dashboard - -# 3. While Claude Code is running, kill the API server -kill $(pgrep -f "bun start") - -# 4. Restart the API server -bun start - -# 5. Check: -# - The Claude Code process should still be running (check with ps) -# - The proxy secret should be the same (check data/sidecar/state.json) -# - The sidecar should show the reconnection in logs -``` - -### 9. Test with pm2 - -```bash -pm2 start ecosystem.config.cjs - -# Check both are running -pm2 status -# → officer-sidecar: online -# → officer: online - -# Restart API server only -pm2 restart officer - -# Check sidecar is still running -pm2 status -curl http://127.0.0.1:5100/health - -# Stop everything -pm2 stop all -``` - -### 10. Queue test - -```bash -# With sidecar running, queue a job via the API -curl -X POST http://localhost:5000/api/queue/jobs \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer " \ - -d '{"lane":"test","type":"gmail-sync"}' - -# List jobs -curl http://localhost:5000/api/queue/jobs \ - -H "Authorization: Bearer " -``` - -## Graceful Shutdown - -On SIGTERM or SIGINT, the sidecar: -1. Flushes pending state to `data/sidecar/state.json` -2. Releases the lockfile -3. Does **NOT** kill spawned processes — they are independent OS processes - -On restart, the sidecar reads the persisted state and checks which PIDs are still alive. - -## Risks & Mitigations - -| Risk | Mitigation | -|------|------------| -| Sidecar dies with live sessions | pm2 auto-restart + state.json + processes are independent OS processes | -| API↔sidecar WebSocket drops | Auto-reconnect with exponential backoff (200ms → 15s) | -| Two sidecar instances running | Lockfile with PID liveness check on startup | -| Sidecar needs DB access | Imports DB modules directly (same user, same filesystem) | -| Queue handlers need server context | Handlers are self-contained modules imported by the sidecar | - -## Future Possibilities - -The current architecture is one API server → one sidecar. But the protocol is sidecar-agnostic and could evolve in the opposite direction: one API server → many sidecars. - -``` - ┌── sidecar-A (machine 1, user X's files) -API server ───┼── sidecar-B (machine 2, heavy compute) - └── sidecar-C (localhost, default) -``` - -In this model the API server becomes the orchestrator — it decides which sidecar to dispatch to based on the user, the task type, or available capacity. A user's Claude Code session runs on the machine where their project files live. A long-running Pi agent gets dispatched to a box with more resources. Quick jobs stay local. - -What would need to change: - -- **Sidecar identity**: Each sidecar needs a name/id. The current hardcoded `ws://127.0.0.1:5100` becomes a registry of endpoints (config file or DB table). -- **Connection pool**: `sidecar-client.ts` becomes a pool of connections rather than a singleton. -- **Routing**: The API server needs a session→sidecar mapping so it knows where to forward browser messages. -- **The protocol itself**: Doesn't change. A `pi:spawn` command works the same whether it goes to a local or remote sidecar.