# Automation System — Context Document

This document is written for AI assistants working on this codebase. It captures architectural decisions, the full vision, and the current state of the automation/tools system built during a long design and implementation session. Read this before touching anything in `src/servers/api/pi/`, `src/servers/api/skills/`, `src/servers/api/tasks/`, `src/servers/api/processes/`, `seed/tools/`, `seed/extensions/`, or `src/servers/sync-*.ts`.

---

## What Officer Is

Officer is a self-hosted AI-native intranet for small and medium businesses. Every user gets a personal AI assistant, terminal, file browser, code editor, workspaces, and automation tools. The platform is multi-user with role-based access: `Member → Admin → Owner → Super Admin`.

The AI backend was originally Claude SDK / OpenCode. It has been migrated to **Pi** (`@mariozechner/pi-coding-agent`) as the primary agent harness, running in RPC mode as a subprocess managed by `src/servers/api/pi/pi-bridge.ts`.

---

## The Automation Ladder

The owner has a clear conceptual model for automation capabilities, ordered from atomic to orchestrated:

| Rung | Description | Status |
|------|-------------|--------|
| **Skills** | Passive markdown documentation the agent reads as context. Reference for CLIs, APIs, services. | ✅ Implemented |
| **Tools** | Executable TypeScript functions Pi can call directly during agent execution. Registered via Pi extensions. | ✅ Implemented |
| **Tasks** | Structured instructions (TASK.md) to accomplish an atomic goal. Uses skills + tools. Has inputs, triggers, steps. | ✅ Skeleton exists |
| **Pipelines** | A linear sequence of tasks, run one after the other (think pipe). | 🔲 Phase 2 |
| **Processes** | A sequence of tasks that forks based on the result of the previous one (conditional branching). | 🔲 Phase 2 |
| **Workflows** | Arbitrary task graphs, like n8n. | 🔲 Phase 3 |
| **Services** | Event-driven execution — watches directories or events, triggers tasks based on configuration. | 🔲 Phase 3 |
| **Crons** | Scheduled execution of a single task, pipeline, process, or workflow. | 🔲 Phase 3 |

**When the owner says "tool", they mean any rung of this ladder**, not just the Pi tool concept.

### Key distinction: agent-interpreted vs headless execution

Currently all tasks are **agent-interpreted** — the agent reads the TASK.md and reasons about how to execute it. In phase 2, well-defined tasks (deterministic inputs → tool call → result) should be executable **headlessly** without an LLM, directly by a task runner. The tool implementations are already standalone TypeScript functions that support this future — they have no dependency on Pi or the agent.

---

## How Pi Is Integrated

Pi runs as a subprocess in **RPC mode** (`pi --mode rpc`). The server communicates via stdin/stdout JSON. See `src/servers/api/pi/pi-bridge.ts` and `src/servers/api/pi/websocket.ts`.

### Spawning Pi (host, non-sandboxed)

```
pi --mode rpc
   --no-skills --no-prompt-templates --no-themes
   --skill /data/skills/{name}           (one per global skill)
   --skill /data/{email}/skills/{name}   (one per user skill)
   --extension /data/extensions/{name}/index.ts
   --model {model}

env:
  PI_CODING_AGENT_DIR = DATA_PATH/pi-config
  PI_TOOLS_DIRS       = DATA_PATH/tools:DATA_PATH/{email}/tools
  PI_SEARXNG_URL      = https://searxng.home.pastilhas.eu
  + all stored API keys
```

Note: `--no-extensions` was intentionally removed to allow our tool-loader extension to work. `--no-skills` is kept because we pass skills explicitly via `--skill` flags to control scope correctly.

### Spawning Pi (sandboxed, Docker container)

Same pattern but via `docker exec`, using container-side paths:

```
docker exec -i -w {workdir} \
  -e PI_CODING_AGENT_DIR=/home/{username}/.pi/agent \
  -e PI_TOOLS_DIRS=/officer/tools:/officer/user/tools \
  -e PI_SEARXNG_URL=... \
  -e {API_KEYS} \
  {containerId} \
  pi --mode rpc --no-skills --no-prompt-templates --no-themes \
     --skill /officer/skills/{name} \
     --skill /officer/user/skills/{name} \
     --extension /officer/extensions/tool-loader/index.ts
```

---

## Directory Structure

```
DATA_PATH/                          # Default: ./data, override with DATA_PATH env
├── pi-config/                      # Pi's global config (models.json, settings.json)
├── skills/                         # Global skills (org-wide)
├── tools/                          # Global tools (org-wide)
├── extensions/                     # Global extensions (managed by server, not users)
├── {email}/
│   ├── home/                       # Mounted as /home/{username} in Docker container
│   ├── skills/                     # User-specific skills
│   ├── tools/                      # User-specific tools
│   └── extensions/                 # User-specific extensions (future)
└── searxng.json                    # SearXNG instance URL config

seed/                               # Bundled with the app (read-only source of truth)
├── skills/                         # Native/officerdev skills
├── tools/                          # Native/officerdev tools
├── extensions/                     # Native/officerdev extensions
│   └── tool-loader/index.ts        # The Pi extension that registers tools
└── tasks/                          # Native/officerdev tasks
```

### Sync on server start (`src/servers/bootstrap.ts`)

- `syncSeedSkills()` → copies `seed/skills/*` → `DATA_PATH/skills/` (skip if exists, preserves user edits)
- `syncSeedTools()` → copies `seed/tools/*` → `DATA_PATH/tools/` (skip if exists)
- `syncSeedExtensions()` → copies `seed/extensions/*` → `DATA_PATH/extensions/` (**always overwrites** — extensions are server-managed code, not user-editable)

---

## The Tool System

### How tools are defined

Each tool lives in a directory with two files:

```
seed/tools/web-fetch/
├── TOOL.md      # Metadata + input schema (read by tool-loader, registered with Pi)
└── index.ts     # Implementation (lazy-loaded when the tool is actually called)
```

**TOOL.md frontmatter format:**
```yaml
---
name: tool_name          # Pi tool name (snake_case)
label: Tool Label        # Human-readable label
description: ...         # What Pi sees in its context (keep concise — this is in every session prompt)
language: typescript     # typescript | bash | python
inputs:
  param_name:
    type: string         # string | number | boolean | enum
    description: ...
    optional: true       # omit if required
  mode:
    type: enum
    values: single,batch # comma-separated for enum (parser limitation)
    description: ...
---
```

**index.ts export signature:**
```typescript
export async function execute(
  toolCallId: string,
  params: { [key: string]: any },
  signal: AbortSignal | undefined,
  onUpdate?: (partial: { content: Array<{ type: string; text: string }> }) => void,
): Promise<{ content: Array<{ type: string; text: string }>; details?: object; isError?: boolean }>
```

The `onUpdate` callback streams progress to the agent. Use it liberally — the frontend shows it in real time.

### How the tool-loader extension works (`seed/extensions/tool-loader/index.ts`)

- Loaded by Pi as an extension via `--extension` flag
- Reads `PI_TOOLS_DIRS` env var (colon-separated list of directories)
- Discovers `TOOL.md` in each directory, parses frontmatter, builds TypeBox schema
- Registers each tool synchronously in the extension factory function (so tools appear in the system prompt)
- Lazy-loads `index.ts` implementation only when the tool is actually called
- User dirs come after global — later registration wins (user tools override global by name)

### Existing tools

| Tool | Location | Description |
|------|----------|-------------|
| `web_fetch` | `seed/tools/web-fetch/` | Fetch URL content. Tries `.md` suffix → `/llms.txt` → plain text → HTML strip |
| `web_search` | `seed/tools/web-search/` | Search via SearXNG. Returns titles, URLs, snippets. Pair with web_fetch |
| `convert_audio_to_mp3` | `seed/tools/convert-audio-to-mp3/` | Convert audio to MP3 via ffmpeg. Single file (% progress) or batch directory (per-file progress) |

### Adding a new tool

1. Create `seed/tools/{name}/TOOL.md` and `seed/tools/{name}/index.ts`
2. Restart server — `syncSeedTools()` copies it to `DATA_PATH/tools/`
3. New Pi sessions automatically get the tool registered

---

## Docker Container Setup

Each user has a persistent Docker container (`officer-terminal-{userId}`). Containers are managed by `src/servers/api/terminal/websocket.ts`.

### Read-only resource mounts

Added to every container at creation time:

```
/officer/skills          → DATA_PATH/skills           (global skills, ro)
/officer/tools           → DATA_PATH/tools            (global tools, ro)
/officer/extensions      → DATA_PATH/extensions       (global extensions, ro)
/officer/user/skills     → DATA_PATH/{email}/skills   (user skills, ro)
/officer/user/tools      → DATA_PATH/{email}/tools    (user tools, ro)
```

The user's home directory (`DATA_PATH/{email}/home`) is mounted read-write as `/home/{username}`.

### Mount migration

`ensureDockerContainer` checks `containerHasResourceMounts()` before reusing an existing container. If the mounts are missing (old container created before this feature), the container is removed and recreated automatically. This is a one-time migration.

---

## Scope & Permission Model

All automation entities (skills, tasks, processes, and future rungs) follow the same three-tier scope model:

| Scope | Location | Who creates | Who can see |
|-------|----------|-------------|-------------|
| `native` | `seed/` (read-only) | officerdev (us) | Everyone |
| `global` | `DATA_PATH/{type}/` | Admins+ | Everyone |
| `user` | `DATA_PATH/{email}/{type}/` | Anyone | Owner + SAs (future) |

### Current API behavior (skills, tasks, processes)

```typescript
function isPrivileged(role: string) {
  return role === 'Admin' || role === 'Owner' || role === 'Super Admin';
}
```

- `POST /` (create): Members → user scope. Admins+ → global scope.
- `DELETE /:name`: Members can only delete their own (user scope). Admins+ can delete global.
- `PUT /:name/chat` / `DELETE /:name/chat`: Members blocked from writing to native/global entries.

### Full vision (phase 2/3 — not yet implemented)

- **Super Admins** can see all users' personal items (currently everyone only sees their own)
- **Members can submit** their personal item for SA review (`status: draft | submitted | approved | rejected` — a flag in frontmatter, not a new scope)
- **SAs can promote** any user's item to global — this is a **move** (not a copy), item leaves user scope
- **SAs can see and use** everyone's tools
- The `CapabilitySummary` type already has `scope` surfaced in the frontend list with a badge

---

## SearXNG Integration

- Config stored in `DATA_PATH/searxng.json`
- Default URL: `https://searxng.home.pastilhas.eu` (hardcoded default in `src/servers/api/server-settings/searxng.ts`)
- API: `GET /api/server-settings/searxng`, `PUT /api/server-settings/searxng`
- Injected into Pi processes as `PI_SEARXNG_URL` env var (both host and container)
- UI configuration is a future task — the API is already there

---

## Attribution & Marketplace Vision (future)

The owner has a clear attribution model:

- **Native/officerdev**: `officerdev/<name>` — built-in, shipped with the product
- **User-created org tools**: attribution via `author: <email>` in frontmatter
- **Marketplace — officerdev official**: `officerdev/<name>`
- **Marketplace — third party**: `<dev>/<name>` or `<organization>/<name>`

Future marketplace will cover: Skills, Tools, Tasks, Pipelines, Processes, Workflows, Services, Crons, **Widgets**, **Apps**, and **Themes**. Widgets and Apps are already bootstrapped in the platform. Each self-hosted instance will connect to a central webstore.

The `native` scope covers both built-in and future marketplace-installed items. When marketplace is implemented, `native` may split into `native` (local seed) and `marketplace` (installed from store, with version + source metadata).

---

## Files Changed / Created

### New files
- `src/servers/sync-tools.ts` — mirrors sync-skills pattern for tools
- `src/servers/sync-extensions.ts` — same for extensions (always overwrites)
- `src/servers/api/server-settings/searxng.ts` — SearXNG config router
- `seed/tools/web-fetch/TOOL.md` + `index.ts`
- `seed/tools/web-search/TOOL.md` + `index.ts`
- `seed/tools/convert-audio-to-mp3/TOOL.md` + `index.ts`
- `seed/extensions/tool-loader/index.ts`

### Modified files
- `src/servers/data-path.ts` — added tool/extension dir helpers
- `src/servers/bootstrap.ts` — calls syncSeedTools, syncSeedExtensions
- `src/servers/api/pi/pi-bridge.ts` — removed `--no-extensions`, added extension/tool flags, PI_TOOLS_DIRS, PI_SEARXNG_URL, container-aware path support
- `src/servers/api/terminal/websocket.ts` — added 5 read-only resource mounts to containers, mount migration check
- `src/servers/api/skills/skills.ts` — fixed scope-aware POST/DELETE/chat endpoints
- `src/servers/api/tasks/tasks.ts` — same fixes
- `src/servers/api/processes/processes.ts` — same fixes
- `src/servers/api/server-settings/server-settings.ts` — registered searxng router
- `seed/skills/convert-audio-to-mp3/SKILL.md` — removed hardcoded paths, references tool instead
- `seed/tasks/convert-to-mp3/TASK.md` — updated to use tool, added `tools:` frontmatter field
- `src/apps/officer-web/Screens/Dashboard/CapabilityPage.tsx` — fixed `scope` type to include `'native'`

---

## What To Do Next (Rough Priority)

1. **Wire `tool_execution_update` events** through `pi-bridge.ts` → websocket → frontend so `onUpdate` progress actually shows in the chat UI
2. **Container support testing** — verify tools and extensions work correctly inside Docker after the read-only mount changes
3. **SA cross-user visibility** — API + UI for Super Admins to see all users' personal items
4. **Submit/approve workflow** — `status` flag in frontmatter, submission UI for members, review UI for SAs
5. **SearXNG UI** — settings panel to configure the URL (API is already done)
6. **Headless task runner** — execute deterministic tasks without an agent (phase 2)