pastilhasandClaude Opus 5
04c0d89057
First pass of the documentation audit. Every doc was read against what the code actually does now; this commit fixes the ones worth keeping and deletes the ones that were only describing a past. Corrected: - CLAUDE.md — said seven WebSocket providers (there are eight, and terminal/vault are byte relays now, not translating bridges), listed channels/ as "Telegram / WhatsApp / Discord bridges" (they are gone; what remains is how /chat drives an agent turn), missed officer-wallet in the PM2 list and notify/ in the layout, and described the per-account email SQLite stores without saying they are the sidecar's and that nothing in the platform opens them. Further Reading pointed at four files that no longer exist and missed the four newest. - docs/working-on-officer.md — PM2 list was four sidecars short, and it still explained the officer-claude rename as news. Replaced with the thing a reader actually needs: which process to restart for which change, and why restarting officer no longer costs you a terminal or an agent session. - TODO.md — the "dead username plumbing" item was mostly resolved by deleting the channels, and two email items pointed at api/email/email-db.ts, which is sidecar/email/store.ts now. - AGENTS.md — trailing paragraph listed the design notes being deleted here. - MUSIC_API.md — playlists were entirely undocumented: seven endpoints the phone app has no reference for. Added from the sidecar's own contract. - docs/jobs-unification.md — phases 1-3 shipped, so it now says so at the top. Phase 4 (push notifications) is the only reason the file still exists, and email sync is explicitly no longer part of it. Deleted, all superseded rather than merely old: - PHONE_APP.md — a February plan for apps that now exist, with their own repo and README. - MARKETING_WEBSITE.md — a plan for a site this repo does not contain. - SECURITY_AUDIT.md + SECURITY_FIXES.md — a February audit of a codebase since restructured; it still cites queue/handlers, which is now empty. - docs/DOCKERIZATION_PLAN.md — cites pty-sidecar, whatsapp and projects, all deleted. - SETUP_GUIDE.md — documents systemd units and setup scripts replaced by PM2 and `bun setup`. Not harmless: /etc/systemd/system/officer-pty-sidecar.service is still enabled on this host, pointing at a `monorepo/` directory that no longer exists, and has been failing to start ever since. That guide is how it got there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
99 lines
7.3 KiB
Markdown
99 lines
7.3 KiB
Markdown
# Jobs Unification — making every task a background job
|
||
|
||
**Status: shipped, except push notifications.** Phases 1–3 (data model, executor, queue, REST API and
|
||
the master-detail `/jobs` screen) are done and live. Phase 4 is the only open item, and is the reason
|
||
this file still exists. Email sync is deliberately NOT part of this any more — it moved into the email
|
||
sidecar with its own scheduling, so it does not appear in the Jobs list.
|
||
|
||
**Goal:** every task run (script, pipeline, later agentic) becomes a persisted, background **job** —
|
||
created over REST, streamed live over WebSocket, resumable/attachable, visible on desktop *and* phone,
|
||
and ending in a push notification. Replaces today's ephemeral script-task WebSocket path.
|
||
|
||
**Context:** single user, forever. No multi-tenant concerns — "is anything running?" is a global check.
|
||
Favor power-user affordances over guardrails. See memory `sole-user-assume-competence`.
|
||
|
||
## Current state (baseline)
|
||
|
||
- **Pipeline tasks** already are jobs: `pipeline_jobs` (Postgres) + `pipeline-job-manager` (in-memory
|
||
live registry, viewer attach/replay buffer, 3s progress flush, finalize, restart-reconciliation) +
|
||
`/jobs` list (`JobsScreen`) + `/jobs/:id` detail (`JobDetail`, live-attach or persisted). WS at
|
||
`/api/tasks/pipeline/ws`.
|
||
- **Script tasks** (convert-video, edit-audio, …) run via `task-executor.ts` over `/api/tasks/run/ws`
|
||
— **ephemeral, zero persistence**, dies with the tab. Modal `TaskRunnerModal` (ScriptRunner branch).
|
||
- A **separate** file-backed queue engine exists at `src/servers/queue/` (lane-serial, retries,
|
||
crash-resume) wired only to email sync. NOT reused here — one Postgres jobs model instead.
|
||
|
||
## Decisions (locked)
|
||
|
||
- **Concurrency:** idle → single `Run` → `/jobs/:id`. Something running → `Queue` + a **red `Run`**
|
||
(force concurrent). Run → start now → `/jobs/:id`. Queue → append `pending` → `/jobs` list.
|
||
Backend (global): FIFO `pending` queue; on any finalize with 0 running, promote the oldest pending.
|
||
Force-Run bypasses the queue. Queue backlog survives restart (promoteNext on startup).
|
||
- **Modal → page:** replace `TaskRunnerModal` with a `/jobs/new` route. Right-click task action
|
||
navigates there instead of mounting the modal. One code path, phone-friendly.
|
||
- **Deep-link:** query params, not hash — `/jobs/new?task=convert-video&root=home&path=<dir>&entry=<name>`.
|
||
- **Phone:** already has a file browser, so it can pick a target path and POST a job. No blocker.
|
||
- **`inline` tasks stay in the modal.** A top-level `inline: true` in `TASK.md` marks quick/interactive
|
||
tasks that run in the modal (ephemeral) instead of as a job. Default = job. Currently flagged:
|
||
`remove-jelly-meta`, `analyze-video`, `clean-playlist-files`. So the modal is **not** retired — it's
|
||
the inline path; the shared input form serves both the modal and `/jobs/new`. The flag is plumbed
|
||
through the parser + task list/detail endpoints (`task.inline`), so the FileBrowser menu can branch.
|
||
|
||
## Plan
|
||
|
||
### Phase 1 — Unified jobs backend
|
||
- **1a. Data model.** Add `mode` (`pipeline|script|agentic`, default `pipeline`) + `exit_code` (int)
|
||
to the jobs table. Log at `DATA_PATH/jobs/<id>.log` (derived from id). *Table/symbol rename
|
||
`pipeline_jobs`→`jobs` is deferred as a cosmetic cleanup — add columns first, keep it working.*
|
||
- **1b. Execution.** Generalize the job manager: `startJob` takes `mode` and dispatches — `pipeline`
|
||
→ existing `executePipeline`; `script` → new `executeScript` (ports task-executor's
|
||
`materializeScript`/`buildInputEnv`/bwrap sandbox/`killTree`/keepalive, but emits job events +
|
||
appends stdout/stderr to the log file, finalizes with `exit_code`). Reuses the manager's
|
||
viewer/broadcast/replay machinery.
|
||
- **1c. Scheduler.** `runningCount()` + FIFO `pending` promotion on finalize; `promoteNext()` on
|
||
startup so a queued backlog resumes.
|
||
|
||
### Phase 2 — REST job API (decouples creation from the socket; enables the phone)
|
||
- `POST /jobs {taskDirName, inputs, cwd, action}` → `{jobId}` (create + start/queue, background).
|
||
- `GET /jobs` (+`?live=1`), `GET /jobs/:id`, `GET /jobs/:id/log?offset=`, `POST /jobs/:id/stop`.
|
||
- Consolidate the two WebSockets into one `/api/tasks/jobs/ws` doing only attach/stop/list.
|
||
|
||
### Phase 3 — Frontend
|
||
- `/jobs/new` → `NewJobScreen`: reads query params, renders the input UI lifted from
|
||
`TaskRunnerModal` (`TaskInputForm` + per-group config + folder probing). Run/Queue per the
|
||
concurrency UX. `JobDetail` gains a script branch (terminal output: live attach, or from log when
|
||
idle). Retire `TaskRunnerModal`/`TaskRunnerDialog`/`useTaskRunner`. Header running-jobs indicator.
|
||
|
||
### Phase 4 — Notifications (later)
|
||
- One `notifyJobDone(job)` hook at finalize → push to the phone app.
|
||
|
||
## Progress
|
||
- [x] 1a data model — `mode` + `exit_code` columns (schema + applied to DB)
|
||
- [x] 1b executeScript + manager dispatch — `execute-script.ts` (spawn/sandbox/killTree port, log file,
|
||
abort poll, returns exitCode), `process-tree.ts` (shared killTree), `pipeline-job-manager` now
|
||
dispatches by `mode` and finalizes script jobs by exit code. *Compiles; runtime-untested until
|
||
a REST caller + restart exist.*
|
||
- [x] 1c scheduler / queue — `enqueueJob(action)` (start now / queue behind running), `promoteNext()`
|
||
on finalize + startup, `getOldestPendingJob`, `markInterruptedJobs` now running-only (pending
|
||
queue survives restart). `startJob` kept as a `enqueueJob(...,'start')` wrapper.
|
||
- [x] 2 REST job API — `POST /jobs` (create script|pipeline, action start/queue), `GET /jobs` (+`?live=1`,
|
||
now returns mode/exitCode/isLive), `GET /jobs/:id`, `GET /jobs/:id/log?offset=`, `POST /jobs/:id/stop`.
|
||
Router mounted at `/jobs` and `/pipeline-jobs`. *Needs a restart to deploy; then curl/phone-testable.*
|
||
WS consolidation still pending (old `/api/tasks/run/ws` + `/api/tasks/pipeline/ws` still live).
|
||
- [x] 3 frontend — master-detail `/jobs`, modal-as-creator, split list, header badges. Done.
|
||
- [x] 3a jobs UI — **master-detail** `JobsPage` (like `/chat`): `WorkspaceLayout` with a list panel
|
||
(left, polls `GET /jobs`, highlights active) + a detail panel (right) that branches by `mode`
|
||
— `ScriptJobDetail` terminal (polls log + status, Stop) or `PipelineJobDetail`. One page serves
|
||
both `/jobs` and `/jobs/:id`; list click navigates. Replaced the old separate list/detail pages.
|
||
- [x] `inline` flag plumbed (parser + list/detail endpoints); 3 quick tasks flagged.
|
||
- [x] 3b/3c task→job via the modal (pragmatic reuse). The task modal is now the job creator: on Run,
|
||
an **inline** task runs ephemerally in-modal; a **non-inline** task `POST /jobs` (start) →
|
||
navigates to `/jobs/:id`. When a job is already running, a red "Run now" + a "Queue" button
|
||
(queue → `/jobs`). Reuses the modal's per-group input UI in place — no separate `/jobs/new`
|
||
page or FileBrowser change needed. *(A standalone deep-linkable `/jobs/new` is deferred; the
|
||
phone creates jobs directly via `POST /jobs`.)*
|
||
- [x] 3d header job indicators — `JobsIndicator` (two always-present badges next to RescanButton +
|
||
UserMenu): **running** (→ running job's `/jobs/:id`) + **queued** (→ `/jobs`), polling
|
||
`GET /jobs/counts` → `{ running, runningJobId, queued }` every 3s; dim at 0.
|
||
- [ ] 4 push notifications
|