diff --git a/docs/jobs-unification.md b/docs/jobs-unification.md new file mode 100644 index 00000000..b20e537f --- /dev/null +++ b/docs/jobs-unification.md @@ -0,0 +1,66 @@ +# Jobs Unification — making every task a background job + +**Goal:** every task run (script, pipeline, later agentic) becomes a persisted, background **job** — +created over REST, streamed live over WebSocket, resumable/attachable, visible on desktop *and* phone, +and ending in a push notification. Replaces today's ephemeral script-task WebSocket path. + +**Context:** single user, forever. No multi-tenant concerns — "is anything running?" is a global check. +Favor power-user affordances over guardrails. See memory `sole-user-assume-competence`. + +## Current state (baseline) + +- **Pipeline tasks** already are jobs: `pipeline_jobs` (Postgres) + `pipeline-job-manager` (in-memory + live registry, viewer attach/replay buffer, 3s progress flush, finalize, restart-reconciliation) + + `/jobs` list (`JobsScreen`) + `/jobs/:id` detail (`JobDetail`, live-attach or persisted). WS at + `/api/tasks/pipeline/ws`. +- **Script tasks** (convert-video, edit-audio, …) run via `task-executor.ts` over `/api/tasks/run/ws` + — **ephemeral, zero persistence**, dies with the tab. Modal `TaskRunnerModal` (ScriptRunner branch). +- A **separate** file-backed queue engine exists at `src/servers/queue/` (lane-serial, retries, + crash-resume) wired only to email sync. NOT reused here — one Postgres jobs model instead. + +## Decisions (locked) + +- **Concurrency:** idle → single `Run` → `/jobs/:id`. Something running → `Queue` + a **red `Run`** + (force concurrent). Run → start now → `/jobs/:id`. Queue → append `pending` → `/jobs` list. + Backend (global): FIFO `pending` queue; on any finalize with 0 running, promote the oldest pending. + Force-Run bypasses the queue. Queue backlog survives restart (promoteNext on startup). +- **Modal → page:** replace `TaskRunnerModal` with a `/jobs/new` route. Right-click task action + navigates there instead of mounting the modal. One code path, phone-friendly. +- **Deep-link:** query params, not hash — `/jobs/new?task=convert-video&root=home&path=&entry=`. +- **Phone:** already has a file browser, so it can pick a target path and POST a job. No blocker. + +## Plan + +### Phase 1 — Unified jobs backend +- **1a. Data model.** Add `mode` (`pipeline|script|agentic`, default `pipeline`) + `exit_code` (int) + to the jobs table. Log at `DATA_PATH/jobs/.log` (derived from id). *Table/symbol rename + `pipeline_jobs`→`jobs` is deferred as a cosmetic cleanup — add columns first, keep it working.* +- **1b. Execution.** Generalize the job manager: `startJob` takes `mode` and dispatches — `pipeline` + → existing `executePipeline`; `script` → new `executeScript` (ports task-executor's + `materializeScript`/`buildInputEnv`/bwrap sandbox/`killTree`/keepalive, but emits job events + + appends stdout/stderr to the log file, finalizes with `exit_code`). Reuses the manager's + viewer/broadcast/replay machinery. +- **1c. Scheduler.** `runningCount()` + FIFO `pending` promotion on finalize; `promoteNext()` on + startup so a queued backlog resumes. + +### Phase 2 — REST job API (decouples creation from the socket; enables the phone) +- `POST /jobs {taskDirName, inputs, cwd, action}` → `{jobId}` (create + start/queue, background). +- `GET /jobs` (+`?live=1`), `GET /jobs/:id`, `GET /jobs/:id/log?offset=`, `POST /jobs/:id/stop`. +- Consolidate the two WebSockets into one `/api/tasks/jobs/ws` doing only attach/stop/list. + +### Phase 3 — Frontend +- `/jobs/new` → `NewJobScreen`: reads query params, renders the input UI lifted from + `TaskRunnerModal` (`TaskInputForm` + per-group config + folder probing). Run/Queue per the + concurrency UX. `JobDetail` gains a script branch (terminal output: live attach, or from log when + idle). Retire `TaskRunnerModal`/`TaskRunnerDialog`/`useTaskRunner`. Header running-jobs indicator. + +### Phase 4 — Notifications (later) +- One `notifyJobDone(job)` hook at finalize → push to the phone app. + +## Progress +- [ ] 1a data model (migration + schema/types) +- [ ] 1b executeScript + manager dispatch +- [ ] 1c scheduler / queue +- [ ] 2 REST job API + WS consolidation +- [ ] 3 /jobs/new page + JobDetail script branch + retire modal + header indicator +- [ ] 4 push notifications