platform/docs/jobs-unification.md

Jobs Unification — making every task a background job

Status: shipped, except push notifications. Phases 1–3 (data model, executor, queue, REST API and the master-detail /jobs screen) are done and live. Phase 4 is the only open item, and is the reason this file still exists. Email sync is deliberately NOT part of this any more — it moved into the email sidecar with its own scheduling, so it does not appear in the Jobs list.

Goal: every task run (script, pipeline, later agentic) becomes a persisted, background job — created over REST, streamed live over WebSocket, resumable/attachable, visible on desktop and phone, and ending in a push notification. Replaces today's ephemeral script-task WebSocket path.

Context: single user, forever. No multi-tenant concerns — "is anything running?" is a global check. Favor power-user affordances over guardrails. See memory sole-user-assume-competence.

Current state (baseline)

• Pipeline tasks already are jobs: pipeline_jobs (Postgres) + pipeline-job-manager (in-memory live registry, viewer attach/replay buffer, 3s progress flush, finalize, restart-reconciliation) + /jobs list (JobsScreen) + /jobs/:id detail (JobDetail, live-attach or persisted). WS at /api/tasks/pipeline/ws.
• Script tasks (convert-video, edit-audio, …) run via task-executor.ts over /api/tasks/run/ws — ephemeral, zero persistence, dies with the tab. Modal TaskRunnerModal (ScriptRunner branch).
• A separate file-backed queue engine exists at src/servers/queue/ (lane-serial, retries, crash-resume) wired only to email sync. NOT reused here — one Postgres jobs model instead.

Decisions (locked)

• Concurrency: idle → single Run → /jobs/:id. Something running → Queue + a red Run (force concurrent). Run → start now → /jobs/:id. Queue → append pending → /jobs list. Backend (global): FIFO pending queue; on any finalize with 0 running, promote the oldest pending. Force-Run bypasses the queue. Queue backlog survives restart (promoteNext on startup).
• Modal → page: replace TaskRunnerModal with a /jobs/new route. Right-click task action navigates there instead of mounting the modal. One code path, phone-friendly.
• Deep-link: query params, not hash — /jobs/new?task=convert-video&root=home&path=<dir>&entry=<name>.
• Phone: already has a file browser, so it can pick a target path and POST a job. No blocker.
• inline tasks stay in the modal. A top-level inline: true in TASK.md marks quick/interactive tasks that run in the modal (ephemeral) instead of as a job. Default = job. Currently flagged: remove-jelly-meta, analyze-video, clean-playlist-files. So the modal is not retired — it's the inline path; the shared input form serves both the modal and /jobs/new. The flag is plumbed through the parser + task list/detail endpoints (task.inline), so the FileBrowser menu can branch.

Plan

Phase 1 — Unified jobs backend

• 1a. Data model. Add mode (pipeline|script|agentic, default pipeline) + exit_code (int) to the jobs table. Log at DATA_PATH/jobs/<id>.log (derived from id). Table/symbol rename pipeline_jobs→jobs is deferred as a cosmetic cleanup — add columns first, keep it working.
• 1b. Execution. Generalize the job manager: startJob takes mode and dispatches — pipeline → existing executePipeline; script → new executeScript (ports task-executor's materializeScript/buildInputEnv/bwrap sandbox/killTree/keepalive, but emits job events + appends stdout/stderr to the log file, finalizes with exit_code). Reuses the manager's viewer/broadcast/replay machinery.
• 1c. Scheduler. runningCount() + FIFO pending promotion on finalize; promoteNext() on startup so a queued backlog resumes.

Phase 2 — REST job API (decouples creation from the socket; enables the phone)

• POST /jobs {taskDirName, inputs, cwd, action} → {jobId} (create + start/queue, background).
• GET /jobs (+?live=1), GET /jobs/:id, GET /jobs/:id/log?offset=, POST /jobs/:id/stop.
• Consolidate the two WebSockets into one /api/tasks/jobs/ws doing only attach/stop/list.

Phase 3 — Frontend

• /jobs/new → NewJobScreen: reads query params, renders the input UI lifted from TaskRunnerModal (TaskInputForm + per-group config + folder probing). Run/Queue per the concurrency UX. JobDetail gains a script branch (terminal output: live attach, or from log when idle). Retire TaskRunnerModal/TaskRunnerDialog/useTaskRunner. Header running-jobs indicator.

Phase 4 — Notifications (later)

• One notifyJobDone(job) hook at finalize → push to the phone app.

Progress

• 1a data model — mode + exit_code columns (schema + applied to DB)
• 1b executeScript + manager dispatch — execute-script.ts (spawn/sandbox/killTree port, log file, abort poll, returns exitCode), process-tree.ts (shared killTree), pipeline-job-manager now dispatches by mode and finalizes script jobs by exit code. Compiles; runtime-untested until a REST caller + restart exist.
• 1c scheduler / queue — enqueueJob(action) (start now / queue behind running), promoteNext() on finalize + startup, getOldestPendingJob, markInterruptedJobs now running-only (pending queue survives restart). startJob kept as a enqueueJob(...,'start') wrapper.
• 2 REST job API — POST /jobs (create script|pipeline, action start/queue), GET /jobs (+?live=1, now returns mode/exitCode/isLive), GET /jobs/:id, GET /jobs/:id/log?offset=, POST /jobs/:id/stop. Router mounted at /jobs and /pipeline-jobs. Needs a restart to deploy; then curl/phone-testable. WS consolidation still pending (old /api/tasks/run/ws + /api/tasks/pipeline/ws still live).
• 3 frontend — master-detail /jobs, modal-as-creator, split list, header badges. Done.
  • 3a jobs UI — master-detail JobsPage (like /chat): WorkspaceLayout with a list panel (left, polls GET /jobs, highlights active) + a detail panel (right) that branches by mode — ScriptJobDetail terminal (polls log + status, Stop) or PipelineJobDetail. One page serves both /jobs and /jobs/:id; list click navigates. Replaced the old separate list/detail pages.
  • inline flag plumbed (parser + list/detail endpoints); 3 quick tasks flagged.
  • 3b/3c task→job via the modal (pragmatic reuse). The task modal is now the job creator: on Run, an inline task runs ephemerally in-modal; a non-inline task POST /jobs (start) → navigates to /jobs/:id. When a job is already running, a red "Run now" + a "Queue" button (queue → /jobs). Reuses the modal's per-group input UI in place — no separate /jobs/new page or FileBrowser change needed. (A standalone deep-linkable /jobs/new is deferred; the phone creates jobs directly via POST /jobs.)
  • 3d header job indicators — JobsIndicator (two always-present badges next to RescanButton + UserMenu): running (→ running job's /jobs/:id) + queued (→ /jobs), polling GET /jobs/counts → { running, runningJobId, queued } every 3s; dim at 0.
• 4 push notifications