platform/docs/jobs-unification.md

Jobs Unification — making every task a background job

Goal: every task run (script, pipeline, later agentic) becomes a persisted, background job — created over REST, streamed live over WebSocket, resumable/attachable, visible on desktop and phone, and ending in a push notification. Replaces today's ephemeral script-task WebSocket path.

Context: single user, forever. No multi-tenant concerns — "is anything running?" is a global check. Favor power-user affordances over guardrails. See memory sole-user-assume-competence.

Current state (baseline)

• Pipeline tasks already are jobs: pipeline_jobs (Postgres) + pipeline-job-manager (in-memory live registry, viewer attach/replay buffer, 3s progress flush, finalize, restart-reconciliation) + /jobs list (JobsScreen) + /jobs/:id detail (JobDetail, live-attach or persisted). WS at /api/tasks/pipeline/ws.
• Script tasks (convert-video, edit-audio, …) run via task-executor.ts over /api/tasks/run/ws — ephemeral, zero persistence, dies with the tab. Modal TaskRunnerModal (ScriptRunner branch).
• A separate file-backed queue engine exists at src/servers/queue/ (lane-serial, retries, crash-resume) wired only to email sync. NOT reused here — one Postgres jobs model instead.

Decisions (locked)

• Concurrency: idle → single Run → /jobs/:id. Something running → Queue + a red Run (force concurrent). Run → start now → /jobs/:id. Queue → append pending → /jobs list. Backend (global): FIFO pending queue; on any finalize with 0 running, promote the oldest pending. Force-Run bypasses the queue. Queue backlog survives restart (promoteNext on startup).
• Modal → page: replace TaskRunnerModal with a /jobs/new route. Right-click task action navigates there instead of mounting the modal. One code path, phone-friendly.
• Deep-link: query params, not hash — /jobs/new?task=convert-video&root=home&path=<dir>&entry=<name>.
• Phone: already has a file browser, so it can pick a target path and POST a job. No blocker.

Plan

Phase 1 — Unified jobs backend

• 1a. Data model. Add mode (pipeline|script|agentic, default pipeline) + exit_code (int) to the jobs table. Log at DATA_PATH/jobs/<id>.log (derived from id). Table/symbol rename pipeline_jobs→jobs is deferred as a cosmetic cleanup — add columns first, keep it working.
• 1b. Execution. Generalize the job manager: startJob takes mode and dispatches — pipeline → existing executePipeline; script → new executeScript (ports task-executor's materializeScript/buildInputEnv/bwrap sandbox/killTree/keepalive, but emits job events + appends stdout/stderr to the log file, finalizes with exit_code). Reuses the manager's viewer/broadcast/replay machinery.
• 1c. Scheduler. runningCount() + FIFO pending promotion on finalize; promoteNext() on startup so a queued backlog resumes.

Phase 2 — REST job API (decouples creation from the socket; enables the phone)

• POST /jobs {taskDirName, inputs, cwd, action} → {jobId} (create + start/queue, background).
• GET /jobs (+?live=1), GET /jobs/:id, GET /jobs/:id/log?offset=, POST /jobs/:id/stop.
• Consolidate the two WebSockets into one /api/tasks/jobs/ws doing only attach/stop/list.

Phase 3 — Frontend

• /jobs/new → NewJobScreen: reads query params, renders the input UI lifted from TaskRunnerModal (TaskInputForm + per-group config + folder probing). Run/Queue per the concurrency UX. JobDetail gains a script branch (terminal output: live attach, or from log when idle). Retire TaskRunnerModal/TaskRunnerDialog/useTaskRunner. Header running-jobs indicator.

Phase 4 — Notifications (later)

• One notifyJobDone(job) hook at finalize → push to the phone app.

Progress

• 1a data model — mode + exit_code columns (schema + applied to DB)
• 1b executeScript + manager dispatch — execute-script.ts (spawn/sandbox/killTree port, log file, abort poll, returns exitCode), process-tree.ts (shared killTree), pipeline-job-manager now dispatches by mode and finalizes script jobs by exit code. Compiles; runtime-untested until a REST caller + restart exist.
• 1c scheduler / queue — enqueueJob(action) (start now / queue behind running), promoteNext() on finalize + startup, getOldestPendingJob, markInterruptedJobs now running-only (pending queue survives restart). startJob kept as a enqueueJob(...,'start') wrapper.
• 2 REST job API — POST /jobs (create script|pipeline, action start/queue), GET /jobs (+?live=1, now returns mode/exitCode/isLive), GET /jobs/:id, GET /jobs/:id/log?offset=, POST /jobs/:id/stop. Router mounted at /jobs and /pipeline-jobs. Needs a restart to deploy; then curl/phone-testable. WS consolidation still pending (old /api/tasks/run/ws + /api/tasks/pipeline/ws still live).
• [~] 3 frontend
  • 3a JobDetail script branch — ScriptJobDetail (terminal, polls GET /jobs/:id/log + status, Stop button) + JobDetailView wrapper routing by mode (pipeline detail renamed PipelineJobDetail). Script jobs are now viewable at /jobs/:id.
  • 3b /jobs/new page — extract the input UI (TaskInputForm + per-group config + folder probing) from TaskRunnerModal into a shared component; Run/Queue → POST /jobs → navigate.
  • 3c FileBrowser task action navigates to /jobs/new?...; retire modal/dialog/useTaskRunner.
  • 3d header running-jobs indicator (GET /jobs?live=1).
• 4 push notifications