The server went down. It had not: it was STOPPED — eleven threads in state T, port 9000 still listening with connections queueing behind a process that would never accept them. Indistinguishable from a crash unless you look, which is why it took a terminal to see. `kill -CONT` brought it back with nothing lost. I did it, with one missing line. `restartBackgroundTask` opened the job's output file to hand the child a descriptor and never closed it, so the PLATFORM became a holder of that file. `pidsHoldingFile` asks `fuser` who holds it — that is how Stop finds a job with no recorded pid — so the platform was in the answer. The next Pause sent SIGSTOP to it. Every part worked as designed. Pause froze exactly what was holding the file, and one of those things was the thing doing the freezing. Two fixes, and the second matters more than the first: **Close the descriptor.** The child gets its own duplicate at spawn, so closing costs it nothing. That removes this instance. **Never signal our own pids, whatever `fuser` says.** `pidsHoldingFile` now filters `process.pid` and `process.ppid`. The leak is fixed; this is what makes the next one survivable. Nothing that manages a job should be able to name the process doing the managing — a rule worth having independently of the bug that suggested it. Verified: after a restart this process holds the output file zero times, `fuser` returns only the spawned shell and its child, and our own pid is filtered even when present. tsgo clean, frontend builds, 901 pass / 7 fail unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
239 lines
10 KiB
TypeScript
239 lines
10 KiB
TypeScript
import { closeSync, openSync } from 'node:fs';
|
|
import { basename, dirname, join } from 'node:path';
|
|
import { runAsArgv } from '../../os-user';
|
|
import { readTextAs } from '../../read-as-user';
|
|
import { findTaskOutput, pidsHoldingFile, type ChatIdentity } from './claude-sessions';
|
|
|
|
/**
|
|
* Run a background job's command again, without asking the agent.
|
|
*
|
|
* ── What this owns, and what it does not ──
|
|
*
|
|
* A background task normally belongs entirely to the harness: it spawns the process, watches it, and emits
|
|
* `task:started` and `task:notification`. Everything downstream — the tray, the log view, reconnects — is
|
|
* built on exactly those two events.
|
|
*
|
|
* A restart cannot be that. The agent is not involved, so nothing would emit either event, and the harness
|
|
* stops watching the moment its own child dies. So the platform takes over supervision of the process it
|
|
* spawned: it emits the same two events for it, and watches for the exit itself.
|
|
*
|
|
* That is a second owner for the same kind of thing, and it is a real cost — accepted deliberately, so
|
|
* that restarting a dev server is a button rather than a sentence typed at an agent that may be mid-turn.
|
|
* The seam is contained here: nothing else learns that a task can have two kinds of parent.
|
|
*
|
|
* ── The output file is the anchor ──
|
|
*
|
|
* The respawned process appends to the SAME `<task-id>.output` the harness created. That is what keeps
|
|
* everything else working untouched: the log view reads that file, and Stop finds the process by asking
|
|
* which pids hold it open. A new file would have meant teaching both about a second location.
|
|
*/
|
|
|
|
/** In-flight restarts, so a second click stops the first rather than racing it. */
|
|
const restarted = new Map<string, { proc: Bun.Subprocess }>();
|
|
|
|
export type RestartResult = { ok: boolean; command?: string; reason?: string };
|
|
|
|
/** A command and the directory it meant something in. The second half is not optional in practice. */
|
|
export type TaskCommand = { command: string; cwd: string | null };
|
|
|
|
/**
|
|
* The command that started a task, recovered by joining the transcript back to itself.
|
|
*
|
|
* The harness records the tool call (`tool_use`, carrying `command`) and, separately, its result — which
|
|
* is the text `Command running in background with ID: <task id>`. Neither alone is enough; together they
|
|
* are a join on `tool_use_id`, and that is the only place the command survives.
|
|
*
|
|
* The transcript is found from the OUTPUT PATH rather than by searching: a task file lives at
|
|
* `<tmp>/claude-<uid>/<project-slug>/<session-uuid>/tasks/<id>.output`, so the slug and session name the
|
|
* transcript exactly. Guessing among a member's transcripts would be both slower and occasionally wrong.
|
|
*/
|
|
export async function commandForTask(who: ChatIdentity, taskId: string): Promise<TaskCommand | null> {
|
|
const output = findTaskOutput(who, taskId);
|
|
if (!output || !output.endsWith('.output')) return null;
|
|
|
|
const sessionDir = dirname(dirname(output));
|
|
const transcript = join(
|
|
who.home,
|
|
'.claude',
|
|
'projects',
|
|
basename(dirname(sessionDir)),
|
|
`${basename(sessionDir)}.jsonl`,
|
|
);
|
|
|
|
let text: string;
|
|
try {
|
|
text = readTextAs(who.osUser, transcript);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
// The cwd is read from the SAME entry as the tool call, not from anywhere else. `bun dev` means one
|
|
// thing in a project and something entirely different in the platform's own directory — see the note on
|
|
// `restartBackgroundTask`.
|
|
const commands = new Map<string, TaskCommand>();
|
|
let toolUseId: string | null = null;
|
|
|
|
for (const line of text.split('\n')) {
|
|
if (!line) continue;
|
|
let entry: unknown;
|
|
try {
|
|
entry = JSON.parse(line);
|
|
} catch {
|
|
continue;
|
|
}
|
|
const record = entry as { message?: { content?: unknown }; cwd?: unknown };
|
|
const content = record?.message?.content;
|
|
if (!Array.isArray(content)) continue;
|
|
|
|
for (const block of content as Record<string, unknown>[]) {
|
|
if (block.type === 'tool_use' && block.name === 'Bash') {
|
|
const command = (block.input as { command?: unknown } | undefined)?.command;
|
|
if (typeof block.id === 'string' && typeof command === 'string') {
|
|
commands.set(block.id, { command, cwd: typeof record.cwd === 'string' ? record.cwd : null });
|
|
}
|
|
}
|
|
if (block.type === 'tool_result') {
|
|
const body = typeof block.content === 'string' ? block.content : JSON.stringify(block.content ?? '');
|
|
// Both phrasings the harness produces carry the id; this is the one that names a shell job.
|
|
if (body.includes(`background with ID: ${taskId}`) && typeof block.tool_use_id === 'string') {
|
|
toolUseId = block.tool_use_id;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
return toolUseId ? (commands.get(toolUseId) ?? null) : null;
|
|
}
|
|
|
|
/**
|
|
* Stop whatever is running for this task and start its command again.
|
|
*
|
|
* `onExit` is how the caller emits `task:notification` — this module deliberately knows nothing about
|
|
* websockets or sessions.
|
|
*/
|
|
export async function restartBackgroundTask(
|
|
who: ChatIdentity,
|
|
taskId: string,
|
|
onExit: (code: number) => void,
|
|
): Promise<RestartResult> {
|
|
const output = findTaskOutput(who, taskId);
|
|
if (!output) return { ok: false, reason: 'no such task' };
|
|
|
|
const recovered = await commandForTask(who, taskId);
|
|
if (!recovered) {
|
|
// Honest about which half failed: the file is there, the command is not recoverable. That happens when
|
|
// the transcript has been pruned, and no amount of retrying will change it.
|
|
return { ok: false, reason: 'cannot tell what command started this job — its transcript is gone' };
|
|
}
|
|
|
|
/**
|
|
* No directory, no restart.
|
|
*
|
|
* This refuses rather than falling back to the server's own cwd, and the reason is that the fallback
|
|
* already happened: a `bun dev` belonging to a project in `~/dockers/officer-atelier` was respawned
|
|
* without a cwd, inherited the PLATFORM's directory, and ran the platform's own `bun dev`. A second
|
|
* server bound port 9000 alongside the real one — Bun allows it through SO_REUSEPORT — and requests
|
|
* were split between two instances at random.
|
|
*
|
|
* A command is only meaningful with the directory it was written for, and guessing produced the worst
|
|
* outcome available: not a failure, but a different program running successfully.
|
|
*/
|
|
const { command, cwd } = recovered;
|
|
if (!cwd) {
|
|
return { ok: false, reason: 'cannot tell which directory this job ran in — refusing to guess' };
|
|
}
|
|
|
|
// Ours first, then anything else holding the file. A second click must replace the first restart, not
|
|
// leave two dev servers fighting over a port.
|
|
const previous = restarted.get(taskId);
|
|
if (previous) {
|
|
try {
|
|
previous.proc.kill();
|
|
} catch {
|
|
/* already dead */
|
|
}
|
|
restarted.delete(taskId);
|
|
}
|
|
|
|
for (const pid of await pidsHoldingFile(who, output)) {
|
|
const argv = who.osUser ? runAsArgv(who.osUser, ['kill', '-TERM', String(pid)]) : null;
|
|
try {
|
|
if (argv) Bun.spawn(argv, { stdout: 'ignore', stderr: 'ignore' });
|
|
else process.kill(pid, 'SIGTERM');
|
|
} catch {
|
|
/* gone between listing and killing */
|
|
}
|
|
}
|
|
// A dev server holding a port needs a moment to release it, and the whole point of a restart is that the
|
|
// new one binds. Not a fix for a process that ignores SIGTERM — that is what the second click is for.
|
|
await Bun.sleep(300);
|
|
|
|
// Appended, never truncated: the log view is reading this file, and a restart is a new chapter in a job's
|
|
// life rather than a new job.
|
|
// The fd is CLOSED immediately after the spawn, and that one line is load-bearing.
|
|
//
|
|
// Leaving it open made the platform a holder of the job's output file — and `pidsHoldingFile` asks
|
|
// `fuser` who holds it. So the next Pause sent `SIGSTOP` to the platform itself: eleven threads
|
|
// stopped, port 9000 still listening with connections queueing behind it and nobody accepting. The
|
|
// server was not down, it was frozen, which looks identical from outside and is why it took a terminal
|
|
// to see. `kill -CONT` brought it back untouched.
|
|
//
|
|
// The child gets its own duplicate of the descriptor at spawn, so closing here costs it nothing.
|
|
const fd = openSync(output, 'a');
|
|
const argv = who.osUser ? runAsArgv(who.osUser, ['sh', '-c', command]) : ['sh', '-c', command];
|
|
const proc = Bun.spawn(argv, { cwd, stdout: fd, stderr: fd, stdin: 'ignore' });
|
|
closeSync(fd);
|
|
restarted.set(taskId, { proc });
|
|
|
|
void proc.exited.then((code) => {
|
|
// Only report the exit of the process we are still tracking — a later restart owns the task by then.
|
|
if (restarted.get(taskId)?.proc === proc) {
|
|
restarted.delete(taskId);
|
|
onExit(code ?? 0);
|
|
}
|
|
});
|
|
|
|
return { ok: true, command };
|
|
}
|
|
|
|
/** Whether the platform (rather than the harness) is currently supervising this task. */
|
|
export const isRestarted = (taskId: string): boolean => restarted.has(taskId);
|
|
|
|
/**
|
|
* Freeze or thaw a job, without ending it.
|
|
*
|
|
* `SIGSTOP` and `SIGCONT`, which is the only pair that means "stop doing things and keep everything" — the
|
|
* process keeps its memory, its open files and its bound ports, and picks up mid-instruction.
|
|
*
|
|
* Worth knowing before pressing it: a paused server does not REFUSE connections, it stops answering them.
|
|
* A browser pointed at it hangs rather than erroring, and an HMR client may give up and reconnect on
|
|
* resume. That is inherent to what pausing is, not a shortcoming of this implementation.
|
|
*
|
|
* Signals go to every pid holding the output file — for a shell job that is the shell and whatever it
|
|
* spawned, and freezing only one of them would be worse than not freezing at all.
|
|
*/
|
|
export async function pauseBackgroundTask(
|
|
who: ChatIdentity,
|
|
taskId: string,
|
|
paused: boolean,
|
|
): Promise<{ ok: boolean; reason?: string }> {
|
|
const output = findTaskOutput(who, taskId);
|
|
if (!output) return { ok: false, reason: 'no such task' };
|
|
|
|
const pids = await pidsHoldingFile(who, output);
|
|
if (pids.length === 0) return { ok: false, reason: 'nothing is running for this job' };
|
|
|
|
const signal = paused ? '-STOP' : '-CONT';
|
|
for (const pid of pids) {
|
|
const argv = who.osUser ? runAsArgv(who.osUser, ['kill', signal, String(pid)]) : null;
|
|
try {
|
|
if (argv) Bun.spawn(argv, { stdout: 'ignore', stderr: 'ignore' });
|
|
else process.kill(pid, paused ? 'SIGSTOP' : 'SIGCONT');
|
|
} catch {
|
|
// Gone between listing and signalling. Nothing to do, and nothing to report: the caller asked for a
|
|
// state, and "not running" satisfies "paused" closely enough to not be worth an error.
|
|
}
|
|
}
|
|
return { ok: true };
|
|
}
|