"generating" is not "has an outstanding background task"

the guard I added yesterday to stop the reviver interrupting a commander
mid-work disabled the feature entirely, and it took a real case to see it.

it asked collectRunningTasks, which counts BACKGROUND TASKS. a commander's
watcher IS a background task, and it stays outstanding until it reports a
terminal state — which a killed process never does. so every commander with an
armed watcher had one outstanding by definition, the guard read "busy" forever,
and the reviver could never wake the one thing it exists for.

found on the deploy commander tonight: loop dead, pid file saying so, and the
task "Platform Deployer watcher — polls origin for deploy/* tags" still marked
running from its own revival six hours earlier. permanently true.

this is the shape I criticised in the nudger yesterday — a guard that could
only ever be right by accident — written by me a few hours later. the right
question was always whether the AGENT is mid-turn, which the sidecar answers
directly and which the nudger was already asking before I removed it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Officer-Session: 656f5c85-e49a-4f1f-b9f3-042a04b4d043
This commit is contained in:
2026-08-26 03:36:39 +00:00
co-authored by Claude Opus 5
parent a7b95a7bd1
commit a2102c5b46
+25 -10
View File
@@ -2,9 +2,8 @@ import { listAllCommanders, getUserById } from 'officerdb';
import { getCommanderStateDir } from '@@/data-path';
import { existsSync, readFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { startClaudeTurn, COMMANDER_DEFAULT_MODEL, collectRunningTasks } from '../chat/websocket';
import { getChatEventsSince } from 'officerdb';
import type { ServerMessage } from '../chat/types';
import { startClaudeTurn, COMMANDER_DEFAULT_MODEL } from '../chat/websocket';
import * as sidecar from '@@/sidecar-registry';
import { clearLoopPid, liveLoopPids } from './spawned';
import { getNotifyServerUrl, notifyProducerHeaders } from '../notify/router';
@@ -129,17 +128,33 @@ function clearAttempts(email: string, dirName: string): void {
* answer to "is this watching", not that a fire was observed being misread. Either way the window is real:
* between a watcher exiting and its agent re-arming, the file names a process that is gone.
*
* Hence `isGenerating`: if a turn is running, the exit is being handled and the re-arm is the last thing
* that turn will do. Waking then is not just wasteful, it is wrong — it would arrive mid-work carrying an
* instruction the agent has no reason to obey.
* Hence the generating check: if a TURN is running, the exit is being handled and the re-arm is the last
* thing that turn will do. Waking then is not just wasteful, it is wrong — it would arrive mid-work
* carrying an instruction the agent has no reason to obey.
*
* ── "generating" is not "has an outstanding background task" ──
*
* This asked `collectRunningTasks` until 2026-08-26, and that was wrong in a way that disabled the whole
* feature. Those are BACKGROUND TASKS, and **a commander's watcher is itself a background task** — one that
* stays outstanding until it reports a terminal state, which a killed process never does. So every
* commander with an armed watcher had one outstanding by definition, the check returned "busy" forever, and
* the reviver could never wake the one thing it exists for.
*
* Found on the deploy commander: its loop was dead, its pid file said so, and the task
* "Platform Deployer watcher — polls origin for deploy/* tags" was still marked running from the revival
* six hours earlier. The guard was permanently true.
*
* The right question is whether the AGENT is mid-turn, which the sidecar answers directly.
*/
async function watchDied(email: string, dirName: string, sessionId: string): Promise<boolean> {
async function watchDied(email: string, dirName: string, sessionId: string, userId: number): Promise<boolean> {
if (!existsSync(join(getCommanderStateDir(email, dirName), PID_FILE))) return false;
if (liveLoopPids(email, dirName).length > 0) return false;
try {
const events = await getChatEventsSince(sessionId, 0);
if (collectRunningTasks(events.map((e) => e.event as ServerMessage)).length > 0) return false;
const live = await sidecar.listLiveClaudeSessions(userId);
const mine = live.find((s) => s.sessionKey === sessionId || s.claudeSessionId === sessionId);
// Absent means idle or collected, and both are fine — `fire` resumes by transcript id.
if (mine?.isGenerating) return false;
} catch {
// Cannot tell whether it is working, so assume it is. A missed revival is reported next tick; a wake
// delivered into a live turn is not recoverable by waiting.
@@ -271,7 +286,7 @@ export function startWatcherRevive(): void {
for (const commander of await listAllCommanders()) {
const user = await getUserById(commander.userId);
if (!user) continue;
if (!(await watchDied(user.email, commander.dirName, commander.sessionId))) {
if (!(await watchDied(user.email, commander.dirName, commander.sessionId, commander.userId))) {
// Healthy, stopped, or mid-turn. A live loop means the last wake worked, so the count resets.
if (liveLoopPids(user.email, commander.dirName).length > 0) clearAttempts(user.email, commander.dirName);
continue;