officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

don't let the agent die when postgres blinks

Browse Source 
resolveOwner already retried forever — but only when the query SUCCEEDED and returned nothing, which is
a fresh install waiting on bootstrap. A query that THREW escaped the function, rejected the top-level
await and exited the process, into exactly the PM2 restart loop its own comment says it exists to avoid.
So any Postgres restart (57P03 'the database system is starting up') or moment of unavailability killed
every live agent session on the machine and spun the sidecar until the database answered.

That is what took a session down on 2026-08-10, and why this process showed 468 restarts against 0 for
every peer that starts without needing the database.

The loop now catches as well as checks. Still retries forever, matching the case beside it: a database
coming back is a matter of time, and an agent that gave up would need a human to notice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-10 01:09:48 +01:00
co-authored by Claude Opus 5
parent 5e141afa18
commit 975673a9a6
1 changed files with 25 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/servers/sidecar/claude/user-instance.ts
+21
View File
@@ -25,13 +25,34 @@ import { getUserByEmail, getOwnerUser, getEmailAccounts } from 'officerdb';
// "The owner" is not a simplification that multi-user will later invalidate. `chat` is an `execution` // "The owner" is not a simplification that multi-user will later invalidate. `chat` is an `execution`
// capability (capabilities/registry.ts) and is never grantable at any level, so no account other than // capability (capabilities/registry.ts) and is never grantable at any level, so no account other than
// the owner can ever reach this sidecar, however many accounts exist. // the owner can ever reach this sidecar, however many accounts exist.
//
// ── Why this retries rather than throws ──
//
// There are two ways this can fail and it used to survive only one of them. A query that SUCCEEDS and
// returns nothing is a fresh install waiting on POST /auth/bootstrap, and was handled. A query that
// THROWS — Postgres restarting (`57P03: the database system is starting up`), or not up yet
// (`ECONNREFUSED`) — escaped this function, rejected the top-level await, and exited the process into
// exactly the PM2 restart loop the comment below says it exists to avoid. It spins until the database
// answers, and every live agent session dies with the first crash.
//
// That is not hypothetical: it cost a session on 2026-08-10, and the restart counter on this sidecar
// read 468 against 0 for every peer that starts without needing the database.
//
// Retrying forever rather than failing fast is deliberate, and matches the case below it: the database
// coming back is a matter of time, and an agent that gave up would need a human to notice and restart it.
async function resolveOwner() { async function resolveOwner() {
const explicit = process.env.CLAUDE_USER_EMAIL?.trim(); const explicit = process.env.CLAUDE_USER_EMAIL?.trim();
for (;;) { for (;;) {
try {
const user = explicit ? await getUserByEmail(explicit) : await getOwnerUser(); const user = explicit ? await getUserByEmail(explicit) : await getOwnerUser();
if (user) return user; if (user) return user;
// Fresh install: wait for POST /auth/bootstrap instead of exiting into a PM2 restart loop. // Fresh install: wait for POST /auth/bootstrap instead of exiting into a PM2 restart loop.
console.log(`[agent] no ${explicit ? `user "${explicit}"` : 'owner account'} yet — retrying in 5s`); console.log(`[agent] no ${explicit ? `user "${explicit}"` : 'owner account'} yet — retrying in 5s`);
} catch (err) {
console.log(
`[agent] database not reachable yet (${err instanceof Error ? err.message : String(err)}) — retrying in 5s`,
);
}
await Bun.sleep(5_000); await Bun.sleep(5_000);
} }
} }