wait for Postgres instead of failing startup work once

The failure here was not a crash — it was the opposite, and that is why it would
never have been noticed.

server.tsx fired initQueue() and cleanupOnStartup() as bare promises with a
.catch() that logged. postgres-js connects lazily, so nothing fails at import;
the first query does. If Postgres is a few seconds behind — exactly what a reboot
looks like, with pm2's resurrect racing Docker starting the container — both log
one line during boot and do nothing else.

cleanupOnStartup is the one that matters. It marks jobs interrupted by the
previous shutdown and promotes the queued backlog, so failing it once leaves
those jobs marked running forever: nothing retries, nothing complains again, and
the only thing that would have corrected them has already run.

officerdb now exports waitForDatabase(timeoutMs = 60s): polls `select 1`, logs
once while waiting, resolves true or false rather than throwing. Bounded on
purpose — an unbounded wait holds a process open with no way to tell starting
from hung, and the caller decides what giving up means.

Deliberately NOT awaited before serve(). The listener is already up by that point
and holding it closed would turn a database thirty seconds late into a reverse
proxy answering connection-refused instead of a page. Requests needing the
database fail honestly in the meantime.

The rest of the estate was already fine, which is worth recording so nobody
"fixes" it again: postgres() opens no socket at construction, officer-agent's
resolveOwner is an unbounded 5s retry loop written after this exact failure cost
a session, and opencode, pty, headscale and the anthropic proxy touch no database
at boot at all.

officer-setup's Services section also waits for pg_isready before starting pm2.
Not because starting early breaks anything, but because Verify would then report
a failure that is really a race — and a red line that is usually noise is a red
line people stop reading.

Verified waitForDatabase against a dead port: logged once, returned false after
the timeout, did not throw.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

scripts/setup/officer-setup.sh

@@ -657,6 +657,14 @@ if ! skip; then
     write_ecosystem
     ok "written — $(ecosystem_file)"
 
+    # Starting against a database that is not answering is not fatal — the server
+    # waits and the agent retries forever — but it makes the Verify section below
+    # report a failure that is really just a race, and that is the kind of noise
+    # that teaches people to ignore a red line.
+    if pg_container_running && ! pg_wait_ready 30; then
+      warn "Postgres is not answering — starting anyway, but Verify may report failures"
+    fi
+
     if OUT="$(pm2_start)"; then
       ok "processes started"
       pm2_save >/dev/null 2>&1 && ok "process list saved (survives a pm2 restart)"

src/databases/officer_db/src/db.ts

@@ -10,3 +10,47 @@ if (!POSTGRES_URL) {
 
 const client = postgres(POSTGRES_URL);
 export const db = drizzle(client, { schema });
+
+/**
+ * Block until Postgres answers, or give up after `timeoutMs`.
+ *
+ * `postgres()` above is LAZY — it opens no socket until the first query — so nothing here fails at
+ * import time when the database is not up yet. That is the right default and it has a cost: startup
+ * work that queries fires, fails once, and is swallowed by whatever `.catch()` it was written with.
+ *
+ * That is not hypothetical. `server.tsx` runs `initQueue()` and `cleanupOnStartup()` as
+ * fire-and-forget promises, and the second marks jobs that were interrupted by the last restart and
+ * promotes the queued backlog. If Postgres is a few seconds behind — which is exactly what happens on
+ * a reboot, when pm2's resurrect races Docker starting the container — both log a line and do nothing.
+ * Interrupted jobs then stay marked running forever, because the only thing that would have corrected
+ * them already ran.
+ *
+ * So: wait, rather than try once. Callers that genuinely cannot proceed without the database await
+ * this first; request handlers do not, since by then it is either up or the request fails honestly.
+ *
+ * Bounded, and it resolves false rather than throwing on timeout. An unbounded wait here would hold a
+ * process open with no way to tell whether it is starting or hung, and the caller is better placed to
+ * decide what "gave up" means than this function is.
+ */
+export async function waitForDatabase(timeoutMs = 60_000): Promise<boolean> {
+  const started = Date.now();
+  let announced = false;
+
+  for (;;) {
+    try {
+      await client`select 1`;
+      if (announced) console.log('[db] Postgres is up');
+      return true;
+    } catch (err) {
+      if (Date.now() - started >= timeoutMs) {
+        console.error(`[db] Postgres did not answer within ${Math.round(timeoutMs / 1000)}s:`, err instanceof Error ? err.message : err);
+        return false;
+      }
+      if (!announced) {
+        console.log('[db] waiting for Postgres…');
+        announced = true;
+      }
+      await new Promise((r) => setTimeout(r, 1_000));
+    }
+  }
+}

src/databases/officer_db/src/index.ts

@@ -21,7 +21,7 @@
 // blank line below is the only thing marking the boundary.
 
 // The connection, and drizzle-kit's view of the schema. See ./schema.ts for the core/plugin split.
-export { db } from './db';
+export { db, waitForDatabase } from './db';
 export * as schema from './schema';
 
 // ── Core ─────────────────────────────────────────────────────────────────────────────────────────

src/server.tsx

@@ -382,11 +382,31 @@ import {
   listAllJobs as queueList,
   readJob as queueGet,
 } from './servers/queue/init';
-initQueue().catch((err) => console.error('[queue] failed to initialize:', err));
-
-// Mark any orphaned pipeline jobs from previous server run
 import { cleanupOnStartup } from './servers/api/tasks/pipeline-job-manager';
-cleanupOnStartup().catch((err) => console.error('[pipeline-jobs] startup cleanup failed:', err));
+import { waitForDatabase } from 'officerdb';
+
+// ── Startup work that needs the database, and waits for it ──
+//
+// Both of these query Postgres, and both used to be fired as bare promises with a `.catch()` that
+// logged. That is fine when the database is up and silently wrong when it is not — which is precisely
+// what a reboot looks like, with pm2's resurrect racing Docker starting the Postgres container.
+//
+// `cleanupOnStartup` is the one that matters: it marks jobs interrupted by the previous shutdown and
+// promotes the queued backlog. Fail it once and those jobs stay marked running forever, because the
+// only thing that would have corrected them has already run. Nothing retries and nothing complains
+// again — the log line scrolls past during boot and the jobs are simply stuck.
+//
+// Deliberately NOT awaited before serve(): the HTTP listener is already up by here, and holding it
+// closed for a minute would turn a database that is thirty seconds late into a reverse proxy serving
+// connection-refused instead of a page. Requests that need the database fail honestly in the meantime.
+void (async () => {
+  if (!(await waitForDatabase())) {
+    console.error('[startup] skipping queue init and pipeline cleanup — the database never answered');
+    return;
+  }
+  await initQueue().catch((err) => console.error('[queue] failed to initialize:', err));
+  await cleanupOnStartup().catch((err) => console.error('[pipeline-jobs] startup cleanup failed:', err));
+})();
 
 // PulseAudio and the `virtual_out` sink used to be set up here, at every boot of a process that has no
 // audio responsibilities. They belong to the music sidecar, which owns both cliamp halves now