wait for Postgres instead of failing startup work once

The failure here was not a crash — it was the opposite, and that is why it would
never have been noticed.

server.tsx fired initQueue() and cleanupOnStartup() as bare promises with a
.catch() that logged. postgres-js connects lazily, so nothing fails at import;
the first query does. If Postgres is a few seconds behind — exactly what a reboot
looks like, with pm2's resurrect racing Docker starting the container — both log
one line during boot and do nothing else.

cleanupOnStartup is the one that matters. It marks jobs interrupted by the
previous shutdown and promotes the queued backlog, so failing it once leaves
those jobs marked running forever: nothing retries, nothing complains again, and
the only thing that would have corrected them has already run.

officerdb now exports waitForDatabase(timeoutMs = 60s): polls `select 1`, logs
once while waiting, resolves true or false rather than throwing. Bounded on
purpose — an unbounded wait holds a process open with no way to tell starting
from hung, and the caller decides what giving up means.

Deliberately NOT awaited before serve(). The listener is already up by that point
and holding it closed would turn a database thirty seconds late into a reverse
proxy answering connection-refused instead of a page. Requests needing the
database fail honestly in the meantime.

The rest of the estate was already fine, which is worth recording so nobody
"fixes" it again: postgres() opens no socket at construction, officer-agent's
resolveOwner is an unbounded 5s retry loop written after this exact failure cost
a session, and opencode, pty, headscale and the anthropic proxy touch no database
at boot at all.

officer-setup's Services section also waits for pg_isready before starting pm2.
Not because starting early breaks anything, but because Verify would then report
a failure that is really a race — and a red line that is usually noise is a red
line people stop reading.

Verified waitForDatabase against a dead port: logged once, returned false after
the timeout, did not throw.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/server.tsx

@@ -382,11 +382,31 @@ import {
   listAllJobs as queueList,
   readJob as queueGet,
 } from './servers/queue/init';
-initQueue().catch((err) => console.error('[queue] failed to initialize:', err));
-
-// Mark any orphaned pipeline jobs from previous server run
 import { cleanupOnStartup } from './servers/api/tasks/pipeline-job-manager';
-cleanupOnStartup().catch((err) => console.error('[pipeline-jobs] startup cleanup failed:', err));
+import { waitForDatabase } from 'officerdb';
+
+// ── Startup work that needs the database, and waits for it ──
+//
+// Both of these query Postgres, and both used to be fired as bare promises with a `.catch()` that
+// logged. That is fine when the database is up and silently wrong when it is not — which is precisely
+// what a reboot looks like, with pm2's resurrect racing Docker starting the Postgres container.
+//
+// `cleanupOnStartup` is the one that matters: it marks jobs interrupted by the previous shutdown and
+// promotes the queued backlog. Fail it once and those jobs stay marked running forever, because the
+// only thing that would have corrected them has already run. Nothing retries and nothing complains
+// again — the log line scrolls past during boot and the jobs are simply stuck.
+//
+// Deliberately NOT awaited before serve(): the HTTP listener is already up by here, and holding it
+// closed for a minute would turn a database that is thirty seconds late into a reverse proxy serving
+// connection-refused instead of a page. Requests that need the database fail honestly in the meantime.
+void (async () => {
+  if (!(await waitForDatabase())) {
+    console.error('[startup] skipping queue init and pipeline cleanup — the database never answered');
+    return;
+  }
+  await initQueue().catch((err) => console.error('[queue] failed to initialize:', err));
+  await cleanupOnStartup().catch((err) => console.error('[pipeline-jobs] startup cleanup failed:', err));
+})();
 
 // PulseAudio and the `virtual_out` sink used to be set up here, at every boot of a process that has no
 // audio responsibilities. They belong to the music sidecar, which owns both cliamp halves now