wait for Postgres instead of failing startup work once

The failure here was not a crash — it was the opposite, and that is why it would
never have been noticed.

server.tsx fired initQueue() and cleanupOnStartup() as bare promises with a
.catch() that logged. postgres-js connects lazily, so nothing fails at import;
the first query does. If Postgres is a few seconds behind — exactly what a reboot
looks like, with pm2's resurrect racing Docker starting the container — both log
one line during boot and do nothing else.

cleanupOnStartup is the one that matters. It marks jobs interrupted by the
previous shutdown and promotes the queued backlog, so failing it once leaves
those jobs marked running forever: nothing retries, nothing complains again, and
the only thing that would have corrected them has already run.

officerdb now exports waitForDatabase(timeoutMs = 60s): polls `select 1`, logs
once while waiting, resolves true or false rather than throwing. Bounded on
purpose — an unbounded wait holds a process open with no way to tell starting
from hung, and the caller decides what giving up means.

Deliberately NOT awaited before serve(). The listener is already up by that point
and holding it closed would turn a database thirty seconds late into a reverse
proxy answering connection-refused instead of a page. Requests needing the
database fail honestly in the meantime.

The rest of the estate was already fine, which is worth recording so nobody
"fixes" it again: postgres() opens no socket at construction, officer-agent's
resolveOwner is an unbounded 5s retry loop written after this exact failure cost
a session, and opencode, pty, headscale and the anthropic proxy touch no database
at boot at all.

officer-setup's Services section also waits for pg_isready before starting pm2.
Not because starting early breaks anything, but because Verify would then report
a failure that is really a race — and a red line that is usually noise is a red
line people stop reading.

Verified waitForDatabase against a dead port: logged once, returned false after
the timeout, did not throw.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-13 02:38:36 +00:00
co-authored by Claude Opus 5
parent 62cbf510cb
commit 085afe7604
4 changed files with 77 additions and 5 deletions
+8
View File
@@ -657,6 +657,14 @@ if ! skip; then
write_ecosystem
ok "written — $(ecosystem_file)"
# Starting against a database that is not answering is not fatal — the server
# waits and the agent retries forever — but it makes the Verify section below
# report a failure that is really just a race, and that is the kind of noise
# that teaches people to ignore a red line.
if pg_container_running && ! pg_wait_ready 30; then
warn "Postgres is not answering — starting anyway, but Verify may report failures"
fi
if OUT="$(pm2_start)"; then
ok "processes started"
pm2_save >/dev/null 2>&1 && ok "process list saved (survives a pm2 restart)"