wait for Postgres instead of failing startup work once
The failure here was not a crash — it was the opposite, and that is why it would never have been noticed. server.tsx fired initQueue() and cleanupOnStartup() as bare promises with a .catch() that logged. postgres-js connects lazily, so nothing fails at import; the first query does. If Postgres is a few seconds behind — exactly what a reboot looks like, with pm2's resurrect racing Docker starting the container — both log one line during boot and do nothing else. cleanupOnStartup is the one that matters. It marks jobs interrupted by the previous shutdown and promotes the queued backlog, so failing it once leaves those jobs marked running forever: nothing retries, nothing complains again, and the only thing that would have corrected them has already run. officerdb now exports waitForDatabase(timeoutMs = 60s): polls `select 1`, logs once while waiting, resolves true or false rather than throwing. Bounded on purpose — an unbounded wait holds a process open with no way to tell starting from hung, and the caller decides what giving up means. Deliberately NOT awaited before serve(). The listener is already up by that point and holding it closed would turn a database thirty seconds late into a reverse proxy answering connection-refused instead of a page. Requests needing the database fail honestly in the meantime. The rest of the estate was already fine, which is worth recording so nobody "fixes" it again: postgres() opens no socket at construction, officer-agent's resolveOwner is an unbounded 5s retry loop written after this exact failure cost a session, and opencode, pty, headscale and the anthropic proxy touch no database at boot at all. officer-setup's Services section also waits for pg_isready before starting pm2. Not because starting early breaks anything, but because Verify would then report a failure that is really a race — and a red line that is usually noise is a red line people stop reading. Verified waitForDatabase against a dead port: logged once, returned false after the timeout, did not throw. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -657,6 +657,14 @@ if ! skip; then
|
||||
write_ecosystem
|
||||
ok "written — $(ecosystem_file)"
|
||||
|
||||
# Starting against a database that is not answering is not fatal — the server
|
||||
# waits and the agent retries forever — but it makes the Verify section below
|
||||
# report a failure that is really just a race, and that is the kind of noise
|
||||
# that teaches people to ignore a red line.
|
||||
if pg_container_running && ! pg_wait_ready 30; then
|
||||
warn "Postgres is not answering — starting anyway, but Verify may report failures"
|
||||
fi
|
||||
|
||||
if OUT="$(pm2_start)"; then
|
||||
ok "processes started"
|
||||
pm2_save >/dev/null 2>&1 && ok "process list saved (survives a pm2 restart)"
|
||||
|
||||
Reference in New Issue
Block a user