officer-url.mjs is now the only file in the tree that touches process.env.PORT.
Twenty-two others read it and supplied their own default; a value with
twenty-two sources is not configuration, it is twenty-two things to keep in sync,
and they had already drifted three ways.
It throws when PORT is unset rather than guessing. A default only covers the case
where .env was never loaded — which is not a machine anyone wants running,
because POSTGRES_URL is missing in the same breath. What the default bought was a
process that starts, binds somewhere unexpected, and fails later for a reason
that does not name the cause. Same posture as jwt.ts with JWT_SECRET.
It is .mjs, not .ts, and that is the whole reason this could be one file. pm2
launches officer-pty with node (ecosystem.config.cjs) and everything else with
bun; node cannot import TypeScript, so a .ts module would have left the pty
sidecar holding the only surviving copy of the default — precisely the thing
being removed. allowJs is already on, so the TS callers still get types. Verified
both runtimes import it, and that PUBLIC_URL-style overrides still work.
It also exports API_URL and OFFICER_API_URL, because nineteen sidecars were
independently building `ws://127.0.0.1:${PORT}` and two more were building the
http form. Those are one listener described in two protocols — no sidecar binds
anything — so they belong beside the port rather than being rediscovered per
file.
server.tsx now takes PORT as a number, so Number(PORT) at the serve site is gone.
Not typechecked (empty node_modules, frozen installs). Every edited file parses
under `bun build --no-bundle`; node and bun both load the new module; the unset
and non-numeric paths were exercised; the pm2 profile still loads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every PORT fallback in the tree now says 9000. There were three answers to one
question, each defensible where it was written and none of them visible from the
others:
server.tsx and 19 sidecars 5000 a default from before there was an installer
user-instance.ts 9010 what scripts/setup-old/setup.sh really wrote
.env.example 9000 what we told people to write
5000 goes first because macOS binds it — AirPlay Receiver has owned it since
Monterey, so a dev server there fails to bind or gets shadowed by something that
answers.
All 22 sites moved together, which is the point. Changing the app alone would have
turned a consistent-but-wrong default into a split one: the app on 9000 while
nineteen sidecars still dialled 5000.
9010 was the interesting one. It was the only value that ever matched a real
machine, because it is what the old installer wrote — and it was in the single
file whose disagreement would have broken chat alone, with nothing else looking
wrong. Its own comment records the same bug being fixed once already, within the
file, by a change that left it disagreeing with everything outside it.
Note what these defaults actually are: the sidecars bind nothing. user-instance.ts
has no listener at all — it builds ws:// and http:// URLs that both address the
app's single listener. So every one of these numbers is a guess at where the app
is, for a value that .env always supplies. Worth removing rather than aligning,
which is a separate change.
Not typechecked (empty node_modules, frozen installs). Every edited file parses
under `bun build --no-bundle`; the pm2 profile loads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit migrated only when SQLite was completely empty, on the assumption that a non-empty
file is an authoritative one. A real account disproved that within minutes of it landing.
The older Gmail backfill had already written SOME keys into SQLite — last_sync_at and the uidvalidity
set — and never the imap_lastuid ones. So the file was non-empty and half-migrated at the same time,
all-or-nothing skipped the migration, and nine imap_lastuid keys stayed only in Postgres. A missing
lastuid makes the next sync refetch that folder from UID 1: on the mailbox this was found on, 18,755
messages and 6.9 GB.
Now merged per key with the file always winning a conflict. That keeps the property all-or-nothing was
protecting — a restored older emails.db still overrides a newer Postgres row for every key it has, so it
cannot be advanced past mail it does not contain — and adds the keys the file never had, which are
exactly the ones whose absence is expensive.
Verified against the live account: all 22 Postgres keys present afterwards, imap_lastuid:INBOX restored
to 208407, and last_sync_at left at the file's older value, so it re-checks a fortnight rather than
skipping it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The messages were in emails.db and the position — last_sync_at, and per-folder uidvalidity/lastuid —
was a jsonb column on email_accounts in Postgres. Two stores for one fact, with an edge that only shows
up when you try to move a mailbox to another machine.
The expensive part of an email account is the first sync: hours of IMAP for a large mailbox, which is
exactly why "copy emails.db to the new server" is the obvious way to bring one across. With the position
in Postgres that silently does not work — the new server's column is empty, !last_sync_at says first
sync, and the whole mailbox downloads again on top of the one just restored.
The other direction is quieter and worse. Restore an OLDER emails.db while Postgres holds a NEWER
position and the sidecar skips every message between the two, permanently, because nothing looks below
lastuid again. Re-syncing is slow; skipping mail is data loss nobody notices.
Not a new idea — the Gmail path already read SQLite and fell back to Postgres, backfilling so the
fallback was taken once. Only the IMAP path had not followed. This extracts that pattern so both use one
copy, and unifies the isFirstSync fork in accounts.ts, which is how the two drifted apart to begin with.
The file wins over Postgres, always, and only migrates when it holds nothing at all. Topping up a
partial position from Postgres would reintroduce precisely the divergence this removes.
email_accounts.sync_meta is kept and marked legacy rather than dropped: it is the one-time backfill
source for every account created before this, and dropping it would strand any that has not synced
since. Nothing writes to it now.
11 tests on the migration, aimed at both expensive failures — migrating when we should not, and failing
to migrate an account that predates the change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adding an email account failed with a bare "failed to add account" toast. three
defects stacked, each hiding the next.
the email sidecar's http.ts reconstructs what the platform's middleware used to
provide, but only did two of three — bodyParser was never remounted, so every
write route read ctx.get('body') as undefined and POST /accounts threw on
body.provider before ever reaching the credentials.
its onError then read `.status` off the thrown custom-error, which carries
`statusCode`. every deliberate 4xx fell through to the 500 branch and had its
message replaced with "internal error", so a rejected IMAP login and a genuine
crash looked identical. it also answered JSON where the rest of the api answers
errors as plain text. now mirrors hono.ts's handler rather than inventing a
second shape.
useClient threw a plain object, so the ~33 sites narrowing with
`err instanceof Error ? err.message : <fallback>` always took the fallback and
discarded the server's message. now throws an ApiError subclass keeping both
status and message, so those sites start surfacing real errors.
only email reads ctx.get('body'); every other sidecar is a pure proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three existed to drive the platform from a chat app. The phone app does that now, so
they are dead weight — three bot gateways, three command parsers, account pairing, admin
config screens and bot tokens sitting in the database.
Gone entirely: telegram/ and whatsapp/, discord/'s bot and command handler, the shared
channel plumbing they were the only users of (pairing.ts, send-and-await.ts, types.ts,
routes.ts and its 20 config/pairing/status endpoints), the six settings components, and
their sections in the integrations screen.
What Discord keeps is the one piece worth keeping — pushing a message out — as
notify/discord.ts, configured by DISCORD_WEBHOOK_URL in the env. No UI, no pairing, no
stored credential, and it never throws: a notification that fails to send is logged and
dropped. Unset means notifications are silently skipped, which is the default state.
Kept deliberately: send-claude-code.ts and send-opencode.ts. They live under channels/ but
have nothing to do with chat apps — they are how /chat and the pipeline executor drive an
agent turn.
Also drops four dependencies with no remaining importer (discord.js,
node-telegram-bot-api, whatsapp-web.js, qrcode), and the /sync-now route added to the email
sidecar an hour ago, whose only consumer was the channel handlers.
The "Channel Models" settings section stays, with its description corrected — it is keyed
off the general access policy rather than anything channel-specific, so it governs non-owner
accounts, not chat apps. Whether that whole class still earns its place is the open
question already noted against origin-validation.
Not touched: the telegram, whatsapp and discord rows in server_integrations, which still
hold their bot tokens. Deleting rows is a different kind of decision and the SQL is in the
handover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stage 2, and the end of the inversion. The two sync handlers (1,093 lines) ran in the
platform's queue, which meant the sidecar reached back over its registration socket to ask
the platform to enqueue work, and the credentials travelled through Postgres job metadata to
get there. Option (A) from the plan: they run here now, and the Jobs screen is left to the
things it actually describes.
The handlers moved almost unedited. Their bodies were already a list of steps taking a
context, so sync-runner.ts synthesizes that context and runs them; what went away is the
JobHandler wrapper and the registration. `job.userId` is the OWNER'S EMAIL rather than a
numeric id — the queue's naming — and it resolves the mail store path, so it is called out
in the type. That is the same field whose absence made the mailbox read as empty two commits
ago; it is set from user.email and checked this time.
Deliberately not a queue: one run per account, no persistence, no retry. A failure is picked
up by the ten-minute cron like any other, and a sync interrupted by a restart resumes from
the stored cursor rather than the beginning. PermanentError survives as a local class — it
signalled "do not retry" to the queue and now just carries its message to the sync state.
accounts.ts asks the runner whether an account is syncing instead of scanning job rows, and
the queue-over-WS shim in index.ts is gone: enqueueViaWs, listJobsViaWs, the pending-response
map and the queue branch in the command handler. Nothing but a port crosses that socket now.
The three chat channels stop opening the mail store directly. They each carried their own
copy of count-rows / enqueue / poll / count-again, coupling three chat bridges to the mail
schema — and they enqueued `gmail-sync` unconditionally, the OAuth path, for an
app-password account that syncs over IMAP, so the command was already broken. One shared
helper calls a new POST /sync-now on the sidecar, which syncs and reports what arrived.
queue/handlers/ is now empty; both handlers there were email. The queue is untouched and
still serves the Jobs screen.
Not moved, and fine where they are: scripts/migrate-emails-to-sqlite.ts and
scripts/seed-imap-uids.ts are one-off maintenance scripts that open the store directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mailbox read as empty after the migration — "No emails synced yet", and Sync Now
reporting no new mail against 18,430 messages that were sitting on disk untouched.
The store is keyed by the owner's email: DATA_PATH/<owner>/email_accounts/<account>/emails.db.
The platform's userMiddleware put the entire user row on the context, so `user.email`
resolved. The proxy injects only an id, and the middleware I wrote to replace it set
`{ id }` and nothing else — so `user.email` was undefined, the path never resolved to the
real mailbox, reads found nothing, and the sync compared against an empty set and concluded
there was nothing new. Nothing was written to the wrong place and no data was touched.
The sidecar loads the full row via getUserById now, matching what the middleware provided.
Cached: it runs on every request and single-user is a hard invariant.
Worth keeping in mind for the sidecars still to be extracted — moving routes across a
process boundary silently changes what is on the context, and it type-checks either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Email was the one sidecar built inside out. The platform held ~1,800 lines — the per-account
SQLite store, all 14 HTTP routes, account CRUD, resync, IMAP validation — while the 314-line
sidecar was a scheduler that reached BACK into the platform to do anything
(`import { performResync } from '../../api/email/resync'`).
The sidecar now serves its own HTTP listener and announces `email:server`, and
/api/email/* on the platform is createSidecarProxy like every other one: 1,801 lines down
to 22, with no mail knowledge left in it — not a message, not a folder, not a credential.
The routes moved verbatim, Hono and all. http.ts only reconstructs what the platform's
middleware used to provide: `user` on the context, from the X-Officer-User header the proxy
injects (trusted because this server binds loopback), and an error handler that turns
custom-errors into status codes.
The /email/events SSE stream went with them, which removes a whole round trip: the IDLE
watcher used to send `email:new` over the registration socket so the platform could push to
its SSE clients. Those clients are here now, so it calls broadcastEmailNew in-process and
`email:new` is gone from the wire protocol.
DELIBERATELY NOT DONE YET, and left backwards on purpose rather than half-moved:
- The two sync handlers (email-sync 381 lines, gmail-sync 712) still run in the platform's
queue and now import the store from its new home — a platform → sidecar import, which is
the wrong direction and is temporary. Moving them is option (A) from the plan: the sidecar
schedules its own syncs, independent of the platform Jobs list.
- accounts.ts still imports queue/init to enqueue a sync and to report sync status, and
index.ts still carries the queue-over-WS shim that inversion needs.
- The three channel handlers still open the mail store directly rather than asking over HTTP.
Two things worth knowing while testing: a from-scratch sync holds a proxied request open
well past the 60s idle default, hence timeoutSeconds on the proxy; and `gmail-sync` is
hardcoded in all three channel handlers even though the only account is provider=gmail with
auth_type=password, which routes to IMAP — so "sync emails" from a chat channel is
almost certainly already broken, and folds into the next stage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pastilhasandClaude Opus 5