Asked for a thorough check that nothing of offscale is left in the platform. The extraction was sound — no router, no permission entry, no app-store entry, no screen, no panel app, no env var anything reads. Two things were genuinely wrong, and both were in setup rather than in the server. ── The secret store minted a key for a plugin ── `officer-setup` bootstrapped two purposes: `jwt` and `headscale`. The second was a key for a PLUGIN, under a name the rename had already replaced, on a machine where the plugin is not installed and nothing would ever ask for it. offscale encrypts under `offscale`, and `getKey` creates on demand — which is precisely what lets core know nothing about plugins. Now core bootstraps `jwt` alone. ── `.env.example` documented variables for a deleted router ── HEADSCALE_URL, HEADSCALE_API_KEY and HEADSCALE_USER were described as driving `/api/vpn`. That router was deleted on 2026-08-14 and nothing has read any of them since. Left as a note saying where the tailnet's credentials live now, because somebody with an older `.env` will otherwise wonder why setting them does nothing. ── Names that outlived the thing ── Eleven comments referred to the plugin's route, panels, console or server registry as "headscale": three sidecar config files describing their own shape by pointing at it, two screens citing `/headscale` as the example of a Workspace/Panel route, the terminal's note about the Console, and the photos types. Also `catalogue.ts`, which still claimed headscale "is CORE now", and `working-on-officer.md`, which told the reader a core install runs six processes including `officer-headscale`. Upstream references are deliberately untouched: "Headscale's API", "the Headscale admin credential", "is this a Headscale server?" all name the thing the plugin drives, and renaming those would make the sentences false. `docs/secret-store.md` argued headscale must be core because the tailnet is the perimeter. Marked superseded rather than deleted, with the distinction that settles it: the perimeter is the tailnet EXISTING, which machine-setup still arranges on a core install — not the platform shipping the screen that administers it. Verified by counting: live code, setup provisioning, env references, permission entries, app-store entries, hono mounts, dock defaults, screens and panel apps all zero. What is left is generated files (Plugins.gen, plugin-schemas.gen), the bundled catalogue entry, test fixtures using "offscale" as an example app name, and history. tsgo clean, frontend builds, 873 pass / 7 fail unchanged, setup scripts parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
263 lines
14 KiB
Markdown
263 lines
14 KiB
Markdown
# The secret store
|
||
|
||
**Status: BUILT 2026-08-13.** `src/databases/officer_db/src/secret-store.ts`, with `jwt.ts` and
|
||
`crypto.ts` reading from it and `officer-setup.sh` section 7 bootstrapping it. Rotation is NOT built —
|
||
the schema carries `retired_at` and the API exposes `retiredKeys()`, but nothing retires or re-encrypts
|
||
yet.
|
||
|
||
A small SQLite database holding every encryption and signing key the platform uses. It replaced
|
||
`VAULT_STORE_KEY` and `JWT_SECRET` in `.env`, and it is the facility a plugin uses instead of inventing
|
||
its own.
|
||
|
||
**One change from the design below: keys are per PURPOSE, not one key for everything.** The original
|
||
plan moved a single at-rest key into the store. What shipped gives `offscale`, `wallet`, `photos`,
|
||
`jellyfin`, `invoiceshelf`, `vault` and `service-connections` a key each, so one leaked key opens one
|
||
plugin's columns rather than all seven. `jwt` is the eighth. A core install bootstraps **one** — `jwt` —
|
||
and every other purpose is created when its owner first asks.
|
||
|
||
(It bootstrapped `headscale` as well until 2026-08-16. That was a key for a plugin, minted under a name
|
||
the rename had already replaced, on machines where the plugin was not installed and nothing would ever
|
||
ask for it. `getKey` creates on demand, which is what makes core needing to know nothing about plugins
|
||
work at all.)
|
||
|
||
---
|
||
|
||
## What is wrong with today
|
||
|
||
Nothing is insecure. The separation is already right — the thing worth keeping is stated first so it is
|
||
not lost in a refactor:
|
||
|
||
> **Secrets live in Postgres. The key that opens them does not.**
|
||
|
||
That is why `officer_db/src/crypto.ts` reads `VAULT_STORE_KEY` from the environment, and it is what
|
||
makes `keys.ts:26` true: _"a stolen database dump is useless without .env, a stolen .env is useless
|
||
without the passphrase"_.
|
||
|
||
What is wrong is narrower, and it is about **blast radius across processes**.
|
||
|
||
`.env` sits in the repository root, and Bun auto-loads it. `ecosystem.config.cjs` says so in as many
|
||
words — it is the reason the Anthropic credential was moved out of the main process. So today
|
||
`VAULT_STORE_KEY` is present in the environment of **all twenty pm2 processes**. `officer-music` holds
|
||
the key that decrypts wallet seed envelopes. Anything that can read `/proc/<pid>/environ` for those
|
||
processes has it, and nineteen of them have no reason to.
|
||
|
||
The second problem is that changing the key is currently unrecoverable rather than an operation. See
|
||
[Rotation](#rotation).
|
||
|
||
---
|
||
|
||
## What the key actually protects
|
||
|
||
Worth listing, because it is wider than the name suggests. Everything below is AES-256-GCM ciphertext in
|
||
Postgres, encrypted through `officer_db/src/crypto.ts` with a key derived as `SHA-256(VAULT_STORE_KEY)`:
|
||
|
||
| column | what it is |
|
||
| ------------------------------- | --------------------------------------------------------------------------------------------- |
|
||
| `headscale_servers.api_key` | a Headscale **admin** credential — the schema notes it "can delete every node on a tailnet" |
|
||
| `service_connections.secret` | every upstream credential the app store stores: gitea, memos, slskd, transmission |
|
||
| `jellyfin_servers.access_token` | Jellyfin session token |
|
||
| `wallets.config` | node credentials — macaroon, rune, LNDHub password, NWC URI. Spending authority |
|
||
| `wallets.seed_envelope` | a BIP39 mnemonic, already sealed under an owner passphrase, encrypted **again** with this key |
|
||
|
||
`decryptSecret` throws when the key does not verify, so a wrong key is not a degraded mode — it is every
|
||
one of those becoming unreadable at once.
|
||
|
||
The seed envelope is the only one protected by a second, independent secret (the owner passphrase, never
|
||
persisted). Everything else in that table has exactly one lock.
|
||
|
||
---
|
||
|
||
## Decisions
|
||
|
||
### 1. The store is SQLite, in the install, outside Postgres
|
||
|
||
Keys cannot live in the database they unlock. A dump would then contain both the ciphertext and the
|
||
thing that opens it, and the property quoted at the top stops being true. Encrypting the key with a
|
||
second key only moves the question — eventually exactly one secret has to be readable without any other
|
||
secret, and the only real decision is _where it lives_.
|
||
|
||
SQLite rather than a flat file, for one reason that is not secrecy: **rotation needs key versions.** A
|
||
rotation has to decrypt with the old key and re-encrypt with the new, and an interrupted rotation needs
|
||
both to still exist. That is a table with `id, purpose, key, created_at, retired_at`, and it is awkward
|
||
as an environment variable or a single-value file. Concurrent access from several sidecars is the second
|
||
reason; SQLite's locking is the part a hand-rolled file store gets wrong.
|
||
|
||
### 2. It is NOT encrypted at rest — and that decision changed shape
|
||
|
||
**As built, the file IS the secret.** Keys are stored as they are used, with no second key unlocking
|
||
them, because a key sitting beside the store it opens buys nothing: whoever can read one can read the
|
||
other. The boundary is `0700` on the directory, `0600` on the file, owned by the service user.
|
||
|
||
That answers open question 4 below — nothing stays outside, and `.env` holds no secret at all.
|
||
|
||
The original reasoning for encrypted-values-in-a-plaintext-file is kept below because the SQLCipher
|
||
finding is still true and still the reason whole-file encryption is not on the table.
|
||
|
||
#### The original note
|
||
|
||
Checked rather than assumed, because `PRAGMA key` appears to work and does not:
|
||
|
||
```
|
||
$ bun --eval 'db.exec("PRAGMA key = \"supersecret\""); … insert …'
|
||
read without key: THE-SECRET-VALUE
|
||
strings enc.db | grep THE-SECRET-VALUE -> found
|
||
```
|
||
|
||
Stock SQLite **silently ignores unknown pragmas**, so `PRAGMA key` succeeds, encrypts nothing, and the
|
||
value sits in the file in plaintext. `bun:sqlite` ships stock SQLite 3.53.0, not SQLCipher.
|
||
|
||
Whole-file encryption therefore needs SQLCipher, which means a native module — and this project already
|
||
knows what one of those costs, since node-pty has no Linux prebuild and compiles from source on every
|
||
machine.
|
||
|
||
So the store holds **encrypted values in an unencrypted file**, the same shape as the Postgres columns.
|
||
What leaks is metadata: which purposes have keys, and when they were rotated. That is an acceptable
|
||
trade and it is written down here so nobody later assumes the file is opaque.
|
||
|
||
`[open]` SQLCipher, if the native-dependency cost ever becomes worth paying.
|
||
|
||
### 3. Where the file goes
|
||
|
||
**`$OFFICER_ROOT/secrets/officer-keys.db`** — a sibling of `platform/` and `data/`, decided 2026-08-13.
|
||
|
||
**Not in `$OFFICER_ROOT/data/`.** That directory holds managed homes and attachments — it is the one
|
||
people back up. A key store that travels in the same tarball as a database dump rebuilds the exact
|
||
problem this design exists to avoid.
|
||
|
||
The setup script says so out loud when it creates the store, because "back this up, but not next to the
|
||
other thing you back up" is not a rule anyone infers.
|
||
|
||
### 4. ~~One secret remains outside~~ — none does
|
||
|
||
Answered 2026-08-13: **no secret remains in `.env`.** The store file is the secret, per decision 2.
|
||
|
||
The point of the exercise still holds, and it was always about blast radius rather than secrecy: **N
|
||
secrets in twenty process environments becomes a file read on demand by the few processes that need
|
||
it.** `.env` is auto-loaded by bun into every pm2 process, so a key there is readable from
|
||
`/proc/<pid>/environ` of twenty processes — `officer-music` held the key that decrypts wallet seed
|
||
envelopes. A file opened by the two or three processes that actually use a key does not.
|
||
|
||
### 5. What moves in
|
||
|
||
- `VAULT_STORE_KEY` — **split into one key per purpose**, rather than moved. See the status note at the
|
||
top: the table above is seven unrelated things, and one key for all of them meant one leak opened all
|
||
of them.
|
||
- `JWT_SECRET` — a signing key rather than an encryption key, but it has the same properties: must
|
||
survive restarts, must never be regenerated silently, and benefits from versioning during a rotation.
|
||
Leaving one in a store and one in `.env` would be the scattering this is meant to end.
|
||
- **The anthropic proxy secret**, purpose `anthropic-proxy`. Agreed 2026-08-12. Neither an encryption
|
||
key nor a signing key — a bearer credential, generated once by `ensureProxySecret` and presented by
|
||
`officer-agent` to `officer-anthropic-proxy` on `127.0.0.1`. It qualifies on the same three
|
||
properties: generated once, shared between two processes, fatal to regenerate silently.
|
||
|
||
It is in the store for a sharper reason than the other two, though. It is not in `.env` today — it
|
||
is in `$DATA_PATH/sidecar/claude-state.json`, mixed in with session records. That is the one
|
||
location [decision 3](#3-where-the-file-goes) rules out by name: `DATA_PATH` is what people back up,
|
||
so the secret already travels in the same tarball as the data it protects.
|
||
|
||
**Naming.** It is called `ANTHROPIC_API_KEY` in `ensureAnthropicEnv`, and that name is wrong in both
|
||
halves — it is not Anthropic's and it is not an API key. Anthropic's real credential is the OAuth
|
||
token in `~/.claude/.credentials.json`, which the proxy swaps this one for on the way out. Our name
|
||
for it is **anthropic-proxy-secret** everywhere we control.
|
||
|
||
The exception is the last line before the spawn. `claude` reads the variable `ANTHROPIC_API_KEY` and
|
||
format-checks the `sk-ant-api03-` prefix, so both are the CLI's contract rather than ours and both
|
||
stay. That one assignment keeps the CLI's name, with a comment saying why.
|
||
|
||
---
|
||
|
||
## Core, and plugins
|
||
|
||
The store is core infrastructure, created at first boot. It is **not** a side effect of installing any
|
||
one sidecar — it exists on a machine that installs nothing, so that a plugin installed in six months
|
||
finds it already there.
|
||
|
||
The core is what `ecosystem.light.config.cjs` runs today — `officer`, `officer-anthropic-proxy`,
|
||
`officer-agent`, `officer-opencode`, `officer-pty` — **plus `officer-headscale`**.
|
||
|
||
**Superseded 2026-08-15.** The tailnet left the repository and is the `offscale` plugin; core is the five
|
||
above. The argument below was not refuted — it was overtaken. What settled it: the perimeter is the
|
||
tailnet EXISTING, which is a machine-setup concern, and that is not the same as the platform shipping the
|
||
screen that administers it. `machine-setup.sh` still walks you onto a tailnet on a core install; only the
|
||
admin UI became optional. Kept because the reasoning is what a future "should X be core?" argument needs.
|
||
|
||
Headscale is core for a stated reason rather than by preference: `CLAUDE.md` says the tailnet _is_ the
|
||
perimeter — origin checking was removed on 2026-08-13 precisely because the tailnet is what stands in
|
||
its place, so the tailnet is now load-bearing rather than one layer of two. A security model
|
||
that rests on the tailnet cannot treat administering the tailnet as an optional extra. Vaultwarden and
|
||
the wallet are not load-bearing that way — nothing else stops working without them — so they become
|
||
plugins.
|
||
|
||
Moving headscale into the light profile also removes it from the app store automatically:
|
||
`catalogue.test.ts` asserts the catalogue equals `full − light`, so the test fails until the entry is
|
||
deleted. That derivation is doing its job and should not be worked around.
|
||
|
||
Headscale is then the store's **first user**, not its creator — `headscale_servers.api_key` is the first
|
||
core credential needing a key.
|
||
|
||
---
|
||
|
||
## The contract
|
||
|
||
What a plugin gets, and is bound by. To be written properly when the first one uses it; the shape is:
|
||
|
||
- **Ask for a key by purpose**, not by name. `getKey('vault')` returns the active key for that purpose,
|
||
creating one on first use.
|
||
- **Never hold it.** Read it at the point of use. A key cached in a long-lived process is the
|
||
process-environment problem in a different container.
|
||
- **Never write to another plugin's purpose.** Same rule `service_connections` already has for rows.
|
||
- **Tolerate rotation.** A key may change between two calls. Anything that decrypts must be prepared to
|
||
be handed the retired key for data written before a rotation.
|
||
|
||
---
|
||
|
||
## Rotation
|
||
|
||
The feature that makes the store worth building, and the reason versions exist.
|
||
|
||
Today, changing `VAULT_STORE_KEY` is not an operation — it is data loss. Every column above becomes
|
||
unreadable, and for `wallets.seed_envelope` that is unrecoverable: the owner passphrase does not help,
|
||
because it opens the inner envelope and the outer one is gone. Unless the mnemonic was written down
|
||
offline, the coins are gone with it.
|
||
|
||
Rotation turns that into a supported action:
|
||
|
||
1. Mint a new key for the purpose, leaving the old one in the store as retired.
|
||
2. For every ciphertext column belonging to that purpose: decrypt with the retired key, re-encrypt with
|
||
the new one.
|
||
3. Retire the old key only when every row has moved.
|
||
|
||
Two properties it must have, both learned from the failure it replaces:
|
||
|
||
- **Transactional.** A half-rotated table is worse than either end state, because nothing afterwards can
|
||
tell which rows are which.
|
||
- **Verify before writing.** Every row must decrypt with the retired key _before_ anything is written.
|
||
A key that is already wrong should fail loudly on row one rather than produce a second layer of
|
||
unreadable data.
|
||
|
||
`[open]` Whether rotation is a UI action, a CLI command, or both. It is a long operation on a large
|
||
wallet table and it cannot be interrupted safely, which argues for something that reports progress.
|
||
|
||
---
|
||
|
||
## What this does not change
|
||
|
||
- Secrets stay in Postgres. This moves the **keys**, not the data.
|
||
- ~~`crypto.ts`'s interface stays~~ — **it did not.** Per-purpose keys mean the purpose has to be named
|
||
at the call site, so it is `encryptSecret('headscale', plaintext)` now and all seven query modules
|
||
were touched. That was the cost of the split, and it is worth stating plainly because this line
|
||
originally promised the opposite.
|
||
- The owner passphrase on wallet seeds is untouched and stays out of every store. Two independent
|
||
secrets is the property that makes a stolen `.env` insufficient, and it survives this design.
|
||
|
||
---
|
||
|
||
## Open questions
|
||
|
||
1. Where the file lives, given it must not be swept up by a backup of `data/`.
|
||
2. Whether the store's own key stays in `.env` or moves to a file read on demand.
|
||
3. Whether rotation is UI, CLI, or both — and how it reports progress on a table that takes minutes.
|
||
4. SQLCipher, and whether whole-file encryption is ever worth a second native dependency.
|
||
5. What happens to a plugin's keys when it is uninstalled. The app store already decided that
|
||
uninstalling never deletes data; the same answer probably applies, but "probably" is not a decision.
|