pastilhasandClaude Opus 5
8207824a81
.env now holds PORT and POSTGRES_URL. Every encryption and signing key lives in $OFFICER_ROOT/secrets/officer-keys.db — 0600, 0700 directory, owned by the service user, created on first use. The design doc planned to move ONE at-rest key into the store. What shipped splits it: headscale, wallet, photos, jellyfin, invoiceshelf, vault and service-connections each get their own, plus jwt. VAULT_STORE_KEY encrypted all seven, so one leak opened all of them — and it was named after whichever plugin needed it first, which is why it read as safe to change if you did not run a vault. A core install bootstraps two, jwt and headscale; the rest appear when their plugin first asks. The file IS the secret. No second key unlocks it, because a key beside the store it opens buys nothing. The gain was never secrecy, it is blast radius: bun auto-loads .env into all twenty pm2 processes, so a key there is readable from /proc/<pid>/environ of twenty processes — officer-music held the key that decrypts wallet seed envelopes. Two defects found by testing the store rather than reading it, both of which would have shipped: The WAL was 0644. Enabling WAL creates -wal and -shm at 0644 rather than inheriting the database's mode, and a freshly written key lives in the WAL before checkpoint — so the 0600 on the database was decorative. The 0700 directory covered it, but only until someone loosened the directory. PRAGMA journal_mode = WAL takes an exclusive lock, and busy_timeout was set AFTER it. With twelve concurrent openers, six died on that line with SQLITE_BUSY. Every sidecar opens this store at boot, so they open it simultaneously by definition: most of them would have failed to start on a cold boot and none on a warm one. Fixed by ordering the pragmas; re-tested with twelve racing processes, one key, one row. crypto.ts takes a purpose as its first argument now, which the design doc had explicitly promised would not happen — 32 call sites across seven query modules. That promise is corrected in the doc rather than quietly dropped. Also live, not just comments: wallet/upstream.ts gated wallet storage on process.env.VAULT_STORE_KEY and would have reported "unconfigured" forever. It asks the store now, and the question it answers changed — not "did somebody set a variable" but "can this process open the store", since the key is created on demand. assertSecretsClosed covers the store, its directory and its WAL. The jwt key mints owner tokens, so a member's shell reading it is strictly worse than the .env leak that check was written for. Not typechecked: node_modules is empty and installs are frozen, so the officerdb/secret-store subpath could not be resolved at runtime here — verified that officerdb/types fails identically, so it is the empty tree and not the new export. The store module itself was tested directly: creation, idempotence across processes, hasKey not creating, permissions, and the twelve-way race. Every changed file parses; the setup section runs and degrades correctly when the import is unavailable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
252 lines
13 KiB
Markdown
252 lines
13 KiB
Markdown
# The secret store
|
||
|
||
**Status: BUILT 2026-08-13.** `src/databases/officer_db/src/secret-store.ts`, with `jwt.ts` and
|
||
`crypto.ts` reading from it and `officer-setup.sh` section 7 bootstrapping it. Rotation is NOT built —
|
||
the schema carries `retired_at` and the API exposes `retiredKeys()`, but nothing retires or re-encrypts
|
||
yet.
|
||
|
||
A small SQLite database holding every encryption and signing key the platform uses. It replaced
|
||
`VAULT_STORE_KEY` and `JWT_SECRET` in `.env`, and it is the facility a plugin uses instead of inventing
|
||
its own.
|
||
|
||
**One change from the design below: keys are per PURPOSE, not one key for everything.** The original
|
||
plan moved a single at-rest key into the store. What shipped gives `headscale`, `wallet`, `photos`,
|
||
`jellyfin`, `invoiceshelf`, `vault` and `service-connections` a key each, so one leaked key opens one
|
||
plugin's columns rather than all seven. `jwt` is the eighth. A core install bootstraps two — `jwt` and
|
||
`headscale` — and every other purpose is created when its plugin first asks.
|
||
|
||
---
|
||
|
||
## What is wrong with today
|
||
|
||
Nothing is insecure. The separation is already right — the thing worth keeping is stated first so it is
|
||
not lost in a refactor:
|
||
|
||
> **Secrets live in Postgres. The key that opens them does not.**
|
||
|
||
That is why `officer_db/src/crypto.ts` reads `VAULT_STORE_KEY` from the environment, and it is what
|
||
makes `keys.ts:26` true: *"a stolen database dump is useless without .env, a stolen .env is useless
|
||
without the passphrase"*.
|
||
|
||
What is wrong is narrower, and it is about **blast radius across processes**.
|
||
|
||
`.env` sits in the repository root, and Bun auto-loads it. `ecosystem.config.cjs` says so in as many
|
||
words — it is the reason the Anthropic credential was moved out of the main process. So today
|
||
`VAULT_STORE_KEY` is present in the environment of **all twenty pm2 processes**. `officer-music` holds
|
||
the key that decrypts wallet seed envelopes. Anything that can read `/proc/<pid>/environ` for those
|
||
processes has it, and nineteen of them have no reason to.
|
||
|
||
The second problem is that changing the key is currently unrecoverable rather than an operation. See
|
||
[Rotation](#rotation).
|
||
|
||
---
|
||
|
||
## What the key actually protects
|
||
|
||
Worth listing, because it is wider than the name suggests. Everything below is AES-256-GCM ciphertext in
|
||
Postgres, encrypted through `officer_db/src/crypto.ts` with a key derived as `SHA-256(VAULT_STORE_KEY)`:
|
||
|
||
| column | what it is |
|
||
| --- | --- |
|
||
| `headscale_servers.api_key` | a Headscale **admin** credential — the schema notes it "can delete every node on a tailnet" |
|
||
| `service_connections.secret` | every upstream credential the app store stores: gitea, memos, slskd, transmission |
|
||
| `jellyfin_servers.access_token` | Jellyfin session token |
|
||
| `wallets.config` | node credentials — macaroon, rune, LNDHub password, NWC URI. Spending authority |
|
||
| `wallets.seed_envelope` | a BIP39 mnemonic, already sealed under an owner passphrase, encrypted **again** with this key |
|
||
|
||
`decryptSecret` throws when the key does not verify, so a wrong key is not a degraded mode — it is every
|
||
one of those becoming unreadable at once.
|
||
|
||
The seed envelope is the only one protected by a second, independent secret (the owner passphrase, never
|
||
persisted). Everything else in that table has exactly one lock.
|
||
|
||
---
|
||
|
||
## Decisions
|
||
|
||
### 1. The store is SQLite, in the install, outside Postgres
|
||
|
||
Keys cannot live in the database they unlock. A dump would then contain both the ciphertext and the
|
||
thing that opens it, and the property quoted at the top stops being true. Encrypting the key with a
|
||
second key only moves the question — eventually exactly one secret has to be readable without any other
|
||
secret, and the only real decision is *where it lives*.
|
||
|
||
SQLite rather than a flat file, for one reason that is not secrecy: **rotation needs key versions.** A
|
||
rotation has to decrypt with the old key and re-encrypt with the new, and an interrupted rotation needs
|
||
both to still exist. That is a table with `id, purpose, key, created_at, retired_at`, and it is awkward
|
||
as an environment variable or a single-value file. Concurrent access from several sidecars is the second
|
||
reason; SQLite's locking is the part a hand-rolled file store gets wrong.
|
||
|
||
### 2. It is NOT encrypted at rest — and that decision changed shape
|
||
|
||
**As built, the file IS the secret.** Keys are stored as they are used, with no second key unlocking
|
||
them, because a key sitting beside the store it opens buys nothing: whoever can read one can read the
|
||
other. The boundary is `0700` on the directory, `0600` on the file, owned by the service user.
|
||
|
||
That answers open question 4 below — nothing stays outside, and `.env` holds no secret at all.
|
||
|
||
The original reasoning for encrypted-values-in-a-plaintext-file is kept below because the SQLCipher
|
||
finding is still true and still the reason whole-file encryption is not on the table.
|
||
|
||
#### The original note
|
||
|
||
Checked rather than assumed, because `PRAGMA key` appears to work and does not:
|
||
|
||
```
|
||
$ bun --eval 'db.exec("PRAGMA key = \"supersecret\""); … insert …'
|
||
read without key: THE-SECRET-VALUE
|
||
strings enc.db | grep THE-SECRET-VALUE -> found
|
||
```
|
||
|
||
Stock SQLite **silently ignores unknown pragmas**, so `PRAGMA key` succeeds, encrypts nothing, and the
|
||
value sits in the file in plaintext. `bun:sqlite` ships stock SQLite 3.53.0, not SQLCipher.
|
||
|
||
Whole-file encryption therefore needs SQLCipher, which means a native module — and this project already
|
||
knows what one of those costs, since node-pty has no Linux prebuild and compiles from source on every
|
||
machine.
|
||
|
||
So the store holds **encrypted values in an unencrypted file**, the same shape as the Postgres columns.
|
||
What leaks is metadata: which purposes have keys, and when they were rotated. That is an acceptable
|
||
trade and it is written down here so nobody later assumes the file is opaque.
|
||
|
||
`[open]` SQLCipher, if the native-dependency cost ever becomes worth paying.
|
||
|
||
### 3. Where the file goes
|
||
|
||
**`$OFFICER_ROOT/secrets/officer-keys.db`** — a sibling of `platform/` and `data/`, decided 2026-08-13.
|
||
|
||
**Not in `$OFFICER_ROOT/data/`.** That directory holds managed homes and attachments — it is the one
|
||
people back up. A key store that travels in the same tarball as a database dump rebuilds the exact
|
||
problem this design exists to avoid.
|
||
|
||
The setup script says so out loud when it creates the store, because "back this up, but not next to the
|
||
other thing you back up" is not a rule anyone infers.
|
||
|
||
### 4. ~~One secret remains outside~~ — none does
|
||
|
||
Answered 2026-08-13: **no secret remains in `.env`.** The store file is the secret, per decision 2.
|
||
|
||
The point of the exercise still holds, and it was always about blast radius rather than secrecy: **N
|
||
secrets in twenty process environments becomes a file read on demand by the few processes that need
|
||
it.** `.env` is auto-loaded by bun into every pm2 process, so a key there is readable from
|
||
`/proc/<pid>/environ` of twenty processes — `officer-music` held the key that decrypts wallet seed
|
||
envelopes. A file opened by the two or three processes that actually use a key does not.
|
||
|
||
### 5. What moves in
|
||
|
||
- `VAULT_STORE_KEY` — **split into one key per purpose**, rather than moved. See the status note at the
|
||
top: the table above is seven unrelated things, and one key for all of them meant one leak opened all
|
||
of them.
|
||
- `JWT_SECRET` — a signing key rather than an encryption key, but it has the same properties: must
|
||
survive restarts, must never be regenerated silently, and benefits from versioning during a rotation.
|
||
Leaving one in a store and one in `.env` would be the scattering this is meant to end.
|
||
- **The anthropic proxy secret**, purpose `anthropic-proxy`. Agreed 2026-08-12. Neither an encryption
|
||
key nor a signing key — a bearer credential, generated once by `ensureProxySecret` and presented by
|
||
`officer-agent` to `officer-anthropic-proxy` on `127.0.0.1`. It qualifies on the same three
|
||
properties: generated once, shared between two processes, fatal to regenerate silently.
|
||
|
||
It is in the store for a sharper reason than the other two, though. It is not in `.env` today — it
|
||
is in `$DATA_PATH/sidecar/claude-state.json`, mixed in with session records. That is the one
|
||
location [decision 3](#3-where-the-file-goes) rules out by name: `DATA_PATH` is what people back up,
|
||
so the secret already travels in the same tarball as the data it protects.
|
||
|
||
**Naming.** It is called `ANTHROPIC_API_KEY` in `ensureAnthropicEnv`, and that name is wrong in both
|
||
halves — it is not Anthropic's and it is not an API key. Anthropic's real credential is the OAuth
|
||
token in `~/.claude/.credentials.json`, which the proxy swaps this one for on the way out. Our name
|
||
for it is **anthropic-proxy-secret** everywhere we control.
|
||
|
||
The exception is the last line before the spawn. `claude` reads the variable `ANTHROPIC_API_KEY` and
|
||
format-checks the `sk-ant-api03-` prefix, so both are the CLI's contract rather than ours and both
|
||
stay. That one assignment keeps the CLI's name, with a comment saying why.
|
||
|
||
---
|
||
|
||
## Core, and plugins
|
||
|
||
The store is core infrastructure, created at first boot. It is **not** a side effect of installing any
|
||
one sidecar — it exists on a machine that installs nothing, so that a plugin installed in six months
|
||
finds it already there.
|
||
|
||
The core is what `ecosystem.light.config.cjs` runs today — `officer`, `officer-anthropic-proxy`,
|
||
`officer-agent`, `officer-opencode`, `officer-pty` — **plus `officer-headscale`**.
|
||
|
||
Headscale is core for a stated reason rather than by preference: `CLAUDE.md` says the tailnet *is* the
|
||
perimeter — origin checking was removed on 2026-08-13 precisely because the tailnet is what stands in
|
||
its place, so the tailnet is now load-bearing rather than one layer of two. A security model
|
||
that rests on the tailnet cannot treat administering the tailnet as an optional extra. Vaultwarden and
|
||
the wallet are not load-bearing that way — nothing else stops working without them — so they become
|
||
plugins.
|
||
|
||
Moving headscale into the light profile also removes it from the app store automatically:
|
||
`catalogue.test.ts` asserts the catalogue equals `full − light`, so the test fails until the entry is
|
||
deleted. That derivation is doing its job and should not be worked around.
|
||
|
||
Headscale is then the store's **first user**, not its creator — `headscale_servers.api_key` is the first
|
||
core credential needing a key.
|
||
|
||
---
|
||
|
||
## The contract
|
||
|
||
What a plugin gets, and is bound by. To be written properly when the first one uses it; the shape is:
|
||
|
||
- **Ask for a key by purpose**, not by name. `getKey('vault')` returns the active key for that purpose,
|
||
creating one on first use.
|
||
- **Never hold it.** Read it at the point of use. A key cached in a long-lived process is the
|
||
process-environment problem in a different container.
|
||
- **Never write to another plugin's purpose.** Same rule `service_connections` already has for rows.
|
||
- **Tolerate rotation.** A key may change between two calls. Anything that decrypts must be prepared to
|
||
be handed the retired key for data written before a rotation.
|
||
|
||
---
|
||
|
||
## Rotation
|
||
|
||
The feature that makes the store worth building, and the reason versions exist.
|
||
|
||
Today, changing `VAULT_STORE_KEY` is not an operation — it is data loss. Every column above becomes
|
||
unreadable, and for `wallets.seed_envelope` that is unrecoverable: the owner passphrase does not help,
|
||
because it opens the inner envelope and the outer one is gone. Unless the mnemonic was written down
|
||
offline, the coins are gone with it.
|
||
|
||
Rotation turns that into a supported action:
|
||
|
||
1. Mint a new key for the purpose, leaving the old one in the store as retired.
|
||
2. For every ciphertext column belonging to that purpose: decrypt with the retired key, re-encrypt with
|
||
the new one.
|
||
3. Retire the old key only when every row has moved.
|
||
|
||
Two properties it must have, both learned from the failure it replaces:
|
||
|
||
- **Transactional.** A half-rotated table is worse than either end state, because nothing afterwards can
|
||
tell which rows are which.
|
||
- **Verify before writing.** Every row must decrypt with the retired key *before* anything is written.
|
||
A key that is already wrong should fail loudly on row one rather than produce a second layer of
|
||
unreadable data.
|
||
|
||
`[open]` Whether rotation is a UI action, a CLI command, or both. It is a long operation on a large
|
||
wallet table and it cannot be interrupted safely, which argues for something that reports progress.
|
||
|
||
---
|
||
|
||
## What this does not change
|
||
|
||
- Secrets stay in Postgres. This moves the **keys**, not the data.
|
||
- ~~`crypto.ts`'s interface stays~~ — **it did not.** Per-purpose keys mean the purpose has to be named
|
||
at the call site, so it is `encryptSecret('headscale', plaintext)` now and all seven query modules
|
||
were touched. That was the cost of the split, and it is worth stating plainly because this line
|
||
originally promised the opposite.
|
||
- The owner passphrase on wallet seeds is untouched and stays out of every store. Two independent
|
||
secrets is the property that makes a stolen `.env` insufficient, and it survives this design.
|
||
|
||
---
|
||
|
||
## Open questions
|
||
|
||
1. Where the file lives, given it must not be swept up by a backup of `data/`.
|
||
2. Whether the store's own key stays in `.env` or moves to a file read on demand.
|
||
3. Whether rotation is UI, CLI, or both — and how it reports progress on a table that takes minutes.
|
||
4. SQLCipher, and whether whole-file encryption is ever worth a second native dependency.
|
||
5. What happens to a plugin's keys when it is uninstalled. The app store already decided that
|
||
uninstalling never deletes data; the same answer probably applies, but "probably" is not a decision.
|