From fc48a572d2713e4ffcc5f2312034c1c1b297b991 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 12:24:13 +0000 Subject: [PATCH 01/99] app store: the catalogue, the install-state table, and what phase 0 must not foreclose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First slice, on a worktree branch so none of it touches the tree the live server runs from. `sidecar_installs` — server-level, no userId, because a sidecar is one process serving the machine. That is the line that keeps the model coherent for several users: installed is server-level and owner-only, configured is per user in service_connections. A member can use Gitea without being able to install it or point it somewhere else. `installed` and `enabled` are separate because they answer different questions, which is what gives the reversible middle ground: disable stops the process and keeps container, config, schema and data. `completedSteps` makes install resumable rather than merely retryable — the failure mode being designed against is a half-installed service that neither works nor uninstalls. The catalogue is data, not code: no functions, no compile-time coupling, because the same shape has to arrive as JSON from marketplace.officer.dev later. Its test pins it to the real estate — it offers exactly the processes the light profile excludes, names processes that exist, and claims capabilities that exist. That last check earned itself immediately: it caught `vault` (no capability at all — it is EXEMPT because Bitwarden clients carry a Vaultwarden bearer, not a platform JWT) and `notify` (which does have one, where I had written null). Docker templates follow the convention already in use across 47 services in ~/dockers: a directory per service, compose inside, relative bind mounts so data sits beside it, USER_UID/USER_GID as the owner. An existing directory is evidence of an existing install and must be adopted, never overwritten. Records what Phase 0 must not foreclose: a remote marketplace, sidecars moving to their own repositories, and third-party plugins — including the note that catalogue.test.ts pins Phase 0's invariant rather than the design's, since that relationship inverts once sidecars leave this repo. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 43 ++- .../officer_db/src/schema/app-store.ts | 80 ++++++ src/servers/app-store/catalogue.test.ts | 88 ++++++ src/servers/app-store/catalogue.ts | 252 ++++++++++++++++++ 4 files changed, 459 insertions(+), 4 deletions(-) create mode 100644 src/databases/officer_db/src/schema/app-store.ts create mode 100644 src/servers/app-store/catalogue.test.ts create mode 100644 src/servers/app-store/catalogue.ts diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index f6c088f9..07e150ab 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -60,8 +60,12 @@ for someone who does not. The prompt is the fork. **Officer is the installer, never the owner.** Concretely: -- A real `docker-compose.yml` per service, written into a **user-owned directory** - (`~/officer-services//`), from our template. +- A real compose file per service, written into a **user-owned directory**, from our template — + following the convention the owner already uses for 47 services in `~/dockers/`: + one directory per service, `docker-compose.yaml` inside, and **relative bind mounts** + (`./data`, `./database`, `./storage`) so configuration and data sit beside the compose file where + both we and the user can find them. Named volumes are used by 3 of those 47 and are the exception; + templates use bind mounts, always. - Started with `docker compose up -d` **as the owner**, not as officer's own identity. - Found again by **label** (`officer.sidecar=`), not by holding a handle. @@ -75,8 +79,16 @@ Consequences, which are the point: The template is what makes this non-technical-user-friendly: sensible defaults, ports, volumes and health checks already correct, so "install Gitea" does not become a tutorial. -`[open]` Rootful Docker runs container processes as root unless `user:` is set. Do we set it? And do we -support Podman for people who want genuinely rootless? +**`USER_UID` / `USER_GID` are set to the owner**, as the existing services already do. That answers the +"do containers run as root" question: no, and this is not a new convention — it is the one in use. + +**An existing directory is evidence, not an obstacle.** `~/dockers/` already holds `gitea`, `memos`, +`immich`, `jellyfin` and `invoice_shelf`. The installer must never write into a directory that exists; +finding one is the strongest possible signal that this is the "you already have one" case, and the store +should offer to ADOPT it — read its ports out of the compose file and write the connection — rather than +provision a second copy or overwrite a running service's data. + +`[open]` Podman, for anyone wanting genuinely rootless. --- @@ -174,6 +186,29 @@ What a plugin author is promised, and bound by. To be written properly; the shap --- +## What Phase 0 must not foreclose + +Three things are coming, and each one constrains a decision that looks free today. + +**1. `marketplace.officer.dev`.** Phase 1 keeps the catalogue inside this repo; later the app lists what +is on a remote marketplace instead. So catalogue entries must stay **serialisable data** — no functions, +no imports, nothing that only means something at compile time. They are plain objects today and must +remain so, because the same shape has to arrive as JSON over HTTP. Compose templates travel with them. + +**2. Every sidecar becomes its own repository.** Today `catalogue.test.ts` asserts the catalogue equals +"everything in ecosystem.config.cjs that light excludes". That is the right check _now_, and it inverts +later: once sidecars live elsewhere, the catalogue entry becomes the source of truth for how to run one +(command, args, env) and the ecosystem file is generated from what is installed, not the other way +round. **Do not treat that test as a permanent law** — it pins Phase 0's invariant, not the design's. + +**3. Third-party plugins.** Already the reason per-sidecar schema is in scope. It is also why the +`service_connections` ID needs namespacing before the marketplace opens, not after. + +The through-line: **nothing in Phase 0 may assume the catalogue is compiled in, or that a sidecar's code +is in this repository.** + +--- + ## Open questions 1. `user:` in compose, and Podman support for rootless. diff --git a/src/databases/officer_db/src/schema/app-store.ts b/src/databases/officer_db/src/schema/app-store.ts new file mode 100644 index 00000000..7f802909 --- /dev/null +++ b/src/databases/officer_db/src/schema/app-store.ts @@ -0,0 +1,80 @@ +import { pgTable, serial, text, boolean, timestamp, jsonb, uniqueIndex } from 'drizzle-orm/pg-core'; + +// What the owner has installed from the app store, and whether it should be running. +// +// ── Why there is no userId ── +// +// A sidecar is ONE process serving the whole machine, so installing one is a server-level act, not a +// per-user one. This is the line that keeps the model coherent once several people share a server: +// +// installed server-level, owner-only — this row +// configured per user — service_connections +// +// Gitea is the worked example. The owner installs it once, and the row here says the process runs; each +// member then holds their own credential in `service_connections`, inheriting the instance URL from the +// owner's row. A member can therefore use a service without being able to install, uninstall or point it +// somewhere else — which is the same split `capabilities/registry.ts` already draws between `app` and +// `admin` kinds. +// +// ── Why `installed` and `enabled` are separate ── +// +// They answer different questions. `installed` means the thing EXISTS: a container was provisioned (or an +// existing instance was named), config was written, schema was applied. `enabled` means the process +// SHOULD BE RUNNING. Disabling is the reversible middle ground the owner asked for — stop the process, +// keep the container, the config, the tables and the data, and start again later at no cost. +// +// Uninstall then has a disposal choice rather than a fixed meaning: keep the data, drop the container, +// or drop both. None of those are this table's business beyond recording that the row is gone. +export const sidecarInstalls = pgTable( + 'sidecar_installs', + { + id: serial('id').primaryKey(), + /** + * The catalogue id — 'gitea', 'photos'. Text rather than an enum for the same reason + * `service_connections.service` is: adding a sidecar must not be a schema change, and a third-party + * one cannot be in an enum we compile. + */ + sidecarId: text('sidecar_id').notNull(), + /** + * How this install was satisfied, which decides what uninstall has to undo: + * + * 'existing' — pointed at an instance the user already runs. We provisioned nothing. + * 'provisioned' — we rendered a compose file and started containers. Ours to offer to remove. + * 'config' — nothing to reach; credentials only (email, wallet). + */ + mode: text('mode').notNull(), + /** + * 'pending' | 'installing' | 'installed' | 'failed'. + * + * `installing` is a real, persisted state rather than a transient one, because install spans a + * container start and a health wait and can be interrupted by a restart in the middle. A row stuck in + * `installing` is the signal to resume, not evidence of a bug. + */ + status: text('status').notNull().default('pending'), + enabled: boolean('enabled').notNull().default(false), + /** + * Which install steps have completed, by name. This is what makes install RESUMABLE rather than + * merely retryable: re-running picks up after the last completed step instead of provisioning a + * second container or re-writing config that is already right. + * + * The failure mode being designed against is a half-installed service that neither works nor + * uninstalls — which is the one users cannot get themselves out of. + */ + completedSteps: jsonb('completed_steps').$type().notNull().default([]), + /** Why the last attempt failed, shown in the UI. Cleared on the next successful step. */ + lastError: text('last_error'), + /** + * Absolute path to the rendered compose directory, for `mode: 'provisioned'` only. + * + * Stored rather than derived because it is the user's directory and he may move it — and because + * uninstall must not guess at a path it is about to run `docker compose down -v` in. + */ + composeDir: text('compose_dir'), + installedAt: timestamp('installed_at', { withTimezone: true }), + createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(), + }, + // One row per sidecar, machine-wide. There is no second install of the same sidecar to disambiguate, + // which is what lets every other column be read without asking "which one". + (t) => [uniqueIndex('uq_sidecar_installs_sidecar').on(t.sidecarId)], +); diff --git a/src/servers/app-store/catalogue.test.ts b/src/servers/app-store/catalogue.test.ts new file mode 100644 index 00000000..6f63b9ae --- /dev/null +++ b/src/servers/app-store/catalogue.test.ts @@ -0,0 +1,88 @@ +import { describe, expect, it } from 'bun:test'; +import { CATALOGUE, byId } from './catalogue'; +import { CAPABILITIES } from '../capabilities/registry'; + +// The catalogue is a hand-written list describing machinery that lives elsewhere, which is the shape of +// thing that rots silently. These tests pin it to the three sources it claims to agree with: +// ecosystem.config.cjs, the light profile, and the capability registry. +// +// The intent is that adding a sidecar to the estate and forgetting the app store FAILS HERE, rather than +// the sidecar being quietly uninstallable and nobody noticing for a release. + +const full = (require('../../../ecosystem.config.cjs') as { apps: { name: string }[] }).apps.map((a) => a.name); +const light = (require('../../../ecosystem.light.config.cjs') as { apps: { name: string }[] }).apps.map((a) => a.name); + +describe('the catalogue against the real estate', () => { + it('offers exactly the processes the light profile leaves out', () => { + // This is the definition of the app store: light is the baseline, everything else is installable. + const notInLight = full.filter((name) => !light.includes(name)).sort(); + const offered = CATALOGUE.map((e) => e.process).sort(); + + expect(offered).toEqual(notInLight); + }); + + it('names a process that actually exists in the ecosystem', () => { + // A typo here would install nothing and report success. + for (const entry of CATALOGUE) expect(full).toContain(entry.process); + }); + + it('does not offer to install the baseline', () => { + // "Uninstall chat" is not a thing the store should be able to express. + for (const entry of CATALOGUE) expect(light).not.toContain(entry.process); + }); +}); + +describe('entries are internally coherent', () => { + it('has unique ids', () => { + const ids = CATALOGUE.map((e) => e.id); + expect(new Set(ids).size).toBe(ids.length); + }); + + it('declares a compose template if and only if it can provision one', () => { + // Both directions matter: a provisioning entry with no template fails at install time, and a + // template on a non-provisioning entry is dead weight nobody will notice is unused. + for (const entry of CATALOGUE) { + if (entry.modes.includes('provisioned')) expect(entry.composeTemplate).toBeTruthy(); + else expect(entry.composeTemplate).toBeUndefined(); + } + }); + + it('asks for connection details exactly when it points at something existing', () => { + for (const entry of CATALOGUE) { + if (entry.modes.includes('existing')) expect(entry.existingFields?.length).toBeGreaterThan(0); + else expect(entry.existingFields).toBeUndefined(); + } + }); + + it('requires a url when pointing at an existing instance', () => { + // Without one there is nothing to connect to, and the `existing` mode is meaningless. + for (const entry of CATALOGUE) { + if (!entry.modes.includes('existing')) continue; + const url = entry.existingFields?.find((f) => f.key === 'url'); + expect(url?.required).toBe(true); + } + }); + + it('declares at least one mode', () => { + for (const entry of CATALOGUE) expect(entry.modes.length).toBeGreaterThan(0); + }); +}); + +describe('capabilities it claims to back', () => { + it('names a capability that exists, or explicitly none', () => { + // `null` is a real answer — notify has no surface of its own — but a WRONG key would silently + // detach the store entry from the permission that governs the feature. + const keys = new Set(CAPABILITIES.map((c) => c.key)); + for (const entry of CATALOGUE) { + if (entry.capability === null) continue; + expect(keys).toContain(entry.capability); + } + }); +}); + +describe('byId', () => { + it('finds a known entry and returns undefined for anything else', () => { + expect(byId('photos')?.process).toBe('officer-photos'); + expect(byId('not-a-sidecar')).toBeUndefined(); + }); +}); diff --git a/src/servers/app-store/catalogue.ts b/src/servers/app-store/catalogue.ts new file mode 100644 index 00000000..acb43b83 --- /dev/null +++ b/src/servers/app-store/catalogue.ts @@ -0,0 +1,252 @@ +// What the app store can install, and what installing each one actually requires. +// +// This is the catalogue, not the state — `sidecar_installs` records what the owner has done, this +// records what is on offer. Everything the installer branches on lives here so the installer itself has +// no per-sidecar knowledge, the same restraint `create-proxy.ts` keeps on the proxy side. +// +// ── The entries are derived from a fact, not invented ── +// +// The fourteen below are exactly the processes in `ecosystem.config.cjs` that the light profile +// excludes. That is not a coincidence to be maintained by hand: `catalogue.test.ts` asserts it, so a +// sidecar added to the estate and forgotten here fails a test rather than being quietly uninstallable. +// +// Light itself (officer, the agent, the anthropic proxy, opencode, pty, gitea) is not in the catalogue. +// Those are the baseline — chat, terminal, file browser — and there is no meaningful "uninstall chat". + +/** + * How an install can be satisfied. A sidecar may support more than one, and the prompt is the fork: + * "do you already have one, or shall we start one for you?" + */ +export type InstallMode = + /** Point at an instance the user already runs, here or elsewhere. We provision nothing. */ + | 'existing' + /** Render our compose template and start it. The connection is then known without asking. */ + | 'provisioned' + /** Nothing to reach. Credentials or local configuration only. */ + | 'config'; + +/** One field the installer asks for before it can finish. */ +export type ConfigField = { + key: string; + label: string; + /** `secret` is written encrypted and never read back to the client. */ + type: 'url' | 'text' | 'secret'; + required: boolean; + placeholder?: string; + help?: string; +}; + +export type CatalogueEntry = { + /** Stable id. Matches `sidecar_installs.sidecar_id` and `service_connections.service` where both exist. */ + id: string; + /** The PM2 process to start and stop. Must exist in ecosystem.config.cjs. */ + process: string; + label: string; + /** One line, shown in the store listing. */ + summary: string; + /** Which of the three shapes this sidecar supports, in the order the UI should offer them. */ + modes: InstallMode[]; + /** + * The capability this sidecar backs, from `capabilities/registry.ts`. Null where the sidecar has no + * user-facing surface of its own (notify produces notifications for other features). + */ + capability: string | null; + /** + * Asked when the user picks `existing`. Skipped entirely for `provisioned`, where we already know the + * answers because we wrote the compose file. + */ + existingFields?: ConfigField[]; + /** Asked for `config` installs, which have no instance to point at. */ + configFields?: ConfigField[]; + /** Name of the compose template under `app-store/templates/`. Required iff `modes` includes 'provisioned'. */ + composeTemplate?: string; + /** + * Why this cannot be installed on some hosts, if so. Shown instead of the install button rather than + * failing halfway through — a check the installer can make before it starts. + */ + requires?: 'docker' | 'linux-display'; +}; + +export const CATALOGUE: CatalogueEntry[] = [ + // ── Point at something you already run, or let us start one ──────────────────────────────────────── + { + id: 'photos', + process: 'officer-photos', + label: 'Photos', + summary: 'Your Immich library — browse, search, upload from the phone', + modes: ['existing', 'provisioned'], + capability: 'photos', + composeTemplate: 'immich', + existingFields: [ + { key: 'url', label: 'Immich URL', type: 'url', required: true, placeholder: 'https://photos.example.com' }, + { key: 'secret', label: 'API key', type: 'secret', required: true, help: 'Immich → Account Settings → API Keys. Create it with all permissions: a scoped key returns 403 per route, which reads as a broken feature.' }, + ], + }, + { + id: 'jellyfin', + process: 'officer-jellyfin', + label: 'Jellyfin', + summary: 'Films and shows, with a player that handles direct, HLS and progressive', + modes: ['existing', 'provisioned'], + capability: 'jellyfin', + composeTemplate: 'jellyfin', + existingFields: [ + { key: 'url', label: 'Jellyfin URL', type: 'url', required: true, placeholder: 'http://localhost:8096' }, + { key: 'secret', label: 'Access token', type: 'secret', required: true }, + ], + }, + { + id: 'memos', + process: 'officer-memos', + label: 'Memos', + summary: 'Quick notes, tagged and searchable', + modes: ['existing', 'provisioned'], + capability: 'memos', + composeTemplate: 'memos', + existingFields: [ + { key: 'url', label: 'Memos URL', type: 'url', required: true }, + { key: 'secret', label: 'Access token', type: 'secret', required: true }, + ], + }, + { + id: 'invoiceshelf', + process: 'officer-invoiceshelf', + label: 'Invoices', + summary: 'InvoiceShelf — clients, estimates and invoices', + modes: ['existing', 'provisioned'], + capability: 'invoices', + composeTemplate: 'invoiceshelf', + existingFields: [ + { key: 'url', label: 'InvoiceShelf URL', type: 'url', required: true }, + { key: 'secret', label: 'API token', type: 'secret', required: true }, + ], + }, + { + id: 'vault', + process: 'officer-vault', + label: 'Vault', + summary: 'Vaultwarden — passwords, reachable by the Bitwarden apps', + modes: ['existing', 'provisioned'], + // No capability entry exists for this one, and the reason is about CREDENTIALS, not routing. + // + // Every request still goes through us: `/api/vault` is mounted on vaultRouter and forwarded by the + // officer-vault sidecar to Vaultwarden. The Bitwarden clients never reach Vaultwarden directly. + // + // What they do NOT carry is a platform JWT — they present their own Vaultwarden bearer token — so + // `userMiddleware` would 401 them and a capability lookup would have no account to resolve. Hence + // `/vault` sits in EXEMPT_API_PREFIXES, gated by origin scoping and Vaultwarden's own auth instead. + // The install still governs whether the sidecar runs at all. + capability: null, + composeTemplate: 'vaultwarden', + existingFields: [{ key: 'url', label: 'Vaultwarden URL', type: 'url', required: true }], + }, + { + id: 'transmission', + process: 'officer-transmission', + label: 'Transmission', + summary: 'Torrents, with the daemon Officer talks to over RPC', + modes: ['existing', 'provisioned'], + capability: 'transmission', + composeTemplate: 'transmission', + existingFields: [ + { key: 'url', label: 'Transmission URL', type: 'url', required: true, placeholder: 'http://localhost:9091' }, + { key: 'path', label: 'RPC path', type: 'text', required: false, placeholder: '/transmission/rpc', help: 'Only differs behind a reverse proxy.' }, + { key: 'username', label: 'RPC username', type: 'text', required: false, help: 'Usually blank — Transmission is normally run with no RPC auth.' }, + { key: 'secret', label: 'RPC password', type: 'secret', required: false }, + ], + }, + { + id: 'slskd', + process: 'officer-slskd', + label: 'Soulseek', + summary: 'slskd — search and download from the Soulseek network', + modes: ['existing', 'provisioned'], + capability: 'soulseek', + composeTemplate: 'slskd', + existingFields: [ + { key: 'url', label: 'slskd URL', type: 'url', required: true }, + { key: 'secret', label: 'API key', type: 'secret', required: true }, + ], + }, + { + id: 'caldav', + process: 'officer-caldav', + label: 'Calendar', + summary: 'Radicale — calendars and contacts over CalDAV/CardDAV', + modes: ['existing', 'provisioned'], + capability: 'calendar', + composeTemplate: 'radicale', + existingFields: [{ key: 'url', label: 'CalDAV URL', type: 'url', required: true }], + }, + { + id: 'headscale', + process: 'officer-headscale', + label: 'Headscale', + summary: 'Your own tailnet control plane', + modes: ['existing'], + capability: 'headscale', + existingFields: [ + { key: 'url', label: 'Headscale URL', type: 'url', required: true }, + { key: 'secret', label: 'API key', type: 'secret', required: true }, + ], + }, + + // ── Nothing to reach: configuration only ─────────────────────────────────────────────────────────── + { + id: 'email', + process: 'officer-email', + label: 'Email', + summary: 'Your IMAP accounts, synced and searchable', + modes: ['config'], + capability: 'email', + // Deliberately empty: accounts are added from /email, which already has a working multi-account + // form. Duplicating it here would be a second place to maintain the same credentials. + configFields: [], + }, + { + id: 'music', + process: 'officer-music', + label: 'Music', + summary: 'Index and play the library on this machine', + modes: ['config'], + capability: 'music', + configFields: [ + { key: 'library', label: 'Music folder', type: 'text', required: true, placeholder: '~/Music' }, + ], + }, + { + id: 'wallet', + process: 'officer-wallet', + label: 'Wallet', + summary: 'Bitcoin and Lightning, with keys held by the sidecar alone', + modes: ['config'], + capability: 'wallet', + configFields: [], + }, + { + id: 'notify', + process: 'officer-notify', + label: 'Notifications', + summary: 'Push to your phone when a job finishes or a turn needs you', + modes: ['config'], + capability: 'notify', + configFields: [], + }, + { + id: 'vnc', + process: 'officer-vnc', + label: 'Desktop', + summary: 'Mirror this machine’s display in the browser', + modes: ['config'], + capability: 'desktop', + // x11vnc against an Xorg display. There is nothing to mirror on a headless box or on macOS, so the + // store should say so rather than install something that starts and immediately fails. + requires: 'linux-display', + configFields: [], + }, +]; + +export const byId = (id: string): CatalogueEntry | undefined => CATALOGUE.find((e) => e.id === id); + +/** Every catalogue id, for validating a request before it reaches the installer. */ +export const CATALOGUE_IDS: ReadonlySet = new Set(CATALOGUE.map((e) => e.id)); From 654cb10711788f62e48f24eadb45c5d779ae664b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 12:29:03 +0000 Subject: [PATCH 02/99] app store: put provisioned containers under the officer root, not the user's MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The install layout a machine should have, seasoned owner or not: ~/officerdev/ platform/ the app data/ DATA_PATH dockers/ services the app store provisioned capabilities/ the file-based item store One root, everything under it. OFFICER_ROOT derives from DATA_PATH rather than being a second variable that has to agree with the first. Deliberately not `~/dockers`, where a seasoned user already keeps their own estate — 47 services on this machine. That separation buys two things. Containers the app store created are distinguishable from the user's own structurally, rather than by a naming convention we would have to enforce and they could break. And we never reason about someone else's compose files: the store does not scan, adopt or modify anything outside its own directory. That also simplifies "I already have one of these" — it is answered by the user giving a URL, never by us finding a directory and guessing whose it is. An earlier draft had the installer adopting existing directories, which meant reading, and potentially writing over, services Officer did not create. This development machine predates the convention and derives an ugly-but-correct path, since the project sits inside ~/dockers/officer.dev. Still isolated, still one root. New installs get the clean shape. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 37 ++++++++++++++------- src/servers/app-store/paths.ts | 60 ++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 11 deletions(-) create mode 100644 src/servers/app-store/paths.ts diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index 07e150ab..42877d61 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -60,12 +60,11 @@ for someone who does not. The prompt is the fork. **Officer is the installer, never the owner.** Concretely: -- A real compose file per service, written into a **user-owned directory**, from our template — - following the convention the owner already uses for 47 services in `~/dockers/`: - one directory per service, `docker-compose.yaml` inside, and **relative bind mounts** - (`./data`, `./database`, `./storage`) so configuration and data sit beside the compose file where - both we and the user can find them. Named volumes are used by 3 of those 47 and are the exception; - templates use bind mounts, always. +- A real compose file per service, written into **`/dockers//`**, from our template — using + the convention the owner already applies to 47 services: one directory per service, + `docker-compose.yaml` inside, and **relative bind mounts** (`./data`, `./database`, `./storage`) so + configuration and data sit beside the compose file where both we and a human can see them. Named + volumes are used by 3 of those 47 and are the exception; templates use bind mounts, always. - Started with `docker compose up -d` **as the owner**, not as officer's own identity. - Found again by **label** (`officer.sidecar=`), not by holding a handle. @@ -82,11 +81,27 @@ health checks already correct, so "install Gitea" does not become a tutorial. **`USER_UID` / `USER_GID` are set to the owner**, as the existing services already do. That answers the "do containers run as root" question: no, and this is not a new convention — it is the one in use. -**An existing directory is evidence, not an obstacle.** `~/dockers/` already holds `gitea`, `memos`, -`immich`, `jellyfin` and `invoice_shelf`. The installer must never write into a directory that exists; -finding one is the strongest possible signal that this is the "you already have one" case, and the store -should offer to ADOPT it — read its ports out of the compose file and write the connection — rather than -provision a second copy or overwrite a running service's data. +**The app store's containers are isolated from the user's own**, and that is the point of the layout: + +``` +~/officerdev/ + platform/ the app + data/ DATA_PATH + dockers/ services the app store provisioned <- exclusively ours + capabilities/ the file-based item store +``` + +`OFFICER_ROOT` is derived as the parent of `DATA_PATH` rather than configured separately — a second +variable that must agree with the first is a second thing to get wrong. + +Deliberately **not** `~/dockers`, which is where a seasoned user already keeps their estate. Two +consequences, both wanted: + +1. Containers the app store created are distinguishable from the user's own **structurally**, not by a + naming convention we would have to enforce and they could break. +2. **We never reason about someone else's compose files.** The store does not scan, adopt or modify + anything outside its own directory. "I already have one of these" is answered by the user giving a + URL (`mode: 'existing'`) — never by us finding a directory and guessing whose it is. `[open]` Podman, for anyone wanting genuinely rootless. diff --git a/src/servers/app-store/paths.ts b/src/servers/app-store/paths.ts new file mode 100644 index 00000000..0e70d2f5 --- /dev/null +++ b/src/servers/app-store/paths.ts @@ -0,0 +1,60 @@ +import { dirname, join } from 'node:path'; +import { DATA_PATH } from '../data-path'; + +// Where the app store puts the containers it provisions. +// +// ── The install layout ── +// +// A machine that runs Officer is meant to look like this, whether the owner is seasoned or not: +// +// ~/officerdev/ +// platform/ the app +// data/ DATA_PATH — managed homes, attachments, job logs +// dockers/ services the app store provisioned <- this file +// capabilities/ the file-based item store +// +// One root, everything under it, nothing scattered. `OFFICER_ROOT` is derived from `DATA_PATH` rather +// than configured separately, because a second environment variable that must agree with the first is a +// second thing to get wrong — and on a correct install `data/` is always a direct child of the root. +// +// (This development machine predates the convention and has it inverted: the whole project sits inside +// `~/dockers/officer.dev/`, so the root derives to `officer.dev` and the app store's directory would be +// `~/dockers/officer.dev/dockers`. Which is ugly, and correct — it is still isolated, still under one +// root, and still not mixed in with anything else. New installs get the clean shape.) +// +// ── Why this is not `~/dockers` ── +// +// That is where a seasoned user already keeps their own estate — 47 services on this machine alone. Two +// reasons to stay out of it: +// +// 1. **Isolation.** Containers the app store created and containers the user manages must be +// distinguishable without inspecting them. A separate root makes that structural rather than a +// naming convention we would have to enforce and they could break. +// 2. **We never reason about someone else's compose files.** The app store does not scan, adopt or +// modify anything outside its own directory. "I already have one of these" is answered by the user +// giving a URL (`mode: 'existing'`), never by us finding a directory and guessing it is theirs. +// +// So this directory is exclusively ours to write, and everything in it was put there by an install. + +/** + * The install root — the parent of `data/`. On a conventional install, `~/officerdev`. + * + * Derived, not configured: see above. + */ +export const OFFICER_ROOT = dirname(DATA_PATH); + +/** Where provisioned services live, one directory each. Created on first install, not at boot. */ +export const DOCKERS_DIR = join(OFFICER_ROOT, 'dockers'); + +/** + * This service's own directory: `/dockers//`, holding `docker-compose.yaml` and — because the + * templates use relative bind mounts rather than named volumes — its data and configuration too. + * + * That is the convention the owner already uses everywhere: `./data`, `./database`, `./storage` beside + * the compose file, so both the app and a human can see exactly what a service is keeping and where. + * A named volume hides it behind `docker volume inspect`, which is the opposite of the point. + */ +export const serviceDir = (sidecarId: string): string => join(DOCKERS_DIR, sidecarId); + +/** The compose file the installer renders and `docker compose` is run against. */ +export const composeFile = (sidecarId: string): string => join(serviceDir(sidecarId), 'docker-compose.yaml'); From 7359867f7f531fbbab5f0bae8be6e447db8405b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 12:35:36 +0000 Subject: [PATCH 03/99] app store: check the host before writing anything MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An install that discovers a missing dependency halfway through has already made a directory, possibly started a container and written a row, and then has to unwind — leaving the user with something that neither works nor uninstalls. A 30ms check first is worth most of that. Verified while writing this: nothing in scripts/ installs Docker, and nothing checks for it. setup-dockers.sh invokes `docker compose` with no preflight, so a fresh host without Docker fails partway through setup with a bare "command not found". Recorded in the design doc rather than fixed here — the intended fix is a setup.sh per sidecar, which is also what a sidecar needs once it ships from its own repository. `docker compose version` is the probe, not `docker --version`: the latter passes with a dead daemon, which is the failure people actually hit. "Not installed" and "daemon unreachable" are reported separately because the remedies differ. Checked per MODE, not per entry. A host without Docker can still install Photos by pointing at an Immich somewhere else; refusing the whole entry is the over-strict check that makes people work around the installer instead of using it. Dropped `requires: 'docker'` from the catalogue type. Needing Docker is exactly "this entry can provision", which `modes` already says, so declaring it twice invites the two to disagree. Derived by needsDocker instead, and a test asserts the derivation matches every entry. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 26 ++++++++ src/servers/app-store/catalogue.ts | 38 ++++++++--- src/servers/app-store/preflight.test.ts | 22 +++++++ src/servers/app-store/preflight.ts | 84 +++++++++++++++++++++++++ 4 files changed, 161 insertions(+), 9 deletions(-) create mode 100644 src/servers/app-store/preflight.test.ts create mode 100644 src/servers/app-store/preflight.ts diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index 42877d61..015d0875 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -105,6 +105,32 @@ consequences, both wanted: `[open]` Podman, for anyone wanting genuinely rootless. +### Docker is assumed, and nothing guarantees it + +Verified: **nothing in `scripts/` installs Docker, and nothing checks for it.** `setup.sh` calls +`setup-dockers.sh`, which invokes `docker compose` with no preflight, so a fresh host without Docker +fails partway through setup with a bare "command not found". + +That is the seam where this project's origin shows — it began as one person's own machine, provisioned +by his own scripts, where Docker was simply always there. + +The intended fix is **a `setup.sh` per sidecar**, ensuring its own dependencies before its compose file +is used. That is also the shape a sidecar needs once it lives in its own repository, so a sidecar package +becomes: + +``` +metadata (catalogue entry) · compose template · setup.sh · schema +``` + +Until that exists, the app store **detects and reports** rather than guessing or half-installing: +`preflight.ts` checks `docker compose version` — which exercises the binary, the daemon connection and +the plugin in one call, unlike `docker --version`, which passes with a dead daemon — and distinguishes +"not installed" from "daemon unreachable", because the remedies differ. + +The check is **per mode, not per entry**: a host without Docker can still install Photos by pointing at +an Immich elsewhere. Refusing the whole entry would be the over-strict check that makes people work +around the installer instead of using it. + --- ## Install state diff --git a/src/servers/app-store/catalogue.ts b/src/servers/app-store/catalogue.ts index acb43b83..4e082f3a 100644 --- a/src/servers/app-store/catalogue.ts +++ b/src/servers/app-store/catalogue.ts @@ -61,10 +61,13 @@ export type CatalogueEntry = { /** Name of the compose template under `app-store/templates/`. Required iff `modes` includes 'provisioned'. */ composeTemplate?: string; /** - * Why this cannot be installed on some hosts, if so. Shown instead of the install button rather than - * failing halfway through — a check the installer can make before it starts. + * A host requirement that is NOT derivable from `modes`. Shown instead of the install button rather + * than failing halfway through. + * + * Docker deliberately does not appear here: needing it is exactly "this entry can provision", which + * `modes` already says. `preflight.needsDocker` derives it, so the two cannot disagree. */ - requires?: 'docker' | 'linux-display'; + requires?: 'linux-display'; }; export const CATALOGUE: CatalogueEntry[] = [ @@ -79,7 +82,13 @@ export const CATALOGUE: CatalogueEntry[] = [ composeTemplate: 'immich', existingFields: [ { key: 'url', label: 'Immich URL', type: 'url', required: true, placeholder: 'https://photos.example.com' }, - { key: 'secret', label: 'API key', type: 'secret', required: true, help: 'Immich → Account Settings → API Keys. Create it with all permissions: a scoped key returns 403 per route, which reads as a broken feature.' }, + { + key: 'secret', + label: 'API key', + type: 'secret', + required: true, + help: 'Immich → Account Settings → API Keys. Create it with all permissions: a scoped key returns 403 per route, which reads as a broken feature.', + }, ], }, { @@ -150,8 +159,21 @@ export const CATALOGUE: CatalogueEntry[] = [ composeTemplate: 'transmission', existingFields: [ { key: 'url', label: 'Transmission URL', type: 'url', required: true, placeholder: 'http://localhost:9091' }, - { key: 'path', label: 'RPC path', type: 'text', required: false, placeholder: '/transmission/rpc', help: 'Only differs behind a reverse proxy.' }, - { key: 'username', label: 'RPC username', type: 'text', required: false, help: 'Usually blank — Transmission is normally run with no RPC auth.' }, + { + key: 'path', + label: 'RPC path', + type: 'text', + required: false, + placeholder: '/transmission/rpc', + help: 'Only differs behind a reverse proxy.', + }, + { + key: 'username', + label: 'RPC username', + type: 'text', + required: false, + help: 'Usually blank — Transmission is normally run with no RPC auth.', + }, { key: 'secret', label: 'RPC password', type: 'secret', required: false }, ], }, @@ -210,9 +232,7 @@ export const CATALOGUE: CatalogueEntry[] = [ summary: 'Index and play the library on this machine', modes: ['config'], capability: 'music', - configFields: [ - { key: 'library', label: 'Music folder', type: 'text', required: true, placeholder: '~/Music' }, - ], + configFields: [{ key: 'library', label: 'Music folder', type: 'text', required: true, placeholder: '~/Music' }], }, { id: 'wallet', diff --git a/src/servers/app-store/preflight.test.ts b/src/servers/app-store/preflight.test.ts new file mode 100644 index 00000000..17af8d27 --- /dev/null +++ b/src/servers/app-store/preflight.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'bun:test'; +import { needsDocker, preflight } from './preflight'; +import { CATALOGUE, byId } from './catalogue'; + +describe('needsDocker is derived, not declared', () => { + it('is true for exactly the entries that can provision', () => { + for (const e of CATALOGUE) expect(needsDocker(e)).toBe(e.modes.includes('provisioned')); + }); +}); + +describe('preflight is per mode, not per entry', () => { + it('allows pointing at an existing instance without Docker', async () => { + // The whole point: a host with no Docker can still use Photos against an Immich elsewhere. + // Refusing the entry outright is the over-strict check that makes people bypass the installer. + const photos = byId('photos')!; + expect(await preflight(photos, 'existing')).toEqual({ ok: true }); + }); + + it('allows a config-only sidecar regardless', async () => { + expect(await preflight(byId('email')!, 'config')).toEqual({ ok: true }); + }); +}); diff --git a/src/servers/app-store/preflight.ts b/src/servers/app-store/preflight.ts new file mode 100644 index 00000000..2634f6d8 --- /dev/null +++ b/src/servers/app-store/preflight.ts @@ -0,0 +1,84 @@ +import type { CatalogueEntry } from './catalogue'; + +// Can this machine install this sidecar at all — asked BEFORE anything is written. +// +// The point is the ordering. An install that discovers a missing dependency halfway through has already +// created a directory, possibly started a container, and written a row; it then has to unwind, and the +// user is left with something that neither works nor uninstalls. A check that costs 30ms up front is +// worth a great deal of that. +// +// ── What this deliberately does NOT do ── +// +// It does not install anything. Today nothing in `scripts/` installs Docker either — `setup.sh` runs +// `setup-dockers.sh`, which invokes `docker compose` without ever checking it exists, so a fresh host +// without Docker fails partway through setup with a bare "command not found". That is a real gap, and +// the intended fix is a per-sidecar `setup.sh` that ensures its own dependencies — which is also the +// shape a sidecar needs once it lives in its own repository and ships independently. +// +// Until that exists, the honest thing is to detect and report rather than to guess or to half-install. + +export type Preflight = + | { ok: true } + | { ok: false; reason: string; /** What the user has to do about it. */ remedy: string }; + +/** + * Docker is needed to PROVISION, never to point at something already running. Derived from `modes` + * rather than declared per entry, so the two cannot drift: an entry that can provision needs Docker, by + * definition, and nobody has to remember to tick a second box. + */ +export const needsDocker = (entry: CatalogueEntry): boolean => entry.modes.includes('provisioned'); + +/** `docker` on PATH, the daemon reachable, and the compose plugin present. All three, or it is not usable. */ +export async function checkDocker(): Promise { + try { + // `docker compose version` exercises the binary, the daemon connection and the plugin in one call. + // `docker --version` would pass with a dead daemon, which is the failure people actually hit. + const proc = Bun.spawn(['docker', 'compose', 'version'], { stdout: 'pipe', stderr: 'pipe' }); + const code = await proc.exited; + if (code === 0) return { ok: true }; + + const err = (await new Response(proc.stderr).text()).trim(); + // The daemon being down and the plugin being absent need different remedies, and the message is the + // only way to tell them apart — the exit code is 1 for both. + if (/permission denied|daemon|cannot connect/i.test(err)) { + return { + ok: false, + reason: 'The Docker daemon is not reachable.', + remedy: + 'Start Docker (`sudo systemctl start docker`), or add your user to the `docker` group and log in again.', + }; + } + return { + ok: false, + reason: 'Docker Compose is not available.', + remedy: 'Install the Docker Compose plugin (`docker-compose-plugin`).', + }; + } catch { + return { + ok: false, + reason: 'Docker is not installed.', + remedy: 'Install Docker Engine, then try again. https://docs.docker.com/engine/install/', + }; + } +} + +/** + * Everything that must be true before `mode` can be attempted for `entry`. + * + * Split by mode on purpose: a host with no Docker can still install Photos by pointing at an Immich + * somewhere else. Refusing the whole entry would be wrong, and is the sort of over-strict check that + * makes people work around the installer instead of using it. + */ +export async function preflight(entry: CatalogueEntry, mode: string): Promise { + if (entry.requires === 'linux-display' && process.platform !== 'linux') { + return { + ok: false, + reason: `${entry.label} mirrors an Xorg display, which this platform does not have.`, + remedy: 'This sidecar only runs on a Linux host with a display.', + }; + } + + if (mode === 'provisioned' && needsDocker(entry)) return checkDocker(); + + return { ok: true }; +} From 890f57a7a6a3e39a74a5baf2b7f3d455c9bfc6f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 12:44:47 +0000 Subject: [PATCH 04/99] app store: compose templates and their setup scripts, with two proven end to end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each provisionable service gets a directory holding a compose template and a setup.sh. Deliberately the shape a sidecar needs once it lives in its own repository: metadata, compose, setup script, schema. The contract (templates/README.md): answers come from the ENVIRONMENT, so the web form fills them in and a person on a VPS is prompted only for what is missing, and only on a TTY — one script for both, not two code paths. Idempotent, writes only inside its own directory, streams progress on stdout (the installer pipes it to a terminal panel), and returns results as OFFICER_RESULT_= lines so nothing has to scrape a log. House conventions throughout: relative bind mounts so data sits beside the compose file rather than hiding behind `docker volume inspect`, containers running as the installing user so downloads are not root-owned, loopback-only ports unless the service's whole job is inbound connections, and no external networks — the owner's own composes attach to an `nginx` network that a fresh VPS does not have. Transmission verified end to end on this machine, on non-conflicting ports, then torn down: renders, starts, waits, reports. Its health check accepts 409 because Transmission rejects the first request by design — only-200 would have waited out the full timeout against a working daemon. Re-run produced exactly one container, and files landed owned by the user rather than root. Vaultwarden covers the case where we GENERATE the credential rather than asking for one. An existing token is reused, never rotated, because rotating during a resumed install would lock the owner out of the admin page. The Argon2 hash has its `$` doubled or compose interpolation mangles it. The token is not returned to the platform at all — the vault sidecar proxies the Bitwarden protocol and never needs it, and a secret we do not hold is one we cannot leak. Corrects the design doc, which assumed provisioning always knows the connection. Three shapes: we set the credential, we generate it, or a human must mint it in the service's UI afterwards (Immich, Jellyfin, Memos). The third makes "provisioned and running but not yet connected" a real state rather than a failure. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 19 ++++ src/servers/app-store/templates/README.md | 82 +++++++++++++++++ .../transmission/docker-compose.yaml | 40 +++++++++ .../app-store/templates/transmission/setup.sh | 90 +++++++++++++++++++ .../templates/vaultwarden/docker-compose.yaml | 35 ++++++++ .../app-store/templates/vaultwarden/setup.sh | 85 ++++++++++++++++++ 6 files changed, 351 insertions(+) create mode 100644 src/servers/app-store/templates/README.md create mode 100644 src/servers/app-store/templates/transmission/docker-compose.yaml create mode 100755 src/servers/app-store/templates/transmission/setup.sh create mode 100644 src/servers/app-store/templates/vaultwarden/docker-compose.yaml create mode 100755 src/servers/app-store/templates/vaultwarden/setup.sh diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index 015d0875..8a6097dc 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -227,6 +227,25 @@ What a plugin author is promised, and bound by. To be written properly; the shap --- +## Provisioning has three shapes, not one + +This document originally said provisioning "writes the connection we already know". That is only true +some of the time, and the difference decides whether an install can finish unattended: + +1. **We set the credentials.** Passed as container environment, so the connection is known the moment it + is up. Transmission (`USER`/`PASS`), Vaultwarden (`ADMIN_TOKEN`). +2. **We generate a secret into a file.** The bind mount lets us write it before first boot, so it is + still known without asking. slskd's API key lives in its `slskd.yml`. +3. **A human must mint a token in the service's own UI after it boots.** Immich, Jellyfin and Memos all + work this way — no environment variable pre-seeds an API key. + +Shape 3 means an install can be **provisioned and running but not yet connected**. That is a real state, +not a failure: the container is up, the compose file is written, and we are waiting for a token. The +step machine stops there, and the UI asks for the key with a link to the page that mints it. Resuming +finishes the job — which is what `completedSteps` was for. + +--- + ## What Phase 0 must not foreclose Three things are coming, and each one constrains a decision that looks free today. diff --git a/src/servers/app-store/templates/README.md b/src/servers/app-store/templates/README.md new file mode 100644 index 00000000..5fd5320e --- /dev/null +++ b/src/servers/app-store/templates/README.md @@ -0,0 +1,82 @@ +# Compose templates and their setup scripts + +One directory per provisionable service. Each holds a `docker-compose.yaml` and a `setup.sh`, and +together they are everything needed to bring that service up. + +This is deliberately the shape a sidecar will need when it lives in **its own repository**: metadata +(the catalogue entry), a compose template, a setup script, and a schema. Nothing here may assume it is +being read out of this repo. + +--- + +## The setup.sh contract + +**Answers come from the environment. It never prompts when they are already there.** + +The app store collects them in a web form and passes them as environment variables. A person running it +by hand on a VPS gets prompted for anything missing, but only when stdin is a TTY — so the same script +serves both, and neither is a second code path. + +```sh +OFFICER_SERVICE_DIR=~/officerdev/dockers/transmission \ +OFFICER_UID=1000 OFFICER_GID=1000 \ +TRANSMISSION_PORT=9091 \ +bash setup.sh +``` + +Every script must: + +| Rule | Why | +|---|---| +| **Be idempotent.** Running twice must be safe and must not create a second anything. | Install is resumable; a retry after a half-failure re-runs steps that already succeeded. | +| **Never prompt when `OFFICER_NONINTERACTIVE=1`.** Fail with a clear message instead. | A prompt behind a web form is a hang with no output, which is the worst failure to diagnose. | +| **Write only inside `OFFICER_SERVICE_DIR`.** | The app store owns that directory and nothing else. The user's own estate is never touched. | +| **Emit progress on stdout.** | The installer streams it to a terminal panel in the UI, so the user watches it happen rather than staring at a spinner. | +| **Print `OFFICER_RESULT_=value` for anything the platform must store.** | How a generated secret or a resolved port gets back to `service_connections` without the installer parsing free text. | + +Exit non-zero on failure, with the reason on stderr. The installer records it in `last_error` and the +row stays `failed` rather than pretending to be installed. + +--- + +## Volumes are always relative bind mounts + +`./data`, `./config`, `./database` — never named volumes. Configuration and data sit beside the compose +file so both the platform and a human can see exactly what a service keeps and where. A named volume +hides it behind `docker volume inspect`, which is the opposite of the point. + +## Containers run as the owner + +`user: "${OFFICER_UID}:${OFFICER_GID}"`, so files a container writes are owned by the person who +installed it and not by root. This is the convention already in use across the owner's own services. + +## Ports bind to loopback unless the service genuinely needs to be reachable + +`127.0.0.1:9091:9091`, not `9091:9091`. Officer reaches these over loopback; anything published on all +interfaces is a service exposed to the network by an installer the user trusted to be careful. The +exception is a service whose whole function is inbound connections — slskd's P2P listener, for example — +and those say so in a comment. + +## No external networks + +The owner's own composes attach to an external `nginx` network that exists on his machine. Templates +must not require one: a fresh VPS has no such network and `docker compose up` would fail before it +started. Default network only. + +--- + +## Three provisioning shapes, not one + +Worth knowing before writing a template, because the design doc originally assumed only the first: + +1. **We set the credentials.** Passed as environment to the container, so the connection is fully known + the moment it is up. Transmission (`USER`/`PASS`), Vaultwarden (`ADMIN_TOKEN`). +2. **We generate a secret into a config file.** The bind mount lets us write it before first boot, so it + is still known without asking. slskd's API key lives in its `slskd.yml`. +3. **A human must mint a token in the service's own UI after it boots.** Immich, Jellyfin and Memos all + work this way — there is no environment variable that pre-seeds an API key. + +Shape 3 means an install can be **provisioned and running, but not yet connected**. That is a real state, +not an error: the container is up, the compose file is written, and the platform is waiting for a token. +The installer stops there with the step recorded, and the UI asks for the key with a link to the page +that mints it. Resuming finishes the job. diff --git a/src/servers/app-store/templates/transmission/docker-compose.yaml b/src/servers/app-store/templates/transmission/docker-compose.yaml new file mode 100644 index 00000000..043fcb2f --- /dev/null +++ b/src/servers/app-store/templates/transmission/docker-compose.yaml @@ -0,0 +1,40 @@ +# Transmission — the daemon Officer's transmission sidecar drives over RPC. +# +# Rendered by setup.sh; ${...} are substituted before this reaches disk. Everything the container keeps +# lives beside this file, so `ls` answers "what is this service storing" without docker involved. +name: officer-transmission + +services: + transmission: + image: lscr.io/linuxserver/transmission:latest + container_name: officer-transmission + restart: unless-stopped + + # As the installing user, so completed downloads are not root-owned — the single most annoying + # thing about a container that writes to a shared folder. + user: '${OFFICER_UID}:${OFFICER_GID}' + + environment: + - PUID=${OFFICER_UID} + - PGID=${OFFICER_GID} + - TZ=${OFFICER_TZ} + # Blank means no RPC auth, which is Transmission's normal posture and is safe HERE specifically + # because the RPC port is bound to loopback below. setup.sh fills these only if the user asked for + # credentials; an empty USER is not the same as the variable being unset. + - USER=${TRANSMISSION_USER} + - PASS=${TRANSMISSION_PASS} + + ports: + # Loopback only. Officer talks to this over 127.0.0.1; nothing else has any business reaching the + # RPC endpoint, and publishing it on all interfaces would put an unauthenticated control API on the + # network because "no auth" is the default above. + - '127.0.0.1:${TRANSMISSION_PORT}:9091' + # Peer traffic, and the exception to the loopback rule: BitTorrent peers must be able to connect + # inbound or the client is crippled to outbound-only connections. + - '${TRANSMISSION_PEER_PORT}:51413' + - '${TRANSMISSION_PEER_PORT}:51413/udp' + + volumes: + - ./config:/config + - ./downloads:/downloads + - ./watch:/watch diff --git a/src/servers/app-store/templates/transmission/setup.sh b/src/servers/app-store/templates/transmission/setup.sh new file mode 100755 index 00000000..0c6fa4bf --- /dev/null +++ b/src/servers/app-store/templates/transmission/setup.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Provision Transmission for Officer. +# +# Answers come from the environment; a human running this by hand is prompted for what is missing, but +# only when stdin is a TTY. See ../README.md for the contract every one of these obeys. +# +# OFFICER_SERVICE_DIR where to write. Everything this script creates is inside it. +# OFFICER_UID/GID who the container runs as +# TRANSMISSION_PORT RPC port on loopback (default 9091) +# TRANSMISSION_PEER_PORT BitTorrent listen port (default 51413) +# TRANSMISSION_USER/PASS optional RPC credentials (default: none, loopback-only) +# +# Idempotent: safe to re-run, which is what makes a resumed install work rather than duplicate. + +set -euo pipefail + +SERVICE_DIR="${OFFICER_SERVICE_DIR:?OFFICER_SERVICE_DIR is required}" +TEMPLATE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +OFFICER_UID="${OFFICER_UID:-$(id -u)}" +OFFICER_GID="${OFFICER_GID:-$(id -g)}" +OFFICER_TZ="${OFFICER_TZ:-$(cat /etc/timezone 2>/dev/null || echo UTC)}" + +# ── Asking ──────────────────────────────────────────────────────────────────────────────────────────── +# A prompt behind a web form is a hang with no output. When the caller says it cannot answer, fail with +# the reason instead of blocking forever on a read nobody will ever satisfy. +ask() { + local var="$1" prompt="$2" default="${3:-}" + local current="${!var:-}" + if [ -n "$current" ]; then return 0; fi + if [ "${OFFICER_NONINTERACTIVE:-0}" = "1" ] || [ ! -t 0 ]; then + if [ -n "$default" ]; then printf -v "$var" '%s' "$default"; return 0; fi + echo "error: $var is required and this is a non-interactive run" >&2 + exit 2 + fi + local answer + read -r -p "$prompt${default:+ [$default]}: " answer + printf -v "$var" '%s' "${answer:-$default}" +} + +ask TRANSMISSION_PORT 'Transmission RPC port (loopback only)' '9091' +ask TRANSMISSION_PEER_PORT 'BitTorrent peer port' '51413' +TRANSMISSION_USER="${TRANSMISSION_USER:-}" +TRANSMISSION_PASS="${TRANSMISSION_PASS:-}" + +# ── Render ──────────────────────────────────────────────────────────────────────────────────────────── +echo "==> Preparing $SERVICE_DIR" +mkdir -p "$SERVICE_DIR/config" "$SERVICE_DIR/downloads" "$SERVICE_DIR/watch" + +# envsubst with an explicit variable list, never the bare form: unrestricted envsubst would also expand +# anything in the template that merely looks like a variable, and a compose file is full of $ that +# belongs to other tools. +export OFFICER_UID OFFICER_GID OFFICER_TZ TRANSMISSION_PORT TRANSMISSION_PEER_PORT TRANSMISSION_USER TRANSMISSION_PASS +envsubst '${OFFICER_UID} ${OFFICER_GID} ${OFFICER_TZ} ${TRANSMISSION_PORT} ${TRANSMISSION_PEER_PORT} ${TRANSMISSION_USER} ${TRANSMISSION_PASS}' \ + < "$TEMPLATE_DIR/docker-compose.yaml" > "$SERVICE_DIR/docker-compose.yaml" + +echo "==> Starting the container" +# `up -d` is already idempotent: an unchanged compose file against a running container is a no-op, and a +# changed one recreates. That is the whole reason a re-run is safe. +docker compose --project-directory "$SERVICE_DIR" up -d + +# ── Wait ────────────────────────────────────────────────────────────────────────────────────────────── +# Reporting success the moment `up -d` returns would be a lie: the container exists, the RPC endpoint is +# not listening yet, and the platform's first call would fail against a service we just said was ready. +echo "==> Waiting for the RPC endpoint" +for i in $(seq 1 60); do + # 409 is the correct healthy answer here — Transmission demands a session id and rejects the first + # request by design. Treating only 200 as healthy would wait out the full timeout on a working daemon. + code="$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:${TRANSMISSION_PORT}/transmission/rpc" || true)" + if [ "$code" = "409" ] || [ "$code" = "200" ] || [ "$code" = "401" ]; then + echo " up after ${i}s (HTTP $code)" + break + fi + if [ "$i" = "60" ]; then + echo "error: Transmission did not answer on 127.0.0.1:${TRANSMISSION_PORT} within 60s" >&2 + echo " check: docker compose --project-directory '$SERVICE_DIR' logs" >&2 + exit 1 + fi + sleep 1 +done + +# ── Hand the connection back ────────────────────────────────────────────────────────────────────────── +# Shape 1 from the README: we set the credentials, so nothing has to be asked for after the fact. These +# lines are the installer's only interface to this script's results — parsed by prefix, never by +# scraping the log above. +echo "OFFICER_RESULT_URL=http://127.0.0.1:${TRANSMISSION_PORT}" +echo "OFFICER_RESULT_PATH=/transmission/rpc" +echo "OFFICER_RESULT_USERNAME=${TRANSMISSION_USER}" +echo "OFFICER_RESULT_SECRET=${TRANSMISSION_PASS}" +echo "==> Done" diff --git a/src/servers/app-store/templates/vaultwarden/docker-compose.yaml b/src/servers/app-store/templates/vaultwarden/docker-compose.yaml new file mode 100644 index 00000000..bcb31d3d --- /dev/null +++ b/src/servers/app-store/templates/vaultwarden/docker-compose.yaml @@ -0,0 +1,35 @@ +# Vaultwarden — the Bitwarden-compatible server behind Officer's vault sidecar. +# +# Every request reaches it through us: app → /api/vault → officer-vault → here. Nothing else should be +# able to, which is why the port below is loopback-only. +name: officer-vault + +services: + vaultwarden: + image: vaultwarden/server:latest + container_name: officer-vault + restart: unless-stopped + + user: '${OFFICER_UID}:${OFFICER_GID}' + + environment: + - TZ=${OFFICER_TZ} + # Argon2 hash of a token setup.sh generated. The plaintext is printed once, to the installer, and + # never written to disk here — a compose file is not a secret store, and this one sits in a + # directory the user is encouraged to read. + - ADMIN_TOKEN=${VAULTWARDEN_ADMIN_TOKEN_HASH} + # Closed by default. An open Vaultwarden on a machine the owner just handed a password manager to + # is the wrong default at every scale, and the owner can open it from the admin page if they want. + - SIGNUPS_ALLOWED=false + # WebSocket notifications, so the Bitwarden clients get live sync rather than polling. + - WEBSOCKET_ENABLED=true + + ports: + # Loopback only, always. This is a password vault: the only thing that should reach it is the + # sidecar on this host, and publishing it on all interfaces would put it on the network. + - '127.0.0.1:${VAULTWARDEN_PORT}:80' + + volumes: + # Holds the SQLite database, the attachments and the RSA keys. Bind-mounted rather than a named + # volume so the owner can see — and back up — exactly where their passwords live. + - ./data:/data diff --git a/src/servers/app-store/templates/vaultwarden/setup.sh b/src/servers/app-store/templates/vaultwarden/setup.sh new file mode 100755 index 00000000..9fb8c5fc --- /dev/null +++ b/src/servers/app-store/templates/vaultwarden/setup.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# Provision Vaultwarden for Officer. +# +# Shape 2 from ../README.md: nothing is asked for, because the one credential that matters is GENERATED +# here. There is no sensible way for a user to invent an admin token, and asking for one produces a +# weaker secret than `openssl rand` does. +# +# OFFICER_SERVICE_DIR where to write +# OFFICER_UID/GID who the container runs as +# VAULTWARDEN_PORT loopback port (default 8222) +# +# Idempotent, including the token: an existing one is REUSED rather than rotated, because rotating on a +# re-run would lock the owner out of the admin page during a routine resumed install. + +set -euo pipefail + +SERVICE_DIR="${OFFICER_SERVICE_DIR:?OFFICER_SERVICE_DIR is required}" +TEMPLATE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +OFFICER_UID="${OFFICER_UID:-$(id -u)}" +OFFICER_GID="${OFFICER_GID:-$(id -g)}" +OFFICER_TZ="${OFFICER_TZ:-$(cat /etc/timezone 2>/dev/null || echo UTC)}" +VAULTWARDEN_PORT="${VAULTWARDEN_PORT:-8222}" + +command -v docker >/dev/null || { echo "error: docker is not installed" >&2; exit 2; } +command -v openssl >/dev/null || { echo "error: openssl is required to generate the admin token" >&2; exit 2; } + +mkdir -p "$SERVICE_DIR/data" + +# ── The admin token ─────────────────────────────────────────────────────────────────────────────────── +# Kept in a 0600 file inside the service directory rather than only in the database, so the owner can +# still reach the admin page if Officer is down — which is exactly when they might need to. +TOKEN_FILE="$SERVICE_DIR/.admin-token" + +if [ -f "$TOKEN_FILE" ]; then + echo "==> Reusing the existing admin token" + VAULTWARDEN_ADMIN_TOKEN="$(cat "$TOKEN_FILE")" +else + echo "==> Generating an admin token" + VAULTWARDEN_ADMIN_TOKEN="$(openssl rand -base64 48 | tr -d '\n')" + ( umask 077; printf '%s' "$VAULTWARDEN_ADMIN_TOKEN" > "$TOKEN_FILE" ) +fi + +# Vaultwarden accepts a plaintext token but warns loudly and recommends an Argon2 hash; `vaultwarden +# hash` does not exist as a standalone binary, so the hash is produced by the image itself. Falling back +# to plaintext rather than failing: a working install with a warning beats no install at all, and the +# token is only reachable over loopback. +echo "==> Hashing it" +if VAULTWARDEN_ADMIN_TOKEN_HASH="$(printf '%s' "$VAULTWARDEN_ADMIN_TOKEN" \ + | docker run --rm -i vaultwarden/server:latest /vaultwarden hash --preset owasp 2>/dev/null \ + | grep -oE '\$argon2[^ ]*' | head -1)" && [ -n "$VAULTWARDEN_ADMIN_TOKEN_HASH" ]; then + # Compose reads `$` as interpolation, so a literal Argon2 hash must have every `$` doubled or the + # container receives a mangled token and rejects every admin login with no useful error. + VAULTWARDEN_ADMIN_TOKEN_HASH="${VAULTWARDEN_ADMIN_TOKEN_HASH//\$/\$\$}" +else + echo " (could not hash — falling back to a plaintext token, which Vaultwarden will warn about)" + VAULTWARDEN_ADMIN_TOKEN_HASH="$VAULTWARDEN_ADMIN_TOKEN" +fi + +# ── Render and start ────────────────────────────────────────────────────────────────────────────────── +echo "==> Preparing $SERVICE_DIR" +export OFFICER_UID OFFICER_GID OFFICER_TZ VAULTWARDEN_PORT VAULTWARDEN_ADMIN_TOKEN_HASH +envsubst '${OFFICER_UID} ${OFFICER_GID} ${OFFICER_TZ} ${VAULTWARDEN_PORT} ${VAULTWARDEN_ADMIN_TOKEN_HASH}' \ + < "$TEMPLATE_DIR/docker-compose.yaml" > "$SERVICE_DIR/docker-compose.yaml" + +echo "==> Starting the container" +docker compose --project-directory "$SERVICE_DIR" up -d + +echo "==> Waiting for Vaultwarden to answer" +for i in $(seq 1 60); do + code="$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:${VAULTWARDEN_PORT}/alive" || true)" + if [ "$code" = "200" ]; then echo " up after ${i}s"; break; fi + if [ "$i" = "60" ]; then + echo "error: Vaultwarden did not answer on 127.0.0.1:${VAULTWARDEN_PORT} within 60s" >&2 + echo " check: docker compose --project-directory '$SERVICE_DIR' logs" >&2 + exit 1 + fi + sleep 1 +done + +# The URL is all the platform stores. The admin token is deliberately NOT returned: the vault sidecar +# proxies the Bitwarden protocol and never needs it, and a secret the platform does not hold is a secret +# it cannot leak. It is in $TOKEN_FILE for the owner. +echo "OFFICER_RESULT_URL=http://127.0.0.1:${VAULTWARDEN_PORT}" +echo "==> Done. Admin token: $TOKEN_FILE (0600, not stored by Officer)" From ee382578569f7855b3ced571b70a48651d1bfc89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 12:57:04 +0000 Subject: [PATCH 05/99] app store: make member provisioning a mechanism, not an install-time loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner installs, but a server may already have members, and a member added next month needs the same work. So the unit is (service × member) reachable from two triggers — install a service, provision existing members; add a member, provision installed services — rather than a loop inside the installer. Only handling the first works on day one and rots. No new table. A member is provisioned exactly when they hold a service_connections row: their own credential, url NULL, inheriting the instance from the owner's. That schema anticipated this before this existed, and a second record of the same fact would only be able to disagree with the first. Three outcomes, declared per catalogue entry so the installer never special-cases a service. `accounts` is fully transparent. `none` is a single-tenant daemon with nothing to do — filtered before the provisioning loop so callers can tell "nothing to do" from "did nothing", which look identical at a call site and matter when someone is asking why a member cannot see a feature. `invite` is not a weaker `accounts`, it is the correct outcome: Vaultwarden derives its encryption key from the master password, so a credential we could mint would mean a vault we could read. Transparent right up to where being transparent would be a defect. The per-service work is an interface implemented beside each sidecar rather than a switch in core — a central function growing a case per service is what would stop any of this shipping from its own repository. Implementations must be idempotent, since both triggers can fire for the same pair and a duplicate account upstream is not ours to undo. Deprovision is optional and defaults to leaving the upstream account alone: deleting an Immich user deletes their photos. Written assuming the vault's multi-user adaptation has landed. Today /api/vault is owner-only by an explicit ownerGate, so a member is refused before Vaultwarden is reached — verified, and out of scope. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 43 +++++++++++ src/servers/app-store/catalogue.test.ts | 20 +++++ src/servers/app-store/catalogue.ts | 32 ++++++++ src/servers/app-store/members.test.ts | 24 ++++++ src/servers/app-store/members.ts | 99 +++++++++++++++++++++++++ 5 files changed, 218 insertions(+) create mode 100644 src/servers/app-store/members.test.ts create mode 100644 src/servers/app-store/members.ts diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index 8a6097dc..3a98cdec 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -246,6 +246,49 @@ finishes the job — which is what `completedSteps` was for. --- +## Members get their own accounts + +The owner installs, but a server may already have members — and a member added next month needs the same +work done. So the unit is **(service × member)**, reachable from two triggers: + +``` +install a service -> provision every member who already exists +add a member -> provision every service already installed +``` + +Only handling the first is the classic thing that works on day one and rots quietly. There is no new +table: a member is provisioned for a service exactly when they hold a `service_connections` row for it — +their own credential, `url` NULL, inheriting the instance from the owner's. That schema was built for +this before this existed. + +Three outcomes, declared per catalogue entry as `members`, so the installer never special-cases a +service: + +| | Meaning | Services | +| ---------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | +| `accounts` | Admin API creates the user **and** mints a credential. Fully transparent — the member just finds it working. | Immich, Jellyfin, Memos, InvoiceShelf, CalDAV | +| `invite` | The account can be created; a usable credential cannot. The member sets their own password. | Vaultwarden | +| `none` | Single-tenant daemon, no user concept. Access is mediated by Officer alone. | Transmission, slskd, headscale, email, music, wallet, notify, vnc | + +**`invite` is not a weaker `accounts`** — it is the correct outcome. Vaultwarden derives its encryption +key from the master password, so a credential we could mint would mean a vault we could read. Transparent +right up to the point where being transparent would be a defect. + +The per-service work is an **interface implemented beside each sidecar**, never a switch in core: a +central function growing one case per service is exactly what would stop any of this shipping from its +own repository. Implementations must be idempotent — both triggers can fire for the same pair, and +creating a second account upstream is not something we can undo. + +Deprovision is deliberately optional and defaults to doing nothing upstream. Deleting a user in Immich +deletes their photos; an app store that destroys data as a side effect of an unrelated action is worse +than one that leaves a stale account behind. + +**Assumed working:** the vault's own multi-user adaptation is being done separately. Today `/api/vault` +is owner-only by an explicit `ownerGate`, so a member is refused before Vaultwarden is reached — this +design is written as though that has landed. + +--- + ## What Phase 0 must not foreclose Three things are coming, and each one constrains a decision that looks free today. diff --git a/src/servers/app-store/catalogue.test.ts b/src/servers/app-store/catalogue.test.ts index 6f63b9ae..239dd6ee 100644 --- a/src/servers/app-store/catalogue.test.ts +++ b/src/servers/app-store/catalogue.test.ts @@ -86,3 +86,23 @@ describe('byId', () => { expect(byId('not-a-sidecar')).toBeUndefined(); }); }); + +describe('member provisioning is declared for every entry', () => { + it('declares how members get access', () => { + // Undeclared would silently mean "no account for anyone", which is invisible until a member + // reports that a feature the owner can see does nothing for them. + for (const entry of CATALOGUE) expect(['accounts', 'invite', 'none']).toContain(entry.members); + }); + + it('never promises accounts for a single-tenant daemon', () => { + // Transmission and slskd have no user concept; claiming otherwise would make the installer try to + // create accounts against an API that does not exist. + for (const id of ['transmission', 'slskd']) expect(byId(id)!.members).toBe('none'); + }); + + it('marks the vault as invite-only, because it cannot be otherwise', () => { + // Vaultwarden derives its encryption key from the master password. A credential we could mint is a + // vault we could read, so 'accounts' here would be a security defect rather than a feature. + expect(byId('vault')!.members).toBe('invite'); + }); +}); diff --git a/src/servers/app-store/catalogue.ts b/src/servers/app-store/catalogue.ts index 4e082f3a..30da888e 100644 --- a/src/servers/app-store/catalogue.ts +++ b/src/servers/app-store/catalogue.ts @@ -60,6 +60,24 @@ export type CatalogueEntry = { configFields?: ConfigField[]; /** Name of the compose template under `app-store/templates/`. Required iff `modes` includes 'provisioned'. */ composeTemplate?: string; + /** + * Whether members get their own account on this service, and how far that can be automated. + * + * 'accounts' — an admin API can create the user AND mint a credential, so provisioning is fully + * transparent: the member simply finds the feature working. Immich, Jellyfin, Gitea, + * Memos. + * 'invite' — an account can be created but a usable credential cannot, and that is a property of + * the service rather than a gap in ours. Vaultwarden is end-to-end encrypted: the + * master password derives the encryption key, so a credential we could mint would mean + * a vault we could read. The member is invited and sets their own password. + * 'none' — a single-tenant daemon with no user concept. Transmission, slskd. Access is mediated + * entirely by Officer, which is already how it works. + * + * Read at two moments, not one: when the service is installed (for every member who already exists) + * and when a member is added (for every service already installed). Only handling the first is the + * classic thing that works on day one and silently rots. + */ + members: 'accounts' | 'invite' | 'none'; /** * A host requirement that is NOT derivable from `modes`. Shown instead of the install button rather * than failing halfway through. @@ -77,6 +95,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-photos', label: 'Photos', summary: 'Your Immich library — browse, search, upload from the phone', + members: 'accounts', modes: ['existing', 'provisioned'], capability: 'photos', composeTemplate: 'immich', @@ -96,6 +115,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-jellyfin', label: 'Jellyfin', summary: 'Films and shows, with a player that handles direct, HLS and progressive', + members: 'accounts', modes: ['existing', 'provisioned'], capability: 'jellyfin', composeTemplate: 'jellyfin', @@ -109,6 +129,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-memos', label: 'Memos', summary: 'Quick notes, tagged and searchable', + members: 'accounts', modes: ['existing', 'provisioned'], capability: 'memos', composeTemplate: 'memos', @@ -122,6 +143,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-invoiceshelf', label: 'Invoices', summary: 'InvoiceShelf — clients, estimates and invoices', + members: 'accounts', modes: ['existing', 'provisioned'], capability: 'invoices', composeTemplate: 'invoiceshelf', @@ -135,6 +157,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-vault', label: 'Vault', summary: 'Vaultwarden — passwords, reachable by the Bitwarden apps', + members: 'invite', modes: ['existing', 'provisioned'], // No capability entry exists for this one, and the reason is about CREDENTIALS, not routing. // @@ -154,6 +177,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-transmission', label: 'Transmission', summary: 'Torrents, with the daemon Officer talks to over RPC', + members: 'none', modes: ['existing', 'provisioned'], capability: 'transmission', composeTemplate: 'transmission', @@ -182,6 +206,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-slskd', label: 'Soulseek', summary: 'slskd — search and download from the Soulseek network', + members: 'none', modes: ['existing', 'provisioned'], capability: 'soulseek', composeTemplate: 'slskd', @@ -195,6 +220,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-caldav', label: 'Calendar', summary: 'Radicale — calendars and contacts over CalDAV/CardDAV', + members: 'accounts', modes: ['existing', 'provisioned'], capability: 'calendar', composeTemplate: 'radicale', @@ -205,6 +231,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-headscale', label: 'Headscale', summary: 'Your own tailnet control plane', + members: 'none', modes: ['existing'], capability: 'headscale', existingFields: [ @@ -219,6 +246,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-email', label: 'Email', summary: 'Your IMAP accounts, synced and searchable', + members: 'none', modes: ['config'], capability: 'email', // Deliberately empty: accounts are added from /email, which already has a working multi-account @@ -230,6 +258,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-music', label: 'Music', summary: 'Index and play the library on this machine', + members: 'none', modes: ['config'], capability: 'music', configFields: [{ key: 'library', label: 'Music folder', type: 'text', required: true, placeholder: '~/Music' }], @@ -239,6 +268,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-wallet', label: 'Wallet', summary: 'Bitcoin and Lightning, with keys held by the sidecar alone', + members: 'none', modes: ['config'], capability: 'wallet', configFields: [], @@ -248,6 +278,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-notify', label: 'Notifications', summary: 'Push to your phone when a job finishes or a turn needs you', + members: 'none', modes: ['config'], capability: 'notify', configFields: [], @@ -257,6 +288,7 @@ export const CATALOGUE: CatalogueEntry[] = [ process: 'officer-vnc', label: 'Desktop', summary: 'Mirror this machine’s display in the browser', + members: 'none', modes: ['config'], capability: 'desktop', // x11vnc against an Xorg display. There is nothing to mirror on a headless box or on macOS, so the diff --git a/src/servers/app-store/members.test.ts b/src/servers/app-store/members.test.ts new file mode 100644 index 00000000..e7abad9e --- /dev/null +++ b/src/servers/app-store/members.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from 'bun:test'; +import { plannedOutcome, servicesNeedingMemberProvision } from './members'; +import { byId } from './catalogue'; + +describe('what adding a member would do', () => { + it('predicts the outcome per service without contacting anything', () => { + expect(plannedOutcome(byId('photos')!)).toBe('provisioned'); + expect(plannedOutcome(byId('vault')!)).toBe('invited'); + expect(plannedOutcome(byId('transmission')!)).toBe('not-applicable'); + }); +}); + +describe('which installed services have work to do', () => { + it('skips single-tenant daemons entirely', () => { + const ids = servicesNeedingMemberProvision(['photos', 'transmission', 'slskd', 'vault']).map((e) => e.id); + expect(ids.sort()).toEqual(['photos', 'vault']); + }); + + it('ignores services that are not installed', () => { + // The catalogue is what COULD be installed; only what IS installed has a member to provision. + expect(servicesNeedingMemberProvision([])).toEqual([]); + expect(servicesNeedingMemberProvision(['jellyfin']).map((e) => e.id)).toEqual(['jellyfin']); + }); +}); diff --git a/src/servers/app-store/members.ts b/src/servers/app-store/members.ts new file mode 100644 index 00000000..c775ee87 --- /dev/null +++ b/src/servers/app-store/members.ts @@ -0,0 +1,99 @@ +import type { CatalogueEntry } from './catalogue'; +import { CATALOGUE } from './catalogue'; + +// Giving every member their own account on the services the owner installed. +// +// ── Why this is not part of install ── +// +// It has two triggers, and only ever handling the first is how this rots: +// +// install a service → provision every member who already exists +// add a member → provision every service already installed +// +// A member added next month to an Immich installed today needs exactly the same work to happen, and +// nobody will remember to do it by hand. So the unit is (service × member), reachable from both sides, +// rather than a loop inside the installer. +// +// ── What "provisioned" means, and where it is recorded ── +// +// There is no new table. A member is provisioned for a service exactly when they have a +// `service_connections` row for it — which already carries their own credential and a NULL `url`, +// inheriting the instance from the owner's row. That schema was built for this before this existed, and +// adding a second record of the same fact would only create the chance for the two to disagree. +// +// ── The ceiling ── +// +// `members: 'invite'` is not a weaker version of `'accounts'`; it is a different outcome. Vaultwarden +// derives its encryption key from the master password, so a credential we could mint would mean a vault +// we could read. The account is created and the member sets their own password. Transparent right up to +// the point where being transparent would be a defect. + +export type MemberProvisionOutcome = + /** Account exists and a credential was written. The member finds the feature working. */ + | { status: 'provisioned' } + /** Account exists; the member must complete it themselves. Carries where to send them. */ + | { status: 'invited'; completeAt: string } + /** Nothing to do — a single-tenant daemon. Not a failure. */ + | { status: 'not-applicable' } + /** Upstream refused. The caller records it; the member simply has no access yet. */ + | { status: 'failed'; error: string }; + +/** + * What provisioning this member for this service would do — without doing it. + * + * Split out so the UI can say "adding Ana will give her Photos and Jellyfin, and invite her to the + * vault" BEFORE anyone commits, and so the decision is testable without an Immich to talk to. + */ +export function plannedOutcome(entry: CatalogueEntry): MemberProvisionOutcome['status'] { + switch (entry.members) { + case 'accounts': + return 'provisioned'; + case 'invite': + return 'invited'; + case 'none': + return 'not-applicable'; + } +} + +/** + * Every installed service that has anything to do for a new member. + * + * `'none'` entries are filtered out here rather than inside the loop that provisions, so the caller can + * distinguish "nothing to do" from "did nothing" — the two look identical at a call site and mean very + * different things when someone is debugging why a member cannot see a feature. + */ +export function servicesNeedingMemberProvision(installedIds: readonly string[]): CatalogueEntry[] { + const installed = new Set(installedIds); + return CATALOGUE.filter((e) => installed.has(e.id) && e.members !== 'none'); +} + +/** + * The per-service work, to be implemented alongside each sidecar rather than centrally. + * + * Deliberately an interface and not a switch: the whole point of the app store is that a sidecar carries + * everything it needs, and a central function that grows a case per service is the thing that stops any + * of this shipping from its own repository. A third-party plugin implements this; nothing in core knows + * its name. + * + * Implementations MUST be idempotent. Both triggers can fire for the same pair — a member added while an + * install is still running is not a rare race, it is a Tuesday — and creating a second account upstream + * is not recoverable from our side. Treat "already exists" as success. + */ +export type MemberProvisioner = { + sidecarId: string; + /** + * Create (or confirm) the member's account upstream and return their credential. + * + * Returning `invited` is a normal outcome, not an error path: it means the account is real and the + * member must finish it. `completeAt` is where to send them. + */ + provision(params: { userId: number; email: string; username: string }): Promise; + /** + * Undone when a member is removed, or when the service is uninstalled with data disposal. + * + * Separate from provision because the honest default is to leave the upstream account alone: deleting + * a user in Immich deletes their photos, and an app store that silently destroys data on an unrelated + * action is worse than one that leaves a stale account behind. + */ + deprovision?(params: { userId: number }): Promise; +}; From 936b96a36e28dce8056985358dc85df033537286 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 13:08:49 +0000 Subject: [PATCH 06/99] app store: uninstall removes containers and never data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There is now no uninstall option that deletes data, rather than a careful one that does. A user uninstalling a sidecar is saying "stop running this", which is not the same sentence as "delete my photo library", and for Immich or Jellyfin getting that wrong once is unrecoverable. No confirmation dialog makes it a good default. So: `docker compose down` without `-v`. Containers and networks go; the service directory and everything under it stays exactly as it was. The bind-mount convention already makes this hard to get wrong, which is worth noting because it means the safety is structural rather than a rule someone has to keep following. Data lives on the host inside the service directory, so `-v` — which only removes NAMED volumes — could not delete it even if a future change added the flag back. `mode: 'existing'` has no disposal question at all: we did not create that service, so uninstall removes our sidecar and our rows and touches nothing else. Reclaiming disk becomes its own feature later, with the sizes in front of the user — "Photos is using 340 GB, delete it?" — as a deliberate act rather than a checkbox inside an uninstall flow. Removed two stale `down -v` references that survived the first pass, one in the schema comment and one in the design doc's table. Leftovers like those are how a rule becomes permission again. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 9 ++++----- src/databases/officer_db/src/schema/app-store.ts | 9 ++++++--- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index 3a98cdec..bb45a288 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -142,11 +142,10 @@ Two independent flags, because they answer different questions: That yields the three outcomes asked for: -| Action | Effect | -| ------------------------ | ------------------------------------------------------------------------------------ | -| **Disable** | Stop the sidecar. Container, config, schema and data all stay. Re-enable is instant. | -| **Uninstall, keep data** | Stop, remove the process. Leave container volumes and rows. | -| **Full uninstall** | Also `docker compose down -v` and drop the sidecar's tables. | +| Action | Effect | +| ------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| **Disable** | Stop the sidecar. Container, config, schema and data all stay. Re-enable is instant. | +| **Uninstall** | Stop the sidecar, `docker compose down` — containers and networks removed. **The service directory and everything in it stays.** | The middle one is the in-between; the user chooses disposal at uninstall time rather than us guessing. diff --git a/src/databases/officer_db/src/schema/app-store.ts b/src/databases/officer_db/src/schema/app-store.ts index 7f802909..f00c46cb 100644 --- a/src/databases/officer_db/src/schema/app-store.ts +++ b/src/databases/officer_db/src/schema/app-store.ts @@ -23,8 +23,11 @@ import { pgTable, serial, text, boolean, timestamp, jsonb, uniqueIndex } from 'd // SHOULD BE RUNNING. Disabling is the reversible middle ground the owner asked for — stop the process, // keep the container, the config, the tables and the data, and start again later at no cost. // -// Uninstall then has a disposal choice rather than a fixed meaning: keep the data, drop the container, -// or drop both. None of those are this table's business beyond recording that the row is gone. +// Uninstall stops the sidecar and removes the containers. It does NOT remove data, and there is no +// option that does: the service directory and everything under it survives. A user uninstalling a +// sidecar is saying "stop running this", not "delete my photo library", and the two are unrecoverably +// different for Immich and Jellyfin. Reclaiming disk is a separate, deliberate feature with the sizes +// shown — not a checkbox in an uninstall flow. export const sidecarInstalls = pgTable( 'sidecar_installs', { @@ -67,7 +70,7 @@ export const sidecarInstalls = pgTable( * Absolute path to the rendered compose directory, for `mode: 'provisioned'` only. * * Stored rather than derived because it is the user's directory and he may move it — and because - * uninstall must not guess at a path it is about to run `docker compose down -v` in. + * uninstall must not guess at a path it is about to run `docker compose down` in. */ composeDir: text('compose_dir'), installedAt: timestamp('installed_at', { withTimezone: true }), From b197c2aebadf84a41eb32697862470b825ac3ad4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 13:11:01 +0000 Subject: [PATCH 07/99] =?UTF-8?q?app=20store:=20settle=20the=20lifecycle?= =?UTF-8?q?=20=E2=80=94=20disable=20stops=20the=20container,=20uninstall?= =?UTF-8?q?=20keeps=20the=20schema?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Disable now stops the container as well as the sidecar. There is no reason to leave Immich holding memory while Photos is switched off. For mode 'existing' there is no container of ours, so disable is only the sidecar. Uninstall stops both, removes the containers, and deletes the install row. It does NOT drop the sidecar's tables — pushing back on "maybe db schema too" for the same reason volumes are kept, because it is the same category. Music favourites, the Jellyfin server registry, photos configuration and saved connections are real data, and someone uninstalling Photos is saying "stop running this", not "forget which albums I favourited". Keeping them also makes reinstall a RESTORE: uninstall in June, reinstall in August, and the configuration is still there. Dropping the schema would hand back a blank service that looks subtly broken to someone who remembers setting it up. An unused table costs a row in information_schema and nothing else. Also removes a line left stale by the previous commit, which still said the user chooses disposal at uninstall time. There is no such choice any more, and a doc that describes an option the code does not have is how the option comes back. Co-Authored-By: Claude Opus 5 --- docs/sidecar-app-store.md | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/docs/sidecar-app-store.md b/docs/sidecar-app-store.md index bb45a288..dbbff301 100644 --- a/docs/sidecar-app-store.md +++ b/docs/sidecar-app-store.md @@ -140,14 +140,32 @@ Two independent flags, because they answer different questions: - **`installed`** — the thing exists: container provisioned, config written, schema applied. - **`enabled`** — the process should be running. -That yields the three outcomes asked for: +| Action | Sidecar process | Container | Data & schema | +| ------------- | --------------- | ------------------------------ | ------------- | +| **Disable** | stopped | stopped | untouched | +| **Enable** | started | started | untouched | +| **Uninstall** | stopped | `docker compose down`, removed | untouched | -| Action | Effect | -| ------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| **Disable** | Stop the sidecar. Container, config, schema and data all stay. Re-enable is instant. | -| **Uninstall** | Stop the sidecar, `docker compose down` — containers and networks removed. **The service directory and everything in it stays.** | +Disable stops the container too — there is no reason to leave Immich holding memory while Photos is +switched off. For `mode: 'existing'` there is no container of ours, so disable is only the sidecar. -The middle one is the in-between; the user chooses disposal at uninstall time rather than us guessing. +Uninstall additionally deletes the `sidecar_installs` row. It does **not** drop the sidecar's tables. + +**Nothing above deletes data, and there is no option that does.** + +### Why the schema survives uninstall too + +Dropping a sidecar's tables is deleting data. Not media, but real: music favourites, the Jellyfin server +registry, photos configuration, saved connections. That is the same category as volumes and gets the +same answer. + +It also buys something. **Reinstall becomes restore** — uninstall Photos in June, reinstall in August, +and the configuration and favourites are still there. Drop the schema and reinstalling hands back a +blank service that looks subtly broken to someone who remembers setting it up. + +Keeping them costs nothing: an unused table is a row in `information_schema`. No queries, no memory, no +maintenance. Dropping them joins volume deletion in the later, deliberate cleanup feature, where the +user sees what they are removing. **Install must be idempotent and resumable.** Provision → health → config → schema → start is five steps and any of them can fail. The failure mode to design against is a half-installed service that neither From 4a03b9f84eef1a4742aef011b34b45823bff941f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 13:15:33 +0000 Subject: [PATCH 08/99] app store: the install step machine, resumable and testable without docker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Install spans a container start, a health wait, an upstream API call and a process start. Any can fail, and one of them — a token only a human can mint — is EXPECTED to stop the run. A straight-line function has two bad options there: unwind everything, or leave a half-installed service that neither works nor uninstalls, which is the state users cannot get out of. So each step is named, completion is persisted, and running install again resumes. planSteps is a pure function of (entry, mode) and the effects are injected, which makes ordering, resume, blocking and failure testable with no Docker, Postgres, PM2 or Immich in sight. 15 tests cover exactly the behaviour that only appears when something goes wrong. Two rules are enforced by the plan rather than remembered at call sites: 'existing' never provisions, so pointing at an instance the user already runs cannot start a container; and the members step is omitted entirely for a service with no user concept, so a Transmission install does not report a step that did nothing — which reads as a silent failure to anyone debugging a member's access. `blocked` is a first-class outcome, not an error. For Immich the container is up and healthy and only its own UI can mint a key; calling that a failure would make a normal install look broken and invite the user to tear down a working container. The blocking step is deliberately NOT recorded as complete, so a resume re-runs the step the human just answered. Results feed forward — provision discovers the URL that connect writes down two steps later — over a copy of the caller's values, so a failure halfway cannot rewrite what an earlier attempt achieved. Co-Authored-By: Claude Opus 5 --- src/servers/app-store/installer.test.ts | 210 ++++++++++++++++++++++++ src/servers/app-store/installer.ts | 182 ++++++++++++++++++++ 2 files changed, 392 insertions(+) create mode 100644 src/servers/app-store/installer.test.ts create mode 100644 src/servers/app-store/installer.ts diff --git a/src/servers/app-store/installer.test.ts b/src/servers/app-store/installer.test.ts new file mode 100644 index 00000000..d61ba769 --- /dev/null +++ b/src/servers/app-store/installer.test.ts @@ -0,0 +1,210 @@ +import { describe, expect, it } from 'bun:test'; +import { planSteps, runInstall, type InstallEffects, type StepName } from './installer'; +import { byId } from './catalogue'; + +// The whole install machine, tested without Docker, Postgres, PM2 or an Immich to talk to — which is +// the reason the effects are injected. What is being pinned here is the behaviour that only shows up +// when something goes wrong: resume, blocking, and not doing work twice. + +const photos = byId('photos')!; // existing|provisioned, members: 'accounts' +const transmission = byId('transmission')!; // existing|provisioned, members: 'none' +const email = byId('email')!; // config only + +/** Records what was actually called, so a test can assert on absence as well as presence. */ +function spyEffects(over: Partial = {}) { + const calls: string[] = []; + const effects: InstallEffects = { + preflight: async () => { + calls.push('preflight'); + return { ok: true }; + }, + provision: async () => { + calls.push('provision'); + return { results: { url: 'http://127.0.0.1:18091' }, composeDir: '/root/dockers/x' }; + }, + connect: async () => { + calls.push('connect'); + return { status: 'done' }; + }, + applySchema: async () => void calls.push('schema'), + startProcess: async () => void calls.push('process'), + provisionMembers: async () => void calls.push('members'), + ...over, + }; + return { effects, calls }; +} + +describe('planSteps', () => { + it('never provisions when pointing at an instance the user already runs', () => { + // The guarantee that matters: choosing 'existing' cannot start a container. Enforced here rather + // than remembered at each call site. + expect(planSteps(photos, 'existing')).not.toContain('provision'); + expect(planSteps(photos, 'provisioned')).toContain('provision'); + }); + + it('omits the members step for a service with no user concept', () => { + // Otherwise a Transmission install reports a members step that did nothing, which reads as a + // silent failure to anyone debugging why a member has no access. + expect(planSteps(transmission, 'provisioned')).not.toContain('members'); + expect(planSteps(photos, 'provisioned')).toContain('members'); + }); + + it('has nothing to connect for a config-only install', () => { + expect(planSteps(email, 'config')).toEqual(['preflight', 'schema', 'process']); + }); + + it('always starts with preflight', () => { + // Checking the host before anything is written is the whole reason a half-install is avoidable. + for (const mode of ['existing', 'provisioned', 'config'] as const) { + expect(planSteps(photos, mode)[0]).toBe('preflight'); + } + }); +}); + +describe('a clean run', () => { + it('executes the plan in order and reports installed', async () => { + const { effects, calls } = spyEffects(); + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, effects }); + + expect(out.status).toBe('installed'); + expect(calls).toEqual(['preflight', 'provision', 'connect', 'schema', 'process', 'members']); + }); + + it('feeds one step’s results forward to the next', async () => { + // `provision` discovers the URL that `connect` writes down two steps later. Without this the + // installer would have to ask the user for something it already knows. + let seen: Record = {}; + const { effects } = spyEffects({ + connect: async (ctx) => { + seen = { ...ctx.values }; + return { status: 'done' }; + }, + }); + + await runInstall({ entry: photos, mode: 'provisioned', values: { given: 'yes' }, effects }); + + expect(seen.url).toBe('http://127.0.0.1:18091'); + expect(seen.composeDir).toBe('/root/dockers/x'); + expect(seen.given).toBe('yes'); + }); + + it('does not mutate the caller’s values', async () => { + const values = { given: 'yes' }; + const { effects } = spyEffects(); + await runInstall({ entry: photos, mode: 'provisioned', values, effects }); + expect(values).toEqual({ given: 'yes' }); + }); +}); + +describe('resuming', () => { + it('skips what an earlier attempt already did', async () => { + // The point of persisting completedSteps: a resume must not provision a second container. + const { effects, calls } = spyEffects(); + const done: StepName[] = ['preflight', 'provision', 'connect']; + + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, completed: done, effects }); + + expect(out.status).toBe('installed'); + expect(calls).toEqual(['schema', 'process', 'members']); + expect(calls).not.toContain('provision'); + }); + + it('reports every completed step, including the ones it skipped', async () => { + const { effects } = spyEffects(); + const out = await runInstall({ + entry: photos, + mode: 'provisioned', + values: {}, + completed: ['preflight'], + effects, + }); + expect(out.completed).toEqual(['preflight', 'provision', 'connect', 'schema', 'process', 'members']); + }); +}); + +describe('blocking on a human', () => { + it('stops without failing, and does NOT mark the blocking step done', async () => { + // Immich: the container is up and healthy, and only its own UI can mint an API key. Recording + // `connect` as complete would mean a resume skipped the very step that is waiting. + const { effects, calls } = spyEffects({ + connect: async () => ({ status: 'blocked', reason: 'Needs an API key', completeAt: 'http://x/keys' }), + }); + + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, effects }); + + expect(out.status).toBe('blocked'); + if (out.status !== 'blocked') throw new Error('unreachable'); + expect(out.at).toBe('connect'); + expect(out.completeAt).toBe('http://x/keys'); + expect(out.completed).toEqual(['preflight', 'provision']); + // Everything after the block is untouched — no process started against a service we cannot reach. + expect(calls).not.toContain('process'); + }); + + it('re-runs the blocking step on resume, once the human has answered', async () => { + const { effects, calls } = spyEffects(); + const out = await runInstall({ + entry: photos, + mode: 'provisioned', + values: { secret: 'now-provided' }, + completed: ['preflight', 'provision'], + effects, + }); + + expect(out.status).toBe('installed'); + expect(calls).toContain('connect'); + }); +}); + +describe('failing', () => { + it('stops at the failing step and keeps what came before', async () => { + const { effects, calls } = spyEffects({ + applySchema: async () => { + throw new Error('relation already exists'); + }, + }); + + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, effects }); + + expect(out.status).toBe('failed'); + if (out.status !== 'failed') throw new Error('unreachable'); + expect(out.at).toBe('schema'); + expect(out.error).toBe('relation already exists'); + // Resumable: the three that worked are recorded, so a retry does not redo them. + expect(out.completed).toEqual(['preflight', 'provision', 'connect']); + expect(calls).not.toContain('process'); + }); + + it('treats a failed preflight as a failure before anything is written', async () => { + const { effects, calls } = spyEffects({ + preflight: async () => { + calls.push('preflight'); + return { ok: false, reason: 'Docker is not installed.', remedy: 'Install it.' }; + }, + }); + + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, effects }); + + expect(out.status).toBe('failed'); + if (out.status !== 'failed') throw new Error('unreachable'); + expect(out.at).toBe('preflight'); + // The remedy travels with the reason: "Docker is not installed" without "install it" is a dead end. + expect(out.error).toContain('Install it.'); + // Nothing provisioned, so there is nothing to unwind — the entire reason preflight goes first. + expect(calls).toEqual(['preflight']); + }); + + it('turns a thrown effect into a recorded failure rather than an escape', async () => { + // An effect that throws is a bug in that effect. If it escaped, the row would be stranded in + // `installing` with nothing to resume from. + const { effects } = spyEffects({ + startProcess: async () => { + throw new Error('pm2 not found'); + }, + }); + const out = await runInstall({ entry: photos, mode: 'provisioned', values: {}, effects }); + expect(out.status).toBe('failed'); + if (out.status !== 'failed') throw new Error('unreachable'); + expect(out.at).toBe('process'); + }); +}); diff --git a/src/servers/app-store/installer.ts b/src/servers/app-store/installer.ts new file mode 100644 index 00000000..2bd50a7a --- /dev/null +++ b/src/servers/app-store/installer.ts @@ -0,0 +1,182 @@ +import type { CatalogueEntry, InstallMode } from './catalogue'; +import type { Preflight } from './preflight'; + +// Installing one sidecar, as a sequence of named steps that can stop anywhere and be resumed. +// +// ── Why a step machine and not a function ── +// +// Install spans a container start, a health wait, an upstream API call and a process start. Any of them +// can fail, and one of them (a token only a human can mint) is EXPECTED to stop the run. A straight-line +// function has two bad options at that point: unwind everything, or leave the user with a half-installed +// service that neither works nor uninstalls. The second is the one people cannot get out of. +// +// So each step is named, its completion is persisted in `sidecar_installs.completed_steps`, and running +// install again resumes from where it stopped. Re-running a completed step is never necessary, but is +// also never harmful — every effect below is required to be idempotent, because the alternative is +// trusting that a crash never lands between "did the thing" and "recorded the thing". +// +// ── Why the effects are injected ── +// +// `planSteps` is pure and `runInstall` takes its side effects as an argument, so the whole machine — +// ordering, resume, blocking, failure — is testable without Docker, Postgres, PM2 or an Immich to talk +// to. The parts that genuinely touch the world stay thin enough to read. + +export type StepName = + /** Host is capable of this: Docker present for a provisioned install, display present for vnc. */ + | 'preflight' + /** Render the compose template and bring the containers up. Provisioned installs only. */ + | 'provision' + /** Write the owner's `service_connections` row — the URL and credential this install is reachable by. */ + | 'connect' + /** Apply the sidecar's own schema. */ + | 'schema' + /** Start the PM2 process. */ + | 'process' + /** Give every existing member their own account, where the service supports it. */ + | 'members'; + +export type StepResult = + | { status: 'done'; results?: Record } + /** + * Everything up to here worked and the run cannot continue without a human. + * + * A real state, not a failure: for Immich, Jellyfin and Memos the container is up and healthy and we + * are waiting for a token only their own UI can mint. Reporting this as an error would make a normal + * install look broken and invite the user to tear down a container that is working perfectly. + */ + | { status: 'blocked'; reason: string; completeAt?: string } + | { status: 'failed'; error: string }; + +/** What a step is given. Deliberately small — a step that needs more probably belongs in the sidecar. */ +export type StepContext = { + entry: CatalogueEntry; + mode: InstallMode; + /** Answers from the install form, plus anything earlier steps returned via `OFFICER_RESULT_*`. */ + values: Record; + /** Progress for the log the UI streams into a terminal panel. */ + log: (line: string) => void; +}; + +/** + * The world, as the installer touches it. Every one of these MUST be idempotent — see above. + */ +export type InstallEffects = { + preflight(entry: CatalogueEntry, mode: InstallMode): Promise; + /** Run the sidecar's setup.sh. Returns whatever it printed as `OFFICER_RESULT_=value`. */ + provision(ctx: StepContext): Promise<{ results: Record; composeDir: string }>; + /** Write the owner's connection row. `blocked` when the service can only be connected by a human. */ + connect(ctx: StepContext): Promise; + applySchema(ctx: StepContext): Promise; + startProcess(ctx: StepContext): Promise; + provisionMembers(ctx: StepContext): Promise; +}; + +/** + * Which steps this install needs, in order — a pure function of the entry and the chosen mode. + * + * Separated from running them so the UI can show what is about to happen, and so ordering is testable + * on its own. The two rules that matter: + * + * - `provision` exists only for `provisioned`. Pointing at an instance the user already runs must + * never start a container, and this is where that is guaranteed rather than remembered. + * - `members` is omitted entirely when the service has no user concept, so a Transmission install does + * not report a members step that did nothing. + */ +export function planSteps(entry: CatalogueEntry, mode: InstallMode): StepName[] { + const steps: StepName[] = ['preflight']; + if (mode === 'provisioned') steps.push('provision'); + // `config` installs have nothing to point at, so there is no connection row to write. + if (mode !== 'config') steps.push('connect'); + steps.push('schema', 'process'); + if (entry.members !== 'none') steps.push('members'); + return steps; +} + +export type InstallOutcome = + | { status: 'installed'; completed: StepName[] } + | { status: 'blocked'; completed: StepName[]; at: StepName; reason: string; completeAt?: string } + | { status: 'failed'; completed: StepName[]; at: StepName; error: string }; + +export type RunInstallParams = { + entry: CatalogueEntry; + mode: InstallMode; + values: Record; + /** Steps already done by an earlier attempt. Passing them is what makes this a resume. */ + completed?: StepName[]; + effects: InstallEffects; + log?: (line: string) => void; +}; + +/** + * Run (or resume) an install. + * + * Returns rather than throws, because every outcome here is something the caller has to record: a + * failure updates `last_error` and leaves the row resumable, and a block is a normal pause. Throwing + * would make the caller's job "catch and guess which of those happened". + */ +export async function runInstall(params: RunInstallParams): Promise { + const { entry, mode, effects } = params; + const log = params.log ?? (() => {}); + const plan = planSteps(entry, mode); + const completed = [...(params.completed ?? [])]; + // Copied, not aliased: a resumed run must not mutate the caller's record of what an earlier attempt + // achieved, or a failure halfway through would silently rewrite history. + const values = { ...params.values }; + + for (const step of plan) { + if (completed.includes(step)) { + log(`· ${step} — already done, skipping`); + continue; + } + + const ctx: StepContext = { entry, mode, values, log }; + log(`▸ ${step}`); + + try { + const result = await runStep(step, ctx, effects); + + if (result.status === 'failed') { + return { status: 'failed', completed, at: step, error: result.error }; + } + if (result.status === 'blocked') { + // Note that `completed` does NOT include this step: resuming re-runs it, which is the point — + // the human has now supplied what it was waiting for. + return { status: 'blocked', completed, at: step, reason: result.reason, completeAt: result.completeAt }; + } + + // Results feed forward: `provision` discovers the URL a `connect` two steps later writes down. + if (result.results) Object.assign(values, result.results); + completed.push(step); + } catch (err) { + // An effect that throws is a bug in that effect, not a different kind of failure. Recording it the + // same way keeps the row resumable instead of stranding it in `installing` forever. + return { status: 'failed', completed, at: step, error: err instanceof Error ? err.message : String(err) }; + } + } + + return { status: 'installed', completed }; +} + +async function runStep(step: StepName, ctx: StepContext, effects: InstallEffects): Promise { + switch (step) { + case 'preflight': { + const check = await effects.preflight(ctx.entry, ctx.mode); + return check.ok ? { status: 'done' } : { status: 'failed', error: `${check.reason} ${check.remedy}` }; + } + case 'provision': { + const { results, composeDir } = await effects.provision(ctx); + return { status: 'done', results: { ...results, composeDir } }; + } + case 'connect': + return effects.connect(ctx); + case 'schema': + await effects.applySchema(ctx); + return { status: 'done' }; + case 'process': + await effects.startProcess(ctx); + return { status: 'done' }; + case 'members': + await effects.provisionMembers(ctx); + return { status: 'done' }; + } +} From 4b9b98efda0ab09e9ffccf13fd9b0476baad49fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 10 Aug 2026 13:20:36 +0000 Subject: [PATCH 09/99] app store: run a sidecar's setup.sh, streaming it and collecting what it returns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the half of the template contract that faces the platform: answers go in as environment and never as prompts, and results come back as OFFICER_RESULT_= lines on stdout. A line protocol rather than JSON because the same stream is the user's live log — it goes to a terminal panel while the install runs. A script that must emit clean JSON cannot also narrate, and one that emits both needs a framing convention anyway. This mirrors the @@officer:progress@@ sentinel the job runner already uses, with the same rule: marker lines are plucked out, everything else passes through. parseResults is pure and tested against the realistic near-misses: a line that MENTIONS the prefix without starting with it, an empty value (Transmission with no RPC auth returns exactly that, and blank is a real answer), a value containing `=` (splitting on every one would truncate a credential), and a prefix with no assignment (a script bug — skipped rather than stored as a blank key). Verified end to end against a real script: environment reaches it, stderr is forwarded (docker compose writes its progress there, so dropping it would hide most of what a user watches), OFFICER_NONINTERACTIVE is set so a script that would block fails loudly instead of hanging behind a web form, and a non-zero exit is reported with the tail. Notes an artifact rather than hiding it: the two streams are pumped concurrently, so the error tail can interleave differently from real time. The live log is correctly ordered; only the summary can read out of order. Serialising the pumps would make a script that writes heavily to one stream block on the other. Co-Authored-By: Claude Opus 5 --- src/servers/app-store/run-script.test.ts | 52 +++++++++ src/servers/app-store/run-script.ts | 132 +++++++++++++++++++++++ 2 files changed, 184 insertions(+) create mode 100644 src/servers/app-store/run-script.test.ts create mode 100644 src/servers/app-store/run-script.ts diff --git a/src/servers/app-store/run-script.test.ts b/src/servers/app-store/run-script.test.ts new file mode 100644 index 00000000..f1ddcfe9 --- /dev/null +++ b/src/servers/app-store/run-script.test.ts @@ -0,0 +1,52 @@ +import { describe, expect, it } from 'bun:test'; +import { parseResults } from './run-script'; + +// The line protocol between a setup script and the installer. Pure, so it is tested without spawning +// anything — and worth testing precisely, because a script's output is interleaved with `docker +// compose`'s and a loose parser would pick up things that merely look like results. + +describe('parseResults', () => { + it('reads the values a script hands back', () => { + const out = parseResults( + ['==> Starting', 'OFFICER_RESULT_URL=http://127.0.0.1:18091', 'OFFICER_RESULT_PATH=/transmission/rpc', '==> Done'].join('\n'), + ); + expect(out).toEqual({ url: 'http://127.0.0.1:18091', path: '/transmission/rpc' }); + }); + + it('ignores everything that is not a result line', () => { + // The same stream is the user's live log. Narration must never be mistaken for a value. + const out = parseResults( + ['Container officer-transmission Started', 'note: OFFICER_RESULT_URL is printed at the end', ''].join('\n'), + ); + // The middle line MENTIONS the prefix but does not start with it, which is the realistic near-miss. + expect(out).toEqual({}); + }); + + it('keeps an empty value, because blank is a real answer', () => { + // Transmission with no RPC auth returns exactly this, and it means "no username", which is + // different from the key being absent. + expect(parseResults('OFFICER_RESULT_USERNAME=')).toEqual({ username: '' }); + }); + + it('keeps everything after the first `=`', () => { + // Tokens and URLs contain `=`; splitting on every one would silently truncate a credential. + expect(parseResults('OFFICER_RESULT_SECRET=abc=def==')).toEqual({ secret: 'abc=def==' }); + }); + + it('lets a later line win', () => { + // A script that reports twice has changed its mind — a retry inside it that finally worked. + expect(parseResults(['OFFICER_RESULT_URL=http://first', 'OFFICER_RESULT_URL=http://second'].join('\n'))).toEqual({ + url: 'http://second', + }); + }); + + it('skips a prefix with no assignment rather than storing a blank key', () => { + // A script bug. Storing `{'': ''}` would look like a deliberate empty value downstream. + expect(parseResults('OFFICER_RESULT_')).toEqual({}); + expect(parseResults('OFFICER_RESULT_=novalue')).toEqual({}); + }); + + it('tolerates indentation, since stderr lines arrive prefixed', () => { + expect(parseResults(' OFFICER_RESULT_URL=http://x ')).toEqual({ url: 'http://x' }); + }); +}); diff --git a/src/servers/app-store/run-script.ts b/src/servers/app-store/run-script.ts new file mode 100644 index 00000000..986fa24e --- /dev/null +++ b/src/servers/app-store/run-script.ts @@ -0,0 +1,132 @@ +import { join } from 'node:path'; +import { serviceDir } from './paths'; + +// Running a sidecar's setup.sh, streaming what it prints, and collecting what it hands back. +// +// The script contract lives in `templates/README.md`. Two halves of it are implemented here: answers go +// in as environment (never as prompts, which behind a web form is a hang with no output), and results +// come back as `OFFICER_RESULT_=value` lines on stdout. +// +// ── Why results are a line protocol and not JSON on stdout ── +// +// The same stream is the user's live log — it goes to a terminal panel while the install runs. A script +// that must emit clean JSON cannot also narrate, and one that emits both needs a framing convention +// anyway. A prefixed line is that convention, it survives being interleaved with `docker compose` +// output, and a human reading the log can see exactly what was handed back. +// +// This mirrors the progress sentinel the job runner already uses (`@@officer:progress@@`), for the same +// reason and with the same rule: the marker lines are plucked out of the stream, and everything else is +// passed through as narration. + +const RESULT_PREFIX = 'OFFICER_RESULT_'; + +/** + * Pull `OFFICER_RESULT_*` assignments out of a script's output. + * + * Pure, so the protocol is testable without spawning anything. + * + * Later lines win. A script that reports a value twice has changed its mind — a retry inside the script + * that finally succeeded, say — and the last word is the one that reflects reality. + */ +export function parseResults(output: string): Record { + const results: Record = {}; + for (const raw of output.split('\n')) { + const line = raw.trim(); + if (!line.startsWith(RESULT_PREFIX)) continue; + const eq = line.indexOf('='); + // A prefix with no `=` is a script bug, not a value. Skipping beats storing a key with an empty + // string, which would look like a deliberate blank later. + if (eq <= RESULT_PREFIX.length) continue; + const key = line.slice(RESULT_PREFIX.length, eq).toLowerCase(); + results[key] = line.slice(eq + 1); + } + return results; +} + +export type RunScriptParams = { + sidecarId: string; + /** Directory holding the template's `setup.sh`, i.e. `app-store/templates/