From b4c8cfad910274718b3b176919cfe1232c899ba4 Mon Sep 17 00:00:00 2001 From: Andre Padez Date: Fri, 21 Aug 2026 00:36:28 +0100 Subject: [PATCH] docs: PLUGIN.md as the current truth, not the extraction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It described a separate atelier Postgres database, providers as a section in Officer's settings, and a BLOCKERS.md that was never committed here — none of which is true. It is now what the code does, and why: the three words that are not interchangeable, where keys live and why getSecret rather than getKey, the catalogue's four rules, why refresh is a button, what a refresh can and cannot learn, and the two ceilings the 240s poll sits under. Written so that someone arriving cold can carry on from here. --- PLUGIN.md | 352 ++++++++++++++++++++++++++---------------------------- 1 file changed, 168 insertions(+), 184 deletions(-) diff --git a/PLUGIN.md b/PLUGIN.md index 9b8ad60..481df49 100644 --- a/PLUGIN.md +++ b/PLUGIN.md @@ -1,204 +1,188 @@ -# Atelier — extracted from a standalone service into a plugin +# Atelier — how it is built, and why -**Status: extracted 2026-08-20, unattended overnight run.** Read `BLOCKERS.md` alongside this file — -it has the decisions made without being able to ask, in the order they were found. This file is what -moved, what didn't, and why; that one is what was uncertain. +Current as of **2026-08-20**. Rewritten that evening, because the previous version described an +extraction rather than the thing that now exists and was wrong in three ways that mattered: it claimed a +separate `atelier` Postgres database, providers as a screen in Officer's settings, and a `BLOCKERS.md` +that was never committed to this repo. -Atelier is image generation as a service: a provider-shaped API over OpenAI's image endpoints, storing -what it makes so every image has a durable reference. It came from -`~/officer-dev/dockers/officer-atelier`, a standalone Bun/Hono/Postgres server predating the plugin -system, still running in production as `officer-atelier` (PM2, port 3002, `atelier.officer.dev`) — see -"What did not move" below before assuming it is safe to touch. +**Code is the truth.** Where this file and the code disagree, fix this file. --- -## What moved, and what changed on the way +## What it is -**Nothing pointed the other direction.** Unlike offscale and music, a grep of the whole platform for -`atelier` found no reverse dependency — no widget, no relay, nothing importing from the standalone -service. It came out whole with no seam to design around, which is the exception rather than the rule -for this kind of extraction. +Image generation as an Officer plugin: a prompt goes in, a job is queued, images come out and stay — +with a gallery, collections, and an OpenAI-compatible API an agent can poll rather than block on. -### The backend — `sidecar/` - -Providers (adapter-shaped: `types.ts`, `openai.ts`, an `index.ts` that dispatches — OpenAI is the only -adapter today, and the point of the shape is that a second one is a new file, not a new screen), the -generation queue (Postgres-backed, `MAX_CONCURRENT_GENERATIONS` workers), collections, and the database -layer moved essentially unchanged. Three things did not: - -1. **Images are served from `GET /files/:filename`, authenticated, not an unauthenticated `PUBLIC_URL` - route.** The standalone service served `/images/*` off Bun's own route table with no session, - because every image needed "a permanent URL" reachable by the mobile app and anything else on the - tailnet. Inside Officer that is not an option — see "What you may NOT do" in the kickoff prompt. The - platform's own `?token=` escape hatch (already used by `FileViewer/file-types.ts` and the music - plugin's `MusicPlayerHost` for exactly this — a browser cannot put an `Authorization` header on an - ``) covers it: the JSON API returns `/files/` (relative, no host, no token — the - sidecar cannot know the platform's public origin any more and does not need to), and the frontend - appends its own token when rendering (`web/lib/api.ts`'s `fileUrl`). - -2. **The OpenAI-key bootstrap fallback reads the secret store before `.env`.** `keyForProvider` still - prefers a provider row's own encrypted key; the fallback that used to be `process.env.OPENAI_API_KEY` - directly is now `getSecret('atelier--key')` first, `.env` second — `getSecret`, never - `getKey`, because a foreign credential must not be silently substituted with a random one. Set it - with `bun run secret:set atelier-openai-key`. - -3. **`provider_settings` did not come along.** It is the one-row-per-adapter shape `providers` replaced - (see the standalone repo's own `providers/schema.ts` comment and `scripts/migrate-providers.ts`), and - grepping the server code found no caller anywhere except that migration script — which also did not - move. Its table and its one row stay in the `atelier` database, untouched. If a machine's migration - never ran, that is a real gap, but it is the standalone service's gap, not one this extraction - introduced. - -`ping`/`echo` (smoke-test routes, no caller) did not move either — the sidecar's `/_health` covers -liveness, matching every other HTTP sidecar in this codebase. - -### The database — stays separate, does not fold into officer_db - -Decision 3 said to check whether `collections` and `provider_settings` were empty before folding them -into `officer_db`. Both had rows (2 and 1), so the `atelier` Postgres database stays separate — the -sidecar keeps its own connection (`sidecar/db/db.ts`), reading `OFFICERDEV_POSTGRES_URL` from the platform -`.env`. **This is new territory for the plugin system**: `db/schema.ts` at a plugin's root is the -`officer_db`-prefixed-tables convention (`pushPluginSchema`, `_` prefix), and this plugin has no -such file — `discoverPlugins()` correctly reports `schema: null` for it, and the install flow's schema -step is simply skipped. Nothing in `src/servers/plugins/` needed to change for this, because "no -`db/schema.ts`" is already a valid state the discovery contract handles — a plugin with an internal, -self-managed database is indistinguishable from a plugin with no tables at all, as far as the platform -is concerned. Worth knowing before assuming every plugin's data lives in `officer_db`. - -**Not set automatically.** `OFFICERDEV_POSTGRES_URL` has no default — see `BLOCKERS.md` for why (mainly: -pointing it at the live production `atelier` database while that database's own standalone service is -still running risks two processes racing the same generation queue). The owner has to add it before -installing. - -**Migrating into `officer_db` as `atelier_*`-prefixed tables is real follow-up work**, not done here — -five tables (`generations`, `collections`, `collection_items`, `providers`, and the dead -`provider_settings`, which should be dropped rather than migrated), real data, needs a human watching it. - -### The frontend — three panels, plus a settings screen - -`web/panels.ts` + `web/layout.ts`: Generate, Gallery (+ Lightbox), Collections — the ≈1,200 lines of -real work the brief identified, ported with the shadcn components Officer already ships -(`@/components/ui/*` — the standalone service's own `src/apps/frontend/components/ui/*` did not move, -per decision 1). Providers did not become a fourth panel: - -**Providers is a platform Settings screen** (`Settings → Integrations → Atelier`, -`src/apps/officer-web/.../IntegrationsSettings/AtelierProviders.tsx`), per decision 2. Worth flagging -loudly: **`IntegrationsSettings/index.tsx` itself carries a comment, written five days earlier when email -was extracted, that says the opposite** — "a plugin contributes panels, not settings sections, so it -now lives inside the email workspace itself." Decision 2 was explicit and pre-made, so it was followed -anyway, and the contradiction is recorded in `BLOCKERS.md` rather than silently resolved either way. -`AtelierProviders.tsx` calls `/api/atelier/providers` over plain HTTP via `useClient()` — it imports -nothing from `plugins/atelier/`, so nothing here reaches back across the plugin boundary; it is exactly -the same relationship any authenticated frontend has with any authenticated backend. - -Panel coordination is URL + one channel, never props (a panel component only ever receives `panelId`): - -- Gallery's lightbox is `?selected=`; Collections' detail is `?collection=` — two different - names because both panels render on the same route at once and would otherwise collide. -- Generate bumps `useAtelierRefresh` (added to `src/workspaces/officerdev/src/channels.ts`, same - stable-bump shape as `useFilesRefresh`) when a job finishes; Gallery subscribes to it. This replaces - the standalone app's `onFinished` callback prop, which only worked because `App.tsx` lifted the - gallery's state above both tabs — not an option once they are independent panels. - -The default layout is a fixed horizontal three-way split (Generate 26% / Gallery 48% / Collections -26%) rather than the standalone app's hash-based tabs — the `LayoutNode` type has no tab-group concept, -only `group` (split) and `panel`, so three panels means three panes. Users can resize or rearrange -freely; this is only the starting arrangement. - -### `sharp` — promoted to a direct dependency - -`generations/inspect.ts` needs it to measure width/height/alpha-channel/opacity of a written image — -load-bearing, not incidental: it is how a `background: transparent` request that silently produced an -opaque image gets caught, which the standalone repo's own comments call "the single most consequential -failure for an asset destined to be an app icon." It was already resolved in `bun.lock` as an optional -transitive dependency of `next`; adding it as a direct dependency pulled in no new package versions — -verified by reading the lockfile diff before committing it. +``` +manifest.ts the plugin's declaration; `sharp` is declared here, shared across plugins +api/router.ts a thin auth-gated proxy. No Atelier knowledge, and it must never grow any +sidecar/ the whole contract: catalogue, providers, queue, database +web/ three panels — Collections, Generate, Gallery +db/schema.ts found by the platform's schema generator; re-exports what lives in sidecar/db +``` --- -## What did not move, and must not be touched +## Three words that are not interchangeable -**The standalone `officer-atelier` service keeps running, unchanged.** Its own `ecosystem.config.cjs` -says plainly: "this is the instance the mobile app talks to, so it holds still." It serves -`https://atelier.officer.dev`, holds real data (87 generations, 2 collections, 5 providers on the -machine this was extracted from), and this extraction did not stop it, did not touch its nginx config, -and did not touch the mobile app. **Cutting the mobile app over to the plugin's route — or deciding -whether to at all — is the owner's call, not something this run made unilaterally.** Until that happens, -having both installed means two independent Atelier deployments existing side by side, which is -expected, not a bug. +Getting these confused is what produced the UI we threw away on 2026-08-20, so they are fixed: + +| | | +|---|---| +| **provider** | A vendor. OpenAI. Curated by us, listed whether you have a key or not. | +| **adapter** | Code that speaks a vendor's HTTP API. Written once, only developers say it. | +| **catalogue** | The curated file: which providers exist, their models, and what each model can do. | + +There is deliberately **no fourth concept**. There used to be — a "provider" was a database row of +adapter + model + key, and choosing a model was therefore an act of configuration. That produced five +rows nobody had created (a migration minted one per model from a hard-coded list), a key icon and a +delete button inside a dropdown item, and no way to answer "which vendors could I use" without having +already set one up. + +**A provider is a vendor with a key or without one. The model is chosen per generation.** + +--- + +## Where things live + +**Tables — `officer_db`, `atelier_` prefixed, like every other plugin.** Two of them: +`atelier_generations`, `atelier_collections` (+ `atelier_collection_items`). The extraction originally +kept a separate `atelier` database; that was reversed the same week. A plugin that behaves differently +from every other plugin is a plugin nobody can reason about. + +**There is no providers table.** With the model off the row, all that was left of one was *the key for +OpenAI* — and Officer already stores keys. + +**Keys — the platform secret store**, `secrets/officer-keys.db`, purpose `atelier--key`. One +active value per purpose, `0600` inside a `0700` directory, deliberately outside `data/` so it does not +travel in the same tarball as a database dump. Purposes are created on first use, so nothing is +provisioned in advance and no plugin can read another's. + +Read with **`getSecret`, never `getKey`**: `getKey` mints a random value for a purpose it cannot find, +which is right for our own encryption keys and catastrophic for a foreign credential — it means +authenticating with something invented. Falls back to `OPENAI_API_KEY` in the environment, which is how +an install supplies a key without a UI. + +**Images — `$OFFICER_ROOT/data/atelier/images/`,** named by generation id, so a row and its file find +each other with no index between them. Not in the plugin directory: that is a git clone an update +replaces, and artefacts kept there vanish on upgrade with nothing logging it. + +**Catalogue — two files.** `sidecar/catalogue/shipped.ts` is the curated baseline, in git, reviewed; +`$OFFICER_ROOT/data/atelier/catalogue.json` is what a refresh writes. The second merges over the first, +and a failed refresh therefore degrades to what we shipped rather than to nothing. + +--- + +## The catalogue's four rules + +1. **Advisory, never authoritative.** It shapes what the UI *offers*; it never decides what the server + *accepts*. `validate.ts` deliberately refuses to enforce per-model rules — a stale copy of them would + reject valid combinations, and the vendor should be the one to answer. A model the catalogue has + never heard of still generates. +2. **Allowed to be incomplete.** A field a refresh cannot establish stays absent, and absence degrades: + no price means a null cost, which is honest. An invented price sums into a total that is wrong. +3. **Fields merge, not objects.** `GET /v1/models` knows ids and nothing else, so an entry built from it + would otherwise wipe the sizes, qualities and prices that were curated by hand. +4. **Retire, never delete.** A model the vendor stopped listing is marked `deprecated`: hidden from the + dropdown, still resolving everywhere else. The gallery labels historical rows from this list, and 88 + images name the model that made them. + +### Refresh is a button + +There was a daily cron and it was removed the day it was written. A refresh matters at exactly one +moment — you are choosing a provider and want to know its models are current — and a button answers that +question, where a nightly pass can only have already answered it, invisibly, or not. A timer also buys a +standing obligation to hit a vendor from every install forever, whether anyone opens Atelier or not. + +`POST /providers/catalogue/refresh?provider=openai`, or without the parameter for all of them. A +provider with no refresher answers 400 rather than a silent success: from a button, "nothing happened" +and "nothing could happen" look identical. + +### What a refresh can and cannot learn + +`GET /v1/models` is authoritative and thin — ids, an owner, a timestamp. Which is why image models are +matched by pattern (`/^(gpt-image|dall-e)/`) rather than a list: `gpt-image-2` appeared this way, on the +day it shipped, with no release of this plugin. + +**There is no pricing endpoint, on any plan.** Prices come off an HTML page, and `openai-refresh.ts` +parses it in a way that gives up rather than guesses — a guess gets written to disk and then believed. +As of 2026-08-20 it gives up: the page renders its prices client-side, and it says so in the log. **The +replacement is not a better regex** — it is handing the page to a model and asking for structured +output, which Officer can already do locally through its own Claude sidecar. That is the next piece of +work here. + +--- + +## The panels + +**Collections** (left, top) — a filterable list, one row per collection: icon, name, count, rename and +delete on hover. Create is a dialog. Deleting asks the one question that cannot be undone — keep the +images, or delete them too — unchecked by default, so the recoverable answer is what you get by pressing +Enter without reading. That needed `DELETE /collections/:id?images=delete`, which reads the members +*before* deleting the collection, since dropping it takes the memberships with it. + +**Generate** (left, bottom) — provider dropdown (the catalogue, one entry per vendor, "no key" shown +inline), model dropdown (that provider's models), then Add/Change key, Update models, All providers, and +the form. + +**Gallery** (right) — every image, or one collection's images when `?collection=` is set. The collection +view lives here rather than in the Collections panel: that panel is a narrow selector, and rendering a +grid inside it replaced the list you were selecting from with the thing you selected. + +The panels coordinate through the URL and one refresh channel, never props: `?selected=` is the +lightbox, `?collection=` is what the gallery shows. Generate bumps `useAtelierRefresh` when a job +finishes; Gallery subscribes. + +**Removing a panel key rewrites saved layouts** that name it — see `dashboards/queries.ts`, which is a +bug that cost an evening on offscale. The install flow clears this plugin's screen rows, so a panel +removed between versions is safe **as long as the plugin is reinstalled rather than merely pulled**. + +--- + +## The job model + +Generation answers **202 with a job id**, never the image. `n: 10` is ten HTTP requests and holding a +connection open for the length of them ties up a socket for minutes; the callers are agents that can +poll. `GET /images/generations/:id?wait=` long-polls up to **240s**, which is bounded by two ceilings +above it — Bun's `idleTimeout` at 255 (256 is rejected outright) and nginx at 320. The innermost has to +be the one that fires, or the caller gets a dropped connection instead of a status. + +Each image is its own row with its own cost and duration, so one failure out of four does not discard +the three that worked. + +The same model is what video will need when it arrives: a two-minute render is exactly the case 202 was +built for. Video needs a longer poll ceiling, per-second pricing (the catalogue's `Pricing` is already a +shape per unit for this reason), and duration/fps on the row — `has_alpha`/`opaque` go null for it, the +way pricing already goes null when we do not know it. + +--- + +## Serving images + +`GET /files/:filename`, authenticated. The standalone service served them from an unauthenticated +`PUBLIC_URL` route; inside Officer that is not an option. A browser cannot put an `Authorization` header +on an ``, so the JSON API returns a relative `/files/` and the frontend appends its own +`?token=` — the same escape hatch `FileViewer/file-types.ts` and the music plugin already use. --- ## Verification -- `bunx tsgo`: clean. -- `bun test`: baseline was 1041 pass / 0 fail across 90 files; after this extraction, 1069 pass / 0 fail - across 92 files — the two new files are `sidecar/serialize.test.ts` and - `sidecar/generations/response.test.ts`, both ported from the standalone repo (URLs updated for the new - `/files/` scheme). No regressions, no file count drop. -- `discoverPlugins()` finds the plugin, the manifest validates, `schema: null` as expected (separate - database). -- The platform-side proxy router (`api/router.ts`) mounts and correctly answers `503 atelier sidecar - not available` with no sidecar running — same behaviour as offscale's. -- The sidecar's internal Hono app was exercised directly (`buildHonoApp()`, real HTTP requests, no port - bound) against the `atelier_dev` Postgres database: `/providers`, `/providers/adapters`, - `/collections`, `/images` all answered `200` with real rows; `/files/` answered `404`; an - unknown route answered the OpenAI-shaped `404` envelope. Old provider rows correctly failed to - decrypt (a fresh `secret.key` was generated for this checkout, as intended — a key that will not - decrypt is broken, not absent, and `keyForProvider` reports `configured: false` rather than crashing). -- **The real production SPA bundle builds successfully with the plugin included.** Called - `generatePluginsModule` and `rebuildFrontend` directly (the same functions `refreshPluginMounts` calls - during a real install) against a synthetic "atelier is installed and enabled" state — no server - running, no port bound. `Bun.build`, with the real Tailwind plugin, compiled `Plugins.gen.tsx`, all - three panels, and the new `AtelierProviders` settings section in 2.2s with zero errors: 3 outputs, no - warnings. This is the same build step an install performs and exercises everything `bunx tsgo` cannot - — module resolution through Bun's own bundler, JSX compilation, Tailwind processing — without needing - a running server. What it does not prove is that the result **renders correctly in a browser**; see - below. -- **Not done**: a live PM2 install/enable/disable/uninstall/purge cycle through `/marketplace`, per the - runbook's own verification table. This machine's live `officer` process serves the owner's real - session and its own `plugins/` directory is the main checkout, not this worktree — installing through - it would write real rows into the shared `officer` Postgres database from code that only exists here. - That crosses into "affects shared systems beyond the local environment," so it was not done - unattended. The safer checks above cover the same ground read-only or against an isolated database. +- `bunx tsgo` — clean, apart from `Cannot find module 'sharp'` in a checkout where the plugin's declared + dependency has not been installed. That is expected outside a real install. +- `bun test plugins/atelier` — 32 pass. `catalogue/merge.test.ts` covers the four merge rules above, + which are the part where a mistake silently rewrites curated data. +- The web bundle builds with the platform; the panels are typechecked against the real `@/components/ui`. --- -## Open, for whoever picks this up +## Open -1. **Publish to `gitea.officer.dev/plugins/atelier` and register it in the marketplace.** Not done — see - `BLOCKERS.md` for why and the exact commands. -2. **`OFFICERDEV_POSTGRES_URL` has to be set before install.** No default, on purpose. -3. **The `atelier` → `officer_db` migration**, described above, once someone can watch it happen. -4. ~~`assets/icon.png` and `assets/splash.png` are `[placeholder]` flat swatches.~~ Done - 2026-08-20: both come from the suite's own masters in the OfficerIcons repo — - `OffAtelierAppIcon` (trimmed, since that master is 1312x1199 rather than square, then centred on - a transparent 256 canvas) and `OffAtelierSplashScreen` at 1024. Same shapes and same convention as - plugins/music. -5. **A real PM2 install/enable/disable/uninstall/purge cycle**, on a machine where doing so does not - touch the owner's live session or shared database. -6. **Deciding whether and when to cut the mobile app over** from `atelier.officer.dev` to the plugin's - route, and what happens to the standalone service afterward. - -## Reversed 2026-08-20: officer_db, not a separate database - -The extraction kept Atelier's own `atelier` Postgres database, because `BLOCKERS.md` decision 3 said to -when the tables already had rows. The owner reversed that afterwards, and the reasoning is worth keeping: - -**A plugin that behaves differently from every other plugin is a plugin nobody can reason about.** The -separate database meant a second connection string to set before install, a second thing to back up, a -second thing to get wrong on a new machine, and failure modes no other plugin shares. None of that was -bought by anything except avoiding a migration. - -So: four `atelier_`-prefixed tables in `officer_db` — `atelier_collections`, `atelier_collection_items`, -`atelier_providers`, `atelier_generations` — created empty by `bun db:push`, like every other plugin. -Index and check constraint names carry the prefix too; those are global in Postgres and would otherwise -be the one thing that collides. - -`ATELIER_POSTGRES_URL` is gone. `sidecar/db/db.ts` reads `OFFICERDEV_POSTGRES_URL ?? POSTGRES_URL`, the -house convention, which the platform already sets. - -**The standalone service's data is NOT here.** It is still in the old `atelier` database, and the old -service is still running against it, untouched. Migrating those rows — 2 collections, 73 items, 5 -providers, 87 generations — is a deliberate, watched step, done after the plugin is installed and proven -on alpha. Nothing about this plugin depends on it happening. +1. **The pricing extractor** — see above. Everything else in a refresh works; this is the one gap, and + it fails loudly rather than silently. +2. **Video.** Adapter, a longer poll, per-second pricing, duration/fps on the row. +3. **`bun db:push`** is required to drop `atelier_providers` and `generations.provider_id` on an install + that predates 2026-08-20. +4. **The 88 historical images** were generated by the standalone service and predate all of this. Their + prompts are the source for `OfficerIcons/ICON_GENERATION_RECIPE.md`.