pastilhasandClaude Opus 5
19bbbab907
Three mechanisms where there should be one. The catalogue was filled by hand, and each host separately polled twenty-one repositories with git to work out whether a release existed — a second source of truth for a fact one place already knew. Shipping photos 2026.09.03.02 correctly and watching alpha not notice for ten minutes is what started this. The timer. scripts/marketplace-cron.sh, every minute, on the machine that publishes. Safe now only because the previous commit made the sync read tags instead of working trees; over the old one this would have published somebody's afternoon on a schedule. Nothing is written unless the release commit differs from the one already published, so a tick with nothing to do is one query, one fetch per repo and no writes — about 2.7s. One timestamped summary line per tick regardless, because a fresh timestamp is the only thing that separates "nothing to publish" from "the timer died"; a named line only for a plugin that moved. The badge. `updateable` now compares the tag at HEAD with the tag the catalogue says is newest — difference, not ordering, since parsing 2026.08.27.01 works right up until somebody tags v2 and then fails silently. refreshRemote() and its two-minute window are gone with the polling they existed to paper over. The git rule stays as the fallback for a plugin the store cannot speak for: an unregistered checkout, or an unreachable store. That case now reads whatever refs the last pull left, which is the accepted cost of not making every machine poll every repository forever. fetchCatalogue()'s cache went from five minutes to one. It only drew tiles before; now the wait for a release to appear would have been the sync's minute plus its five. Also: the Update button names its destination whenever a release is known, not only on `new-release`. An untagged checkout reports `untagged` and used to lose the tag with it, so the button said "Update" and the owner had to guess where it landed — the marketplace knows, even for a checkout that has never fetched it. Verified: forced a stale row, the timer republished it inside 60s and logged one line. Badge decisions checked against the live catalogue for seven plugins, including the two shapes that matter — at a tag the store agrees with, and untagged with a release known only to the store. 9 release tests pass, 100% of release.ts covered, four of them new and all against real repositories. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
189 lines
11 KiB
Markdown
189 lines
11 KiB
Markdown
# marketplace.officer.dev
|
|
|
|
The public plugin catalogue, and the API the platform installs from. Built 2026-08-16.
|
|
|
|
## What it is, in one paragraph
|
|
|
|
A website listing plugins, backed by its own Postgres database, serving two read-only JSON routes. The
|
|
platform reads those routes to populate its Marketplace screen and to find out where to clone a plugin
|
|
from. **The website installs nothing** — installing means cloning a repository, pushing a schema and
|
|
starting a process on someone else's machine, so it happens entirely inside the owner's own Officer.
|
|
|
|
```
|
|
gitea.officer.dev/** the repositories (source of truth)
|
|
│ bare mirrors, read AT THE NEWEST TAG — src/servers/marketplace/sync.ts
|
|
│ every minute, scripts/marketplace-cron.sh
|
|
▼
|
|
officer_marketplace (postgres) plugins + plugin_metadata
|
|
│ src/servers/marketplace/server.tsx :9100
|
|
▼
|
|
marketplace.officer.dev GET /api/plugins, /api/plugins/:appName
|
|
│ fetchCatalogue() live → cached → bundled
|
|
▼
|
|
an Officer install /marketplace → clone → push schema → mount
|
|
and the update badge — see "Publishing" below
|
|
```
|
|
|
|
## The parts
|
|
|
|
| Path | What it is |
|
|
| ------------------------------------ | ----------------------------------------------------------------------------------------------- |
|
|
| `src/databases/marketplace_db/` | Schema and queries for `officer_marketplace`. Deliberately NOT a workspace package — see below. |
|
|
| `scripts/marketplace-sync.ts` | Fills the database from the plugin repositories. Run by hand. |
|
|
| `src/servers/marketplace/server.tsx` | The public process. Two GET routes, a health route, and the SPA. |
|
|
| `src/apps/marketplace/` | The website. |
|
|
| `src/servers/plugins/marketplace.ts` | The PLATFORM's client. Fetch, cache, fall back. |
|
|
|
|
## Why a second database
|
|
|
|
`officer` is one owner's machine — their accounts, dashboards, mail, wallet. The marketplace is a public
|
|
catalogue that is identical for everyone and is served by a process with no login, no session and no
|
|
user. Sharing a connection would put an unauthenticated public surface one query away from every private
|
|
table on the box. Keeping it separate means the catalogue API structurally cannot read any of it.
|
|
|
|
It is also the one database here that can be dropped and rebuilt from the repositories it indexes, which
|
|
is a very different operational posture from the platform's.
|
|
|
|
`MARKETPLACE_POSTGRES_URL` overrides the connection; by default it is `POSTGRES_URL` with the database
|
|
name swapped, because in every deployment that exists they are on the same server.
|
|
|
|
## Why the marketplace is not a core process
|
|
|
|
`officer-setup`'s `CORE_PROCESSES` does not include it, on purpose. It is one website that officer.dev
|
|
hosts — not something every install should run. A copy on every machine would serve a stale catalogue on
|
|
a port nobody asked to open.
|
|
|
|
On the host that serves it, the PM2 entry is added to that machine's `ecosystem.config.cjs` by hand:
|
|
|
|
```js
|
|
{ name: 'officer-marketplace', script: 'bun', args: 'run src/servers/marketplace/server.tsx',
|
|
cwd: '<platform dir>', watch: false, env: { NODE_ENV: 'production', MARKETPLACE_PORT: '9100' } }
|
|
```
|
|
|
|
then `pm2 start ecosystem.config.cjs --only officer-marketplace && pm2 save`.
|
|
|
|
## Two tables, split by owner
|
|
|
|
`plugins` is the **registry**: app name, developer, clone URL, `listed`. Authoritative, hand-curated,
|
|
small. Nothing outside this database may write it.
|
|
|
|
`plugin_metadata` is a **cache** of what each repository says about itself — label, summary, icon,
|
|
permissions, os dependencies, README. All of it already exists in the plugin's `manifest.ts`, which stays
|
|
the source of truth.
|
|
|
|
Keeping them apart is what makes the cache safe to be wrong: a metadata row can be deleted, rebuilt or
|
|
found stale, and none of that can lose the fact that a plugin exists or where it lives. Merging them
|
|
would make a refresh a write to the registry — which is how a failed fetch turns into a plugin
|
|
disappearing from the store.
|
|
|
|
**Nothing here is trusted by an install.** The catalogue is what the owner sees _before_ consenting; after
|
|
the clone, the plugin's own manifest is what the platform reads. A metadata row that disagrees with the
|
|
repository is a marketplace bug, never a second source of truth.
|
|
|
|
## Publishing
|
|
|
|
**Push a tag. That is the whole procedure.** A timer on officer-dev picks it up within a minute and every
|
|
Officer install sees it on the next page load.
|
|
|
|
```bash
|
|
bun --env-file=.env scripts/marketplace-sync.ts # all of them, now
|
|
bun --env-file=.env scripts/marketplace-sync.ts photos # one
|
|
bun --env-file=.env scripts/marketplace-sync.ts --add https://gitea.officer.dev/plugins/finances.git
|
|
bun --env-file=.env scripts/marketplace-sync.ts --force # re-publish rows this file's own logic changed
|
|
```
|
|
|
|
### It reads releases, never a working tree
|
|
|
|
Every registered plugin is mirrored bare into `~/.cache/officer-marketplace/mirrors` and read at its
|
|
newest tag — manifest, README and file list all out of `git cat-file`. There is no working tree in the
|
|
path, so publishing something uncommitted is not a mistake that can be made.
|
|
|
|
That is a change, and it was load-bearing. Until 2026-09-03 the sync imported `manifest.ts` off
|
|
`platform/plugins/` — the directories THIS machine has installed — so publishing meant "get officer-dev's
|
|
checkout to the release, then run the script", and nothing showed when that had not happened. On the day
|
|
it was rewritten the live store was advertising four plugins from untagged working copies: av-club
|
|
2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of which had ever
|
|
been tagged. It is also why the job could not go on a timer before: a cron over it published whatever was
|
|
in a working tree, on a schedule.
|
|
|
|
### The registry decides which plugins, and where from
|
|
|
|
The list comes from the `plugins` table — the half this document has always called authoritative, and
|
|
which was in fact a by-product of what happened to be checked out. Now it decides. A plugin nobody has
|
|
installed here can still be published, and a plugin lives wherever its row says: `fofos-admin` is under
|
|
`fofos/`, not `plugins/`, which an org listing would have missed. Registering one is `--add`.
|
|
|
|
### A plugin with no tag is left alone
|
|
|
|
It has never made a release, so the sync has nothing true to say about it: the existing row stays and the
|
|
run names it. Deleting it would drop somebody's plugin out of the store on the day the rule changed.
|
|
Tagging it is the fix, and the report is the reminder.
|
|
|
|
### `release_tag`, and why it is not `version`
|
|
|
|
`version` is the manifest's own claim; `release_tag` is what was pushed. They match every time somebody
|
|
tags correctly, and when they do not, the tag is the fact — a manifest whose version nobody bumped would
|
|
otherwise make a real release invisible to anything comparing versions. The sync warns on a mismatch.
|
|
|
|
Icon URLs pin to the tag too. `/raw/HEAD/…` served whatever was on the default branch, via a 303, so a
|
|
plugin could show one icon in the store and ship another.
|
|
|
|
### The timer
|
|
|
|
`scripts/marketplace-cron.sh`, `* * * * *`, logging to `officer-dev/marketplace-sync.log`. One
|
|
timestamped summary line per tick — always, including the ticks that do nothing, because a fresh
|
|
timestamp is the only thing that distinguishes "nothing to publish" from "the timer died". A plugin-level
|
|
line appears only when one actually moved.
|
|
|
|
A tick costs about 2.7s with nothing to do: one query, one fetch per repository, no writes. Nothing is
|
|
written unless the release commit differs from the one already published.
|
|
|
|
A Gitea webhook would cut the delay from a minute to a second, at the price of a public write route and a
|
|
shared secret. A minute was never the problem anybody had — the complaint that started this was a release
|
|
that never appeared at all.
|
|
|
|
### Remotes
|
|
|
|
`--add` rewrites SSH to HTTPS. A `git@gitea…` origin in the catalogue hands every installer a URL that
|
|
only works on the machine that published it, and the install fails at authentication — which reads as
|
|
"the plugin is broken" rather than "the catalogue is wrong".
|
|
|
|
## The update badge reads the catalogue
|
|
|
|
`updateable` on `/api/plugins` compares the tag at the checkout's HEAD with the tag the catalogue says is
|
|
newest. Difference, not ordering: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
|
|
then it fails silently.
|
|
|
|
Before this, each host ran a rate-limited background `git fetch` against all twenty-one repositories to
|
|
re-derive a fact one place already knew. That is gone. `releaseState()` still holds the git rule as a
|
|
fallback for a plugin the store cannot speak for — an unregistered checkout, or a store that could not be
|
|
reached — and in that case its refs are whatever the last `pullPlugin` left, which is the accepted cost.
|
|
|
|
`fetchCatalogue()`'s cache dropped from five minutes to one when the badge started depending on it:
|
|
otherwise the wait for a release to appear was the sync's minute plus the cache's five.
|
|
|
|
## The platform degrades, it does not break
|
|
|
|
`fetchCatalogue()` answers **live → cached → bundled**, in that order, and never throws. The Marketplace
|
|
screen is also where an owner manages plugins they already have, so the store being unreachable must not
|
|
take that page with it. The cache is per-process with a 5-minute TTL; the bundled list is the old
|
|
constant, kept current, so a machine with no internet still shows something true.
|
|
|
|
Responses are validated per entry. Not against a hostile marketplace — one that serves a malicious
|
|
`source` has already won, since the owner asked to install what it lists — but against a **wrong** one: a
|
|
proxy serving HTML as JSON, a half-deployed API, a field renamed on one side. A non-`https://` source is
|
|
refused outright, because that string reaches `git clone`.
|
|
|
|
## Open items
|
|
|
|
The full list, ranked, is `docs/marketplace-todo.md`. The ones that shape the design:
|
|
|
|
- **`[blocker]` Third-party submissions cannot use this sync.** It reads a manifest by importing it, which
|
|
is code execution. Fine for our own repositories; not a mechanism a stranger's plugin can go through.
|
|
Submissions need either a sandbox or a published `manifest.json` alongside the TypeScript one.
|
|
- **`[open]` `rawUrl()` assumes Gitea.** A GitHub-hosted plugin needs the other URL form. The forge should
|
|
become part of the registry rather than inferred from the URL.
|
|
- **`[open]` No versions table.** `plugin_metadata` is one row per plugin — a snapshot, not a history. A
|
|
store that can offer "install 1.2 instead of 1.3" needs a different shape.
|
|
- **`[open]` No search, no categories, no counts.** Three plugins do not need them.
|