Compare commits

...
2 Commits
Author SHA1 Message Date
pastilhasandClaude Opus 5 19bbbab907 publishing a plugin is pushing a tag
Three mechanisms where there should be one. The catalogue was filled by hand, and each host separately
polled twenty-one repositories with git to work out whether a release existed — a second source of
truth for a fact one place already knew. Shipping photos 2026.09.03.02 correctly and watching alpha
not notice for ten minutes is what started this.

The timer. scripts/marketplace-cron.sh, every minute, on the machine that publishes. Safe now only
because the previous commit made the sync read tags instead of working trees; over the old one this
would have published somebody's afternoon on a schedule. Nothing is written unless the release commit
differs from the one already published, so a tick with nothing to do is one query, one fetch per repo
and no writes — about 2.7s. One timestamped summary line per tick regardless, because a fresh timestamp
is the only thing that separates "nothing to publish" from "the timer died"; a named line only for a
plugin that moved.

The badge. `updateable` now compares the tag at HEAD with the tag the catalogue says is newest —
difference, not ordering, since parsing 2026.08.27.01 works right up until somebody tags v2 and then
fails silently. refreshRemote() and its two-minute window are gone with the polling they existed to
paper over. The git rule stays as the fallback for a plugin the store cannot speak for: an unregistered
checkout, or an unreachable store. That case now reads whatever refs the last pull left, which is the
accepted cost of not making every machine poll every repository forever.

fetchCatalogue()'s cache went from five minutes to one. It only drew tiles before; now the wait for a
release to appear would have been the sync's minute plus its five.

Also: the Update button names its destination whenever a release is known, not only on `new-release`.
An untagged checkout reports `untagged` and used to lose the tag with it, so the button said "Update"
and the owner had to guess where it landed — the marketplace knows, even for a checkout that has never
fetched it.

Verified: forced a stale row, the timer republished it inside 60s and logged one line. Badge decisions
checked against the live catalogue for seven plugins, including the two shapes that matter — at a tag
the store agrees with, and untagged with a release known only to the store. 9 release tests pass, 100%
of release.ts covered, four of them new and all against real repositories.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 13:13:43 +01:00
pastilhasandClaude Opus 5 78e89ebcec the marketplace publishes releases, not working trees
The sync read `platform/plugins/` — the directories this machine has installed — and imported each
manifest off the working tree. So publishing meant "get officer-dev's checkout to the release, then
run the script", and nothing showed when that had not happened. Live proof at the time of writing:
av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26 were all in
the store, and none of the four has ever been tagged.

It is also why the job could not go on a timer. A cron over it published whatever was in a working
tree, on a schedule.

Now every registered plugin is mirrored bare into ~/.cache/officer-marketplace/mirrors and read AT ITS
NEWEST TAG — manifest, README and file list all out of `git cat-file`. No working tree in the path, so
publishing something uncommitted stops being a mistake anyone can make.

docs/marketplace.md proposed reading manifests over Gitea's API, which needed a sandbox or a committed
manifest.json. Git gives the same guarantee with neither, and the trust model is unchanged: we import
our own repositories' code. Third-party submissions still need one of those two.

The list of plugins now comes from the `plugins` registry table, which the docs always called the
authoritative half and which was in fact a by-product of what happened to be installed. That is what
lets `finances` be published without being installed here, and it spans orgs — fofos-admin lives under
fofos/, which an org listing would have missed. Adding one is `--add <clone url>`.

A plugin with no tag is left alone and named in the report, not deleted. It has made no release, so
there is nothing true to say about it; dropping its row would be a decision about somebody's plugin
dressed up as a refresh.

Also: `release_tag` on plugin_metadata, because the tag is the fact when a manifest version disagrees
with it; the sync warns on a mismatch. And icon URLs pin to the tag — `/raw/HEAD/` served whatever was
on the default branch, via a 303, so a plugin could show one icon and ship another.

Verified: 21/25 published from a release, 4 named as untagged, all 14 tag-pinned icons answer 200.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 13:05:50 +01:00
15 changed files with 729 additions and 267 deletions
+13 -9
View File
@@ -29,17 +29,21 @@ Ranked by what actually blocks something.
## Staleness ## Staleness
- [ ] **Nothing refreshes the catalogue automatically.** Publishing a plugin update changes nothing until - [x] **Nothing refreshes the catalogue automatically.** Done 2026-09-03: `scripts/marketplace-cron.sh`
somebody runs `bun --env-file=.env scripts/marketplace-sync.ts`. A Gitea webhook or a cron is the every minute. A Gitea webhook would make it a second instead, at the price of a public write route
obvious next step. on the one process here with no login — not worth it for a minute.
- [ ] **The sync reads local clones, not the remotes.** The machine running it must have every plugin - [x] **The sync reads local clones, not the remotes.** Done 2026-09-03. Bare mirrors, read at the newest
checked out, so the catalogue describes that machine's copies. The version that clones each repo tag, driven by the registry rather than by a directory listing. The catalogue no longer describes
into a temp directory is a small change. any machine's working copies.
- [ ] **`rawUrl()` assumes Gitea and a `main` branch.** A GitHub-hosted plugin, or one whose default branch - [ ] **`rawUrl()` assumes Gitea.** A GitHub-hosted plugin gets a 404 for its icon. The forge belongs in
is `master`, gets a 404 for its icon. The forge belongs in the registry rather than inferred from a the registry rather than inferred from a URL shape. (The `main`-vs-`master` half of this is gone —
URL shape. URLs pin to the release tag now.)
- [ ] **Four plugins have never been tagged**, so the store still shows rows the old sync took from a
working tree: av-club, kanban, openseo, fofos-admin. They are frozen until somebody tags them, and
the sync names them on every run somebody asks for.
## Scale ## Scale
+82 -17
View File
@@ -10,8 +10,9 @@ from. **The website installs nothing** — installing means cloning a repository
starting a process on someone else's machine, so it happens entirely inside the owner's own Officer. starting a process on someone else's machine, so it happens entirely inside the owner's own Officer.
``` ```
gitea.officer.dev/plugins/* the repositories (source of truth) gitea.officer.dev/** the repositories (source of truth)
│ scripts/marketplace-sync.ts (manual, reads manifest.ts + README.md) │ bare mirrors, read AT THE NEWEST TAG — src/servers/marketplace/sync.ts
│ every minute, scripts/marketplace-cron.sh
▼ ▼
officer_marketplace (postgres) plugins + plugin_metadata officer_marketplace (postgres) plugins + plugin_metadata
│ src/servers/marketplace/server.tsx :9100 │ src/servers/marketplace/server.tsx :9100
@@ -20,6 +21,7 @@ marketplace.officer.dev GET /api/plugins, /api/plugins/:appName
│ fetchCatalogue() live → cached → bundled │ fetchCatalogue() live → cached → bundled
▼ ▼
an Officer install /marketplace → clone → push schema → mount an Officer install /marketplace → clone → push schema → mount
and the update badge — see "Publishing" below
``` ```
## The parts ## The parts
@@ -78,19 +80,87 @@ disappearing from the store.
the clone, the plugin's own manifest is what the platform reads. A metadata row that disagrees with the the clone, the plugin's own manifest is what the platform reads. A metadata row that disagrees with the
repository is a marketplace bug, never a second source of truth. repository is a marketplace bug, never a second source of truth.
## Sync ## Publishing
**Push a tag. That is the whole procedure.** A timer on officer-dev picks it up within a minute and every
Officer install sees it on the next page load.
```bash ```bash
bun --env-file=.env scripts/marketplace-sync.ts bun --env-file=.env scripts/marketplace-sync.ts # all of them, now
bun --env-file=.env scripts/marketplace-sync.ts photos # one
bun --env-file=.env scripts/marketplace-sync.ts --add https://gitea.officer.dev/plugins/finances.git
bun --env-file=.env scripts/marketplace-sync.ts --force # re-publish rows this file's own logic changed
``` ```
Reads every directory under `plugins/`, imports its `manifest.ts`, derives `has` from the directory ### It reads releases, never a working tree
rather than from a declaration, reads `README.md`, and upserts. Re-running is always safe: it overwrites,
never merges.
It rewrites SSH remotes to HTTPS. `example` and `offscale` had `git@gitea…` origins, which would have Every registered plugin is mirrored bare into `~/.cache/officer-marketplace/mirrors` and read at its
handed every installer a URL that only works on the machine that published it — the install fails at newest tag — manifest, README and file list all out of `git cat-file`. There is no working tree in the
authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong". path, so publishing something uncommitted is not a mistake that can be made.
That is a change, and it was load-bearing. Until 2026-09-03 the sync imported `manifest.ts` off
`platform/plugins/` — the directories THIS machine has installed — so publishing meant "get officer-dev's
checkout to the release, then run the script", and nothing showed when that had not happened. On the day
it was rewritten the live store was advertising four plugins from untagged working copies: av-club
2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of which had ever
been tagged. It is also why the job could not go on a timer before: a cron over it published whatever was
in a working tree, on a schedule.
### The registry decides which plugins, and where from
The list comes from the `plugins` table — the half this document has always called authoritative, and
which was in fact a by-product of what happened to be checked out. Now it decides. A plugin nobody has
installed here can still be published, and a plugin lives wherever its row says: `fofos-admin` is under
`fofos/`, not `plugins/`, which an org listing would have missed. Registering one is `--add`.
### A plugin with no tag is left alone
It has never made a release, so the sync has nothing true to say about it: the existing row stays and the
run names it. Deleting it would drop somebody's plugin out of the store on the day the rule changed.
Tagging it is the fix, and the report is the reminder.
### `release_tag`, and why it is not `version`
`version` is the manifest's own claim; `release_tag` is what was pushed. They match every time somebody
tags correctly, and when they do not, the tag is the fact — a manifest whose version nobody bumped would
otherwise make a real release invisible to anything comparing versions. The sync warns on a mismatch.
Icon URLs pin to the tag too. `/raw/HEAD/…` served whatever was on the default branch, via a 303, so a
plugin could show one icon in the store and ship another.
### The timer
`scripts/marketplace-cron.sh`, `* * * * *`, logging to `officer-dev/marketplace-sync.log`. One
timestamped summary line per tick — always, including the ticks that do nothing, because a fresh
timestamp is the only thing that distinguishes "nothing to publish" from "the timer died". A plugin-level
line appears only when one actually moved.
A tick costs about 2.7s with nothing to do: one query, one fetch per repository, no writes. Nothing is
written unless the release commit differs from the one already published.
A Gitea webhook would cut the delay from a minute to a second, at the price of a public write route and a
shared secret. A minute was never the problem anybody had — the complaint that started this was a release
that never appeared at all.
### Remotes
`--add` rewrites SSH to HTTPS. A `git@gitea…` origin in the catalogue hands every installer a URL that
only works on the machine that published it, and the install fails at authentication — which reads as
"the plugin is broken" rather than "the catalogue is wrong".
## The update badge reads the catalogue
`updateable` on `/api/plugins` compares the tag at the checkout's HEAD with the tag the catalogue says is
newest. Difference, not ordering: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
then it fails silently.
Before this, each host ran a rate-limited background `git fetch` against all twenty-one repositories to
re-derive a fact one place already knew. That is gone. `releaseState()` still holds the git rule as a
fallback for a plugin the store cannot speak for — an unregistered checkout, or a store that could not be
reached — and in that case its refs are whatever the last `pullPlugin` left, which is the accepted cost.
`fetchCatalogue()`'s cache dropped from five minutes to one when the badge started depending on it:
otherwise the wait for a release to appear was the sync's minute plus the cache's five.
## The platform degrades, it does not break ## The platform degrades, it does not break
@@ -111,13 +181,8 @@ The full list, ranked, is `docs/marketplace-todo.md`. The ones that shape the de
- **`[blocker]` Third-party submissions cannot use this sync.** It reads a manifest by importing it, which - **`[blocker]` Third-party submissions cannot use this sync.** It reads a manifest by importing it, which
is code execution. Fine for our own repositories; not a mechanism a stranger's plugin can go through. is code execution. Fine for our own repositories; not a mechanism a stranger's plugin can go through.
Submissions need either a sandbox or a published `manifest.json` alongside the TypeScript one. Submissions need either a sandbox or a published `manifest.json` alongside the TypeScript one.
- **`[open]` Sync reads local clones, not the remotes.** The machine running it must have the plugins - **`[open]` `rawUrl()` assumes Gitea.** A GitHub-hosted plugin needs the other URL form. The forge should
checked out. The version that clones each repo into a temp directory is a small change and was left out become part of the registry rather than inferred from the URL.
of the first pass.
- **`[open]` `rawUrl()` assumes Gitea and a `main` branch.** A GitHub-hosted plugin or one defaulting to
`master` gets a 404 for its icon. The forge should become part of the registry rather than inferred.
- **`[open]` No versions table.** `plugin_metadata` is one row per plugin — a snapshot, not a history. A - **`[open]` No versions table.** `plugin_metadata` is one row per plugin — a snapshot, not a history. A
store that can offer "install 1.2 instead of 1.3" needs a different shape. store that can offer "install 1.2 instead of 1.3" needs a different shape.
- **`[open]` Nothing refreshes automatically.** Publishing a plugin update does not change the catalogue
until somebody runs the sync. A cron or a Gitea webhook is the obvious next step.
- **`[open]` No search, no categories, no counts.** Three plugins do not need them. - **`[open]` No search, no categories, no counts.** Three plugins do not need them.
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Publish every plugin release to the marketplace catalogue. Meant for cron, on the ONE machine that
# publishes.
#
# ── Why a timer at all, when a webhook would be instant ──
#
# The sync reads bare mirrors at their newest tag, so running it on a schedule cannot publish anything
# that was not pushed — which is exactly what made a timer unsafe before, and is no longer true. A Gitea
# webhook would cut the delay from a minute to a second, and it would also mean a public write route, a
# shared secret, and a second path into the same code. A minute is not the problem anybody had: the
# complaint that started this was a release that never appeared at all.
#
# ── What it costs ──
#
# About 2.7s per tick with nothing to do — one query, one fetch per repository, no writes. Gitea is on
# this same machine, so none of that leaves the box.
#
# ── The log ──
#
# One timestamped summary line per tick, ALWAYS, including the ticks that do nothing. A fresh timestamp
# is the only thing that distinguishes "nothing to publish" from "the timer died". Plugin-level lines
# appear only for a plugin that actually moved, which is what `--quiet` buys.
set -uo pipefail
PLATFORM="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
LOG="${MARKETPLACE_SYNC_LOG:-$(dirname "$PLATFORM")/marketplace-sync.log}"
LOCK="/tmp/officer-marketplace-sync.lock"
# Cron's PATH is three entries and bun is in none of them.
export PATH="$HOME/.bun/bin:/usr/local/bin:/usr/bin:/bin"
cd "$PLATFORM" || exit 1
# Skip rather than queue. A tick that finds the previous one still running has nothing to add — the run
# in progress will publish whatever this one would have.
exec 9>"$LOCK"
flock -n 9 || exit 0
bun --env-file=.env scripts/marketplace-sync.ts --quiet >>"$LOG" 2>&1
# Bounded without logrotate, because this is one file on one machine and a config file for it would be
# more moving parts than the thing it manages.
if [ "$(wc -l <"$LOG")" -gt 20000 ]; then
tail -n 5000 "$LOG" >"$LOG.tmp" && mv "$LOG.tmp" "$LOG"
fi
+77 -164
View File
@@ -1,43 +1,40 @@
#!/usr/bin/env bun #!/usr/bin/env bun
import { existsSync, readFileSync } from 'node:fs'; import { upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
import { basename, join, resolve } from 'node:path'; import { syncAll, syncPlugin } from '../src/servers/marketplace/sync';
import { upsertMetadata, upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
import { discoverPlugins } from '../src/servers/plugins/discover';
// Fill the marketplace database from the plugin repositories. /**
// * Fill the marketplace catalogue from the plugins' releases.
// ── What this reads, and why it is not the marketplace's own opinion ── *
// * bun --env-file=.env scripts/marketplace-sync.ts every registered plugin
// Every field except "where to clone from" comes out of the plugin's `manifest.ts` and its README. The * bun --env-file=.env scripts/marketplace-sync.ts photos just one
// marketplace stores a COPY so the website can render without cloning three repositories per page load; * bun --env-file=.env scripts/marketplace-sync.ts --add <url> register a new one, then sync it
// the manifest stays the source of truth, and the platform re-reads it after cloning. A row here that * bun --env-file=.env scripts/marketplace-sync.ts --force re-publish releases already published
// disagrees with the repository is a stale cache, not a second opinion — which is why re-running this is * bun --env-file=.env scripts/marketplace-sync.ts --quiet print only what changed
// always safe and never merges: it overwrites. *
// * All the reasoning is in `src/servers/marketplace/sync.ts` — this file is the command line around it.
// ── Reading the manifest by importing it ── * It is a thin wrapper on purpose: the same functions are what the Gitea webhook calls, so a plugin
// * published by hand and one published by pushing a tag go through exactly the same code.
// A manifest is TypeScript, not JSON, so it is imported and evaluated rather than parsed. That is code *
// execution, and it is fine for OUR repositories and not fine as a general mechanism. * Re-running is always safe. It overwrites and never merges.
// */
// `[blocker for third-party submissions]` A stranger's plugin cannot be imported by the machine that
// serves the catalogue. Whoever builds submissions has to either evaluate manifests in a sandbox or make
// authors publish a `manifest.json` alongside. Recorded rather than solved: nobody can submit yet.
//
// ── Local clones, for now ──
//
// It syncs from `plugins/` on this machine rather than by cloning each repo into a temp directory. That
// keeps tonight's scope to "get real data into the database" — and it means the machine running the sync
// must have the plugins checked out. See `docs/marketplace.md` → "Sync" for the version that clones.
type Sync = { appName: string; developer: string; repoUrl: string }; const argv = process.argv.slice(2);
/*
`--quiet` is what the timer runs: one line per plugin that actually moved, and a summary. Not silence —
the summary is always printed, because a log whose last line is old is the only way to tell "nothing to
publish" from "the timer died", and that question does come up.
*/
const quiet = argv.includes('--quiet');
const force = argv.includes('--force');
const args = argv.filter((a) => a !== '--quiet' && a !== '--force');
/** /**
* A clone URL the whole internet can use. * A clone URL the whole internet can use.
* *
* `plugins/example` and `plugins/offscale` have `git@gitea.officer.dev:plugins/<name>.git` remotes, * Several plugin repos have `git@gitea.officer.dev:plugins/<name>.git` remotes, because whoever set them
* because whoever set them up had a key on that host. Publishing that to a catalogue would hand every * up had a key on that host. Registering one of those would hand every installer a URL that only works
* installer a URL that only works on the machine it was published from — the install fails at * on the machine it was published from — the install fails at authentication, which reads as "the plugin
* authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong". * is broken" rather than "the catalogue is wrong".
*/ */
export function toHttpsRemote(remote: string): string | null { export function toHttpsRemote(remote: string): string | null {
const trimmed = remote.trim(); const trimmed = remote.trim();
@@ -60,142 +57,58 @@ export function toHttpsRemote(remote: string): string | null {
return null; return null;
} }
const git = (dir: string, args: string[]): string | null => { if (args[0] === '--add') {
const result = Bun.spawnSync(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' }); const raw = args[1];
if (result.exitCode !== 0) return null; if (!raw) {
return new TextDecoder().decode(result.stdout).trim() || null; console.error('usage: marketplace-sync.ts --add <clone url>');
}; process.exit(1);
/*
Which plugins directory to publish FROM.
There is one: `platform/plugins/`, holding what this machine has INSTALLED — updated by
`plugins:update`, which is a git pull, so it sits at whatever version was last installed.
Publishing from it means the catalogue advertises the version this machine happens to be running
rather than the version that was released. That is exactly what happened on 2026-08-27: 21 plugins
were tagged and the store went on showing 1.0.0.
A second directory — `officer-dev/plugins/`, every repo in the org cloned side by side — was created
to fix that, and never did: its entries were SYMLINKS into this one, so both paths read the same
working copy and published the same version. It was removed on 2026-08-28.
**The problem is still open, and a second checkout was never the answer.** Each directory here is a
real git repo with its own tags, so the released manifest can be read out of the tag itself —
`git show <newest-tag>:manifest.ts` — without caring what is checked out. That is the fix; it is not
written yet, and until it is, this script advertises what is installed.
*/
const PLUGINS_DIR = process.env.PLUGINS_DIR ?? resolve(import.meta.dir, '..', 'plugins');
async function syncOne(dir: string): Promise<Sync | null> {
const appName = basename(dir);
const manifestPath = join(dir, 'manifest.ts');
if (!existsSync(manifestPath)) {
console.warn(`[marketplace-sync] ${appName}: no manifest.ts — skipped`);
return null;
} }
const repoUrl = toHttpsRemote(raw);
const remote = git(dir, ['remote', 'get-url', 'origin']);
const repoUrl = remote ? toHttpsRemote(remote) : null;
if (!repoUrl) { if (!repoUrl) {
console.warn(`[marketplace-sync] ${appName}: no usable https remote (${remote ?? 'none'}) — skipped`); console.error(`[marketplace-sync] ${raw} is not a clone URL this can turn into https`);
return null; process.exit(1);
}
const appName = repoUrl.replace(/\.git$/, '').split('/').pop() ?? '';
if (!/^[a-z0-9][a-z0-9-]{0,63}$/.test(appName)) {
console.error(`[marketplace-sync] "${appName}" is not a usable app name`);
process.exit(1);
} }
const { manifest } = (await import(manifestPath)) as { manifest: Record<string, unknown> }; // Registered with a placeholder developer, then immediately synced — which reads the real publisher
if (!manifest) { // out of the released manifest and overwrites it. A repository with no tag stays registered and
console.warn(`[marketplace-sync] ${appName}: manifest.ts exports no \`manifest\` — skipped`); // unpublished, which is the correct state for one that has not made a release yet.
return null; await upsertPlugin({ appName, developer: 'unknown', repoUrl });
console.log(`[marketplace-sync] registered ${appName} → ${repoUrl}`);
const one = await syncPlugin(appName, force);
report([one]);
process.exit(one.status === 'failed' ? 1 : 0);
}
const results = args[0] ? [await syncPlugin(args[0], force)] : await syncAll(force);
report(results);
process.exit(results.some((r) => r.status === 'failed') ? 1 : 0);
function report(results: Awaited<ReturnType<typeof syncAll>>): void {
const stamp = new Date().toISOString();
for (const r of results) {
if (r.status === 'synced') console.log(`[marketplace-sync] ${r.appName} ← ${r.tag}`);
else if (r.status === 'failed') console.error(`[marketplace-sync] ${r.appName} failed: ${r.error}`);
else if (quiet) continue;
else if (r.status === 'unchanged') console.log(`[marketplace-sync] ${r.appName} = ${r.tag}`);
else console.warn(`[marketplace-sync] ${r.appName}: no tag — nothing published`);
} }
// What the plugin SHIPS, read from the directory rather than declared — the same rule discovery const synced = results.filter((r) => r.status === 'synced').length;
// follows, so a catalogue entry cannot claim a sidecar the repository does not have. const unchanged = results.filter((r) => r.status === 'unchanged').length;
const has = { const unreleased = results.filter((r) => r.status === 'unreleased').map((r) => r.appName);
api: existsSync(join(dir, 'api', 'router.ts')), const failed = results.filter((r) => r.status === 'failed').map((r) => r.appName);
schema: existsSync(join(dir, 'db', 'schema.ts')),
sidecar: existsSync(join(dir, 'sidecar', 'index.ts')) || existsSync(join(dir, 'sidecar', 'index.mjs')),
web: existsSync(join(dir, 'web')),
};
const readmePath = ['README.md', 'readme.md'].map((n) => join(dir, n)).find((p) => existsSync(p)); console.log(`[marketplace-sync] ${stamp} — ${synced} published, ${unchanged} unchanged, ${unreleased.length} untagged`);
const readme = readmePath ? readFileSync(readmePath, 'utf-8') : null; // Named rather than counted, and not under `--quiet`: "4 untagged" is a number nobody acts on, but a
// timer that reprints the same four names every minute is a log nobody reads. The names belong in a
const developer = String(manifest.publisher ?? 'unknown'); // run somebody asked for.
const pluginId = await upsertPlugin({ appName, developer, repoUrl }); if (unreleased.length && !quiet) {
console.warn(`[marketplace-sync] never tagged, so not updated: ${unreleased.join(', ')}`);
await upsertMetadata(pluginId, { }
version: String(manifest.version ?? '0.0.0'), if (failed.length) console.error(`[marketplace-sync] failed: ${failed.join(', ')}`);
platform: String(manifest.platform ?? '>=1.0.0'),
label: String(manifest.label ?? appName),
summary: String(manifest.summary ?? ''),
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
// quietly disagreeing with the repository it claims to mirror.
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
color: (manifest.color as string) ?? null,
tile: (manifest.tile as string) ?? null,
// Artwork lives in the repo, so the URL is built from the remote rather than stored by hand. Only
// claimed when the file is actually there — a broken image is worse than a glyph.
iconUrl: existsSync(join(dir, 'assets', 'icon.png')) ? rawUrl(repoUrl, 'assets/icon.png') : null,
icon: (manifest.icon as string) ?? null,
permissions: manifest.permissions ?? [],
osDependencies: manifest.osDependencies ?? null,
has,
readme,
sourceCommit: git(dir, ['rev-parse', 'HEAD']),
});
console.log(
`[marketplace-sync] ${appName} ← ${repoUrl}${readme ? '' : ' (no README)'}${has.web ? '' : ' (no web/)'}`,
);
return { appName, developer, repoUrl };
} }
/**
* A raw-file URL for a Gitea repository.
*
* `HEAD`, not a branch name. This said `branch/main` and the comment here warned that "a repo whose
* default branch is `master` gets a 404" — which then happened: four plugin repositories were created
* on `master` and every one of them served a broken image in the marketplace on any server that reads
* the remote catalogue. A warning in a comment is not a guard.
*
* Gitea resolves `HEAD` to whatever the default branch is, so the question stops being asked. Verified
* against both spellings.
*
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
* be part of the registry rather than inferred from the URL.
*/
function rawUrl(repoUrl: string, path: string): string {
return `${repoUrl.replace(/\.git$/, '')}/raw/HEAD/${path}`;
}
const dirs = existsSync(PLUGINS_DIR)
? [...new Bun.Glob('*/').scanSync({ cwd: PLUGINS_DIR, onlyFiles: false })]
// Not a plugin: the shared install target for plugin npm packages, which lives here because bun
// resolves node_modules up the tree. `discoverPlugins` skips it for the same reason.
.filter((d) => d.replace(/\/$/, '') !== 'node_modules')
.map((d) => join(PLUGINS_DIR, d))
: [];
if (dirs.length === 0) {
console.log(`[marketplace-sync] no plugin directories under ${PLUGINS_DIR} — nothing to sync`);
process.exit(0);
}
// Discovery is not used to FIND them — a plugin that fails discovery is exactly the kind of thing a
// catalogue should still be able to describe — but its broken list is worth surfacing here.
const { broken } = await discoverPlugins().catch(() => ({ broken: [] as { appName: string; error: string }[] }));
for (const { appName, error } of broken) console.warn(`[marketplace-sync] ${appName} is broken on disk: ${error}`);
const synced: Sync[] = [];
for (const dir of dirs) {
const result = await syncOne(dir).catch((err) => {
console.error(`[marketplace-sync] ${basename(dir)} failed: ${err instanceof Error ? err.message : err}`);
return null;
});
if (result) synced.push(result);
}
console.log(`[marketplace-sync] ${synced.length}/${dirs.length} plugin(s) in the catalogue`);
process.exit(0);
@@ -14,6 +14,14 @@ export type CatalogueEntry = {
appName: string; appName: string;
publisher: string; publisher: string;
version: string; version: string;
/**
* The git tag this entry was published from.
*
* Optional on the wire, not in the database, and the two mean different things. Here it is absent for
* a row that predates release-based publishing OR for a catalogue too old to send the field at all —
* so a consumer must treat "no tag" as "cannot tell" and fall back to whatever it knew before.
*/
releaseTag?: string;
platform: string; platform: string;
label: string; label: string;
summary: string; summary: string;
@@ -52,6 +60,7 @@ function toCatalogueEntry(row: Row): CatalogueEntry | null {
appName: row.appName, appName: row.appName,
publisher: row.developer, publisher: row.developer,
version: m.version, version: m.version,
...(m.releaseTag ? { releaseTag: m.releaseTag } : {}),
platform: m.platform, platform: m.platform,
label: m.label, label: m.label,
summary: m.summary, summary: m.summary,
@@ -106,6 +115,36 @@ export async function catalogueEntry(appName: string): Promise<CatalogueEntry |
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) }; return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
} }
/**
* A registry row, plus the one piece of metadata the sync needs before deciding to do any work.
*
* `sourceCommit` is what the last sync published, and it is here rather than fetched per plugin so a
* run that has nothing to do costs one query instead of twenty-five. Null means never synced.
*/
export type RegistryRow = { id: number; appName: string; repoUrl: string; sourceCommit: string | null };
/**
* Every registered plugin — identity and clone URL, no metadata.
*
* This is what the sync iterates, and it is the reason the registry is called authoritative: the
* catalogue lists what somebody deliberately registered, not what happens to be checked out on the
* machine that runs the job. Unlisted rows are included; hiding a plugin from the public API is not the
* same as no longer tracking it, and one taken down for a bad release has to be able to come back
* current.
*/
export async function listRegistry(): Promise<RegistryRow[]> {
return marketplaceDb
.select({
id: plugins.id,
appName: plugins.appName,
repoUrl: plugins.repoUrl,
sourceCommit: pluginMetadata.sourceCommit,
})
.from(plugins)
.leftJoin(pluginMetadata, eq(pluginMetadata.pluginId, plugins.id))
.orderBy(plugins.appName);
}
// ── Writes. Used by the sync script, never by the public API ── // ── Writes. Used by the sync script, never by the public API ──
type RegisterParams = { appName: string; developer: string; repoUrl: string }; type RegisterParams = { appName: string; developer: string; repoUrl: string };
@@ -73,6 +73,19 @@ export const pluginMetadata = pgTable(
.references(() => plugins.id, { onDelete: 'cascade' }), .references(() => plugins.id, { onDelete: 'cascade' }),
version: text('version').notNull(), version: text('version').notNull(),
/**
* The git tag this row was read from — what "published" actually means.
*
* `version` above is the manifest's own claim, and the two are the same string every time somebody
* tags correctly. They are kept apart because when they DISAGREE the tag is the fact: it is what was
* pushed, and a manifest whose version nobody bumped would otherwise make a real release invisible
* to anything comparing versions.
*
* Nullable because rows existed before releases were the unit of publishing. A null one is a
* snapshot taken from a working tree by the old sync, and every consumer should read it as "unknown"
* rather than "no release".
*/
releaseTag: text('release_tag'),
/** The platform range it declares support for, e.g. `>=1.0.0`. */ /** The platform range it declares support for, e.g. `>=1.0.0`. */
platform: text('platform').notNull(), platform: text('platform').notNull(),
label: text('label').notNull(), label: text('label').notNull(),
+4 -2
View File
@@ -4,7 +4,7 @@ import { isSuperAdmin } from '../../super-admin';
import { mountPrefix } from '../../plugins/manifest'; import { mountPrefix } from '../../plugins/manifest';
import { snapshotPlugins } from '../../plugins/mount'; import { snapshotPlugins } from '../../plugins/mount';
import { manualInstallHint, reportDependencies } from '../../plugins/os-deps'; import { manualInstallHint, reportDependencies } from '../../plugins/os-deps';
import { fetchCatalogue, fetchReadme } from '../../plugins/marketplace'; import { fetchCatalogue, fetchReadme, publishedTags } from '../../plugins/marketplace';
import { findOrphanTables } from '../../plugins/orphans'; import { findOrphanTables } from '../../plugins/orphans';
import { fetchPlugin } from '../../plugins/fetch'; import { fetchPlugin } from '../../plugins/fetch';
import { discoverPlugins } from '../../plugins/discover'; import { discoverPlugins } from '../../plugins/discover';
@@ -57,7 +57,9 @@ pluginsRouter.get('/catalogue/:appName/readme', async (ctx) => {
}); });
pluginsRouter.get('/', async (ctx) => { pluginsRouter.get('/', async (ctx) => {
const { states, broken } = await snapshotPlugins(); // The catalogue, for the update badge — cached, never throws, and falls back to the git rule per
// plugin when the store has nothing to say. See `plugins/release.ts`.
const { states, broken } = await snapshotPlugins(await publishedTags());
// Asked per plugin rather than once, because `pm2 jlist` is a fork and most plugins have no sidecar to // Asked per plugin rather than once, because `pm2 jlist` is a fork and most plugins have no sidecar to
// ask about. A plugin that is installed and enabled but whose process is not online is the state worth // ask about. A plugin that is installed and enabled but whose process is not online is the state worth
// rendering differently — it is the difference between "off" and "broken". // rendering differently — it is the difference between "off" and "broken".
+8 -3
View File
@@ -18,9 +18,14 @@ import { catalogueEntry, listCatalogue } from '../../databases/marketplace_db/sr
* *
* ── Read-only, and structurally so ── * ── Read-only, and structurally so ──
* *
* There are two GET routes and no others. Registering a plugin is `scripts/marketplace-sync.ts`, run by * There are two GET routes and no others. Everything that WRITES the catalogue — registering a plugin,
* hand against the database. That is not a temporary shortcut standing in for a write API: while * publishing a release — is `scripts/marketplace-sync.ts` against the database, on the one machine that
* submissions do not exist, a public write surface is all risk and no feature. * publishes. That is not a temporary shortcut standing in for a write API: while submissions do not
* exist, a public write surface is all risk and no feature.
*
* It stayed that way when publishing went onto a timer, which is the moment it would have been easy to
* add a `POST /api/sync` for a Gitea webhook. A minute of delay is cheaper than a write route on the one
* process here that has no login.
* *
* ── CORS is wide open, deliberately ── * ── CORS is wide open, deliberately ──
* *
+301
View File
@@ -0,0 +1,301 @@
import { existsSync, mkdirSync } from 'node:fs';
import { homedir, tmpdir } from 'node:os';
import { join } from 'node:path';
import {
listRegistry,
type RegistryRow,
upsertMetadata,
upsertPlugin,
} from '../../databases/marketplace_db/src/plugins/queries';
/**
* Fill the catalogue from what each plugin has RELEASED.
*
* ── The bug this file exists to remove ──
*
* Until 2026-09-03 the sync read `platform/plugins/` — the directories this machine has INSTALLED — and
* imported each `manifest.ts` off the working tree. So "publish" meant "make sure officer-dev's checkout
* happens to be at the release, then run the script", and nobody could see when it was not. On the day
* this was rewritten the live catalogue was advertising four plugins from untagged working copies:
* av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of
* which had ever been tagged. The store was describing somebody's afternoon.
*
* That is also why the old script could not be put on a timer: a cron over it would have published
* whatever was sitting in a working tree, on a schedule.
*
* ── What replaces it: a bare mirror, read at a tag ──
*
* Every plugin in the registry is mirrored (`git clone --mirror`) into a cache directory, refreshed, and
* read AT ITS NEWEST TAG — the manifest, the README, the file list, all out of `git cat-file`. There is
* no working tree anywhere in this path, so publishing something uncommitted is not a mistake that can
* be made. A mirror is cheap: the whole set clones in a couple of seconds and refreshes in well under
* one.
*
* `docs/marketplace.md` proposed instead reading manifests over Gitea's API, which needed either a
* sandbox to evaluate a fetched string or a committed `manifest.json` for every plugin. Git gives the
* same guarantee — content at a named release, independent of any checkout — with neither, and the trust
* model is unchanged from today: we import our own repositories' code.
*
* `[still a blocker for third-party submissions]` It is unchanged BECAUSE it is unchanged. Importing a
* stranger's manifest is code execution no matter which ref it came from. Submissions still need a
* sandbox or a published `manifest.json`.
*
* ── Which plugins, and where from: the registry, and only the registry ──
*
* The list comes from the `plugins` table, which `docs/marketplace.md` has always called the
* authoritative, hand-curated half. It was not actually being used that way — the old sync discovered
* plugins by listing a directory, so the registry was a by-product of whatever officer-dev had
* installed. Now it decides, which means:
*
* · a plugin nobody has installed here can still be published (`finances` is in exactly that state),
* · a plugin lives wherever its registry row says, across orgs — `fofos-admin` is under `fofos/`,
* not `plugins/`, so an org listing would have missed it,
* · and adding one is a deliberate act: `marketplace-sync.ts --add <https clone url>`.
*
* ── A plugin with no tag is left ALONE, not removed ──
*
* It has never made a release, so this file has nothing true to say about it and says nothing: the
* existing metadata row stays, and the run reports it. Deleting the row would drop it out of the store
* on the day the rule changed, which is a decision about somebody's plugin dressed up as a refresh.
* Tagging it is the fix, and the report is there to ask for one.
*/
export type SyncOutcome =
| { appName: string; status: 'synced'; tag: string; version: string }
/** The release on the remote is the one already published. Nothing was written. */
| { appName: string; status: 'unchanged'; tag: string }
| { appName: string; status: 'unreleased' }
| { appName: string; status: 'failed'; error: string };
/*
Where the mirrors live.
A fixed path under the user's cache directory, NOT one derived from the environment. This runs three
ways — by hand with `--env-file=.env`, from the webhook inside `officer-marketplace`, and from cron —
and those three do not have the same environment. A path that moved with `DATA_PATH` would have each
of them re-cloning twenty-five repositories into its own copy, and the "is it already mirrored" check
would answer differently depending on who asked.
Not in the source tree, because it is a cache of remote repositories and a `.gitignore` entry is a
worse way to keep several hundred megabytes out of a repo than not putting them there.
*/
const MIRROR_DIR =
process.env.MARKETPLACE_MIRROR_DIR ?? join(homedir(), '.cache', 'officer-marketplace', 'mirrors');
const GIT_TIMEOUT_MS = 60_000;
type Git = { ok: true; out: string } | { ok: false; err: string };
async function git(args: string[], cwd?: string): Promise<Git> {
// A missing `cwd` makes posix_spawn fail with ENOENT, which reads as "git is not installed" and sent
// a first run of this looking for git for ten minutes. It means the mirror is not there yet.
if (cwd && !existsSync(cwd)) return { ok: false, err: `${cwd} does not exist` };
const proc = Bun.spawn(['git', ...args], {
cwd,
stdout: 'pipe',
stderr: 'pipe',
// Never inherited: a mirror fetch that hits a host asking for a password would otherwise sit on a
// terminal prompt forever, and in the webhook server there is no terminal to sit on.
env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GIT_ASKPASS: '/bin/true' },
signal: AbortSignal.timeout(GIT_TIMEOUT_MS),
});
const [out, err] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]);
return (await proc.exited) === 0 ? { ok: true, out: out.trim() } : { ok: false, err: err.trim() || 'git failed' };
}
/** The mirror for a plugin, cloned if this is the first time and refreshed if it is not. */
async function mirror(appName: string, repoUrl: string): Promise<string> {
mkdirSync(MIRROR_DIR, { recursive: true });
const dir = join(MIRROR_DIR, `${appName}.git`);
// Re-cloned rather than repaired when it is not a healthy bare repo: an interrupted clone leaves a
// directory that every later command fails against, and there is nothing in it worth keeping.
const existing = await git(['rev-parse', '--is-bare-repository'], dir);
if (!existing.ok || existing.out !== 'true') {
if (existsSync(dir)) await Bun.$`rm -rf ${dir}`.quiet().nothrow();
const cloned = await git(['clone', '--mirror', '--quiet', repoUrl, dir]);
if (!cloned.ok) throw new Error(`clone failed: ${cloned.err}`);
return dir;
}
// The registry is allowed to move a plugin. Setting the URL every time is one command and removes the
// failure where a repo is relocated and the mirror keeps refreshing from where it used to be.
await git(['remote', 'set-url', 'origin', repoUrl], dir);
// `--prune --prune-tags`: a deleted or MOVED tag has to disappear here too. Without it, a tag that was
// pushed by mistake and deleted would stay in the mirror forever and could go on being published as
// the newest release — a store advertising a version its repository no longer has.
const fetched = await git(['fetch', '--quiet', '--prune', '--prune-tags', '--tags', 'origin'], dir);
if (!fetched.ok) throw new Error(`fetch failed: ${fetched.err}`);
return dir;
}
/**
* The newest release, or null for a plugin that has never made one.
*
* Asked of git rather than by sorting version strings, which is the same rule `servers/plugins/release.ts`
* follows and for the same reason: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
* then fails silently. `--abbrev=0` gives the closest tag reachable from HEAD, and a mirror's HEAD is the
* remote's default branch, so this is "the newest release on the branch the repository points at".
*/
async function newestTag(dir: string): Promise<string | null> {
const described = await git(['describe', '--tags', '--abbrev=0', 'HEAD'], dir);
return described.ok ? described.out : null;
}
/** A file at a ref, or null when it is not in that tree. */
async function fileAt(dir: string, ref: string, path: string): Promise<string | null> {
const shown = await git(['cat-file', '-p', `${ref}:${path}`], dir);
return shown.ok ? shown.out : null;
}
/**
* The manifest at a release, as an object.
*
* Written to a temp file and imported, because a manifest is TypeScript. Keyed by the COMMIT, so the
* module cache is correct rather than merely convenient: the same commit is the same file, and a tag
* moved to a different commit gets a different path and a fresh import. That matters in the webhook
* server, which is long-lived and would otherwise serve the first version of a manifest it ever read.
*
* The `import type { PluginManifest }` at the top of every manifest is erased by the transpiler, so the
* file has no runtime dependency on the platform's path aliases and imports fine from anywhere.
*/
async function manifestAt(dir: string, ref: string, appName: string, commit: string): Promise<Record<string, unknown>> {
const source = await fileAt(dir, ref, 'manifest.ts');
if (source === null) throw new Error(`no manifest.ts at ${ref}`);
const scratch = join(tmpdir(), 'officer-marketplace-manifests');
mkdirSync(scratch, { recursive: true });
const path = join(scratch, `${appName}-${commit}.ts`);
await Bun.write(path, source);
const mod = (await import(path)) as { manifest?: Record<string, unknown> };
if (!mod.manifest) throw new Error(`manifest.ts at ${ref} exports no \`manifest\``);
return mod.manifest;
}
/**
* A raw-file URL for a Gitea repository, PINNED TO THE RELEASE.
*
* It used to be `/raw/HEAD/…`, which works only because Gitea 303s it to `/raw/branch/HEAD/…`; the
* artwork it served was whatever was on the default branch, so a plugin could show one icon in the store
* and ship another. `/raw/tag/<tag>/…` answers 200 directly and shows the icon that belongs to the
* version being advertised.
*
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
* be part of the registry rather than inferred from the URL.
*/
function rawUrl(repoUrl: string, tag: string, path: string): string {
return `${repoUrl.replace(/\.git$/, '')}/raw/tag/${encodeURIComponent(tag)}/${path}`;
}
/*
`force` re-publishes a release that is already published.
Needed because "unchanged" is judged on the plugin's commit, and that is only half the input: the other
half is THIS file. Change how a field is derived — as the switch to tag-pinned icon URLs did — and every
row is stale while every commit still matches. Then the run to fix them all is `--force`.
*/
async function syncRegistered(row: RegistryRow, force: boolean): Promise<SyncOutcome> {
const { appName, repoUrl } = row;
const dir = await mirror(appName, repoUrl);
const tag = await newestTag(dir);
if (!tag) return { appName, status: 'unreleased' };
const commitAt = await git(['rev-list', '-n', '1', tag], dir);
if (!commitAt.ok) throw new Error(`cannot resolve ${tag}`);
const commit = commitAt.out;
// Before the manifest is read, not after: this is what makes a per-minute run affordable. Twenty-five
// repositories with nothing new cost one query, twenty-five fetches of nothing, and no writes at all.
if (!force && row.sourceCommit === commit) return { appName, status: 'unchanged', tag };
const manifest = await manifestAt(dir, tag, appName, commit);
// What the plugin SHIPS, read from the tree at the release rather than from a declaration — the same
// rule discovery follows on disk, so a catalogue entry cannot claim a sidecar the release does not
// have.
const listed = await git(['ls-tree', '-r', '--name-only', tag], dir);
const files = new Set(listed.ok ? listed.out.split('\n') : []);
const has = {
api: files.has('api/router.ts'),
schema: files.has('db/schema.ts'),
sidecar: files.has('sidecar/index.ts') || files.has('sidecar/index.mjs'),
web: [...files].some((f) => f.startsWith('web/')),
};
const readme = (await fileAt(dir, tag, 'README.md')) ?? (await fileAt(dir, tag, 'readme.md'));
const version = String(manifest.version ?? tag);
if (version !== tag) {
// Not an error and not corrected here. The tag is what was released and the manifest is what the
// plugin will report about itself once installed; when they disagree, an update badge that compares
// versions goes quiet. Saying so is the only way anyone finds out.
console.warn(`[marketplace-sync] ${appName}: tag ${tag} but manifest says ${version} — they should match`);
}
const developer = String(manifest.publisher ?? 'unknown');
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
await upsertMetadata(pluginId, {
version,
releaseTag: tag,
platform: String(manifest.platform ?? '>=1.0.0'),
label: String(manifest.label ?? appName),
summary: String(manifest.summary ?? ''),
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
// quietly disagreeing with the repository it claims to mirror.
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
color: (manifest.color as string) ?? null,
tile: (manifest.tile as string) ?? null,
iconUrl: files.has('assets/icon.png') ? rawUrl(repoUrl, tag, 'assets/icon.png') : null,
icon: (manifest.icon as string) ?? null,
permissions: manifest.permissions ?? [],
osDependencies: manifest.osDependencies ?? null,
has,
readme,
sourceCommit: commit,
});
return { appName, status: 'synced', tag, version };
}
/** One plugin by app name. Unknown to the registry is a failure, not a silent no-op. */
export async function syncPlugin(appName: string, force = false): Promise<SyncOutcome> {
const row = (await listRegistry()).find((r) => r.appName === appName);
if (!row) return { appName, status: 'failed', error: 'not in the registry' };
return syncRegistered(row, force).catch((err) => ({
appName,
status: 'failed' as const,
error: err instanceof Error ? err.message : String(err),
}));
}
/**
* Every registered plugin, one at a time.
*
* Sequential on purpose. Twenty-four concurrent fetches against one Gitea is a thundering herd for a job
* nothing is waiting on, and the whole set finishes in about a second anyway.
*
* Unlisted rows are included: `listed` hides a plugin from the public API, usually after a bad release,
* and it has to be able to come back with current metadata rather than with whatever was true when it
* was pulled.
*/
export async function syncAll(force = false): Promise<SyncOutcome[]> {
const rows = await listRegistry();
const out: SyncOutcome[] = [];
for (const row of rows) {
out.push(
await syncRegistered(row, force).catch((err) => ({
appName: row.appName,
status: 'failed' as const,
error: err instanceof Error ? err.message : String(err),
})),
);
}
return out;
}
+34 -1
View File
@@ -39,6 +39,15 @@ export type CatalogueEntry = {
appName: string; appName: string;
publisher: string; publisher: string;
version: string; version: string;
/**
* The git tag this entry was published from — the newest RELEASE, as opposed to whatever a manifest
* claims about itself.
*
* Optional, and absent means "cannot tell" rather than "no release": an older catalogue does not send
* the field at all, and a plugin that has never been tagged has nothing to put in it. Both send the
* badge back to the git rule.
*/
releaseTag?: string;
platform: string; platform: string;
label: string; label: string;
summary: string; summary: string;
@@ -556,7 +565,15 @@ const MARKETPLACE_URL = process.env.MARKETPLACE_URL ?? 'https://marketplace.offi
* plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a * plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a
* restart is always a fresh look. * restart is always a fresh look.
*/ */
const CACHE_TTL_MS = 5 * 60 * 1000; /*
One minute, and the number changed when the badge started depending on it.
Five was fine while this only drew the store's tiles; it is not fine now that "is there an update" is
answered from the catalogue, because the wait for a release to appear became the sync's minute PLUS
this one. The cost is one small request per process per minute, and only while somebody is looking at
a page that lists plugins.
*/
const CACHE_TTL_MS = 60 * 1000;
/** /**
* How long to wait for the store before giving up on it. * How long to wait for the store before giving up on it.
@@ -639,6 +656,22 @@ export async function fetchCatalogue(): Promise<CatalogueEntry[]> {
} }
} }
/**
* What the store says the newest release of each plugin is, keyed by app name.
*
* This is the fact the update badge is built on. It replaces every host in the fleet running `git fetch`
* against twenty-one repositories to re-derive something the catalogue already knows — see
* `plugins/release.ts` for the rule and for what happens to a plugin that has no entry here.
*
* A missing key and a null value mean the same thing to a caller and are kept apart for the reader: no
* entry at all (unregistered, or a store that could not be reached) versus an entry with no release yet.
* Both fall through to the git rule, which is the only honest answer when the store cannot speak.
*/
export async function publishedTags(): Promise<Map<string, string | null>> {
const entries = await fetchCatalogue().catch(() => []);
return new Map(entries.map((e) => [e.appName, e.releaseTag ?? null]));
}
/** One entry by app name, or null. Used by install to find where to clone from. */ /** One entry by app name, or null. Used by install to find where to clone from. */
export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> { export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
const all = await fetchCatalogue().catch(() => BUNDLED); const all = await fetchCatalogue().catch(() => BUNDLED);
+8 -2
View File
@@ -59,7 +59,13 @@ export type PluginsSnapshot = {
* the tree while the row stayed, and there is nothing to mount, describe or offer. The row is left in the * the tree while the row stayed, and there is nothing to mount, describe or offer. The row is left in the
* database on purpose — deleting it here would turn "somebody moved the checkout" into silent data loss. * database on purpose — deleting it here would turn "somebody moved the checkout" into silent data loss.
*/ */
export async function snapshotPlugins(): Promise<PluginsSnapshot> { /**
* @param published Newest released tag per plugin, from the marketplace — `publishedTags()`. Passed by
* the one caller that renders update badges, and by nothing else on purpose: every other caller wants
* "what is on disk", and boot is among them. Making this function fetch the catalogue itself would put
* a network request with a timeout in front of mounting plugins on a machine with no internet.
*/
export async function snapshotPlugins(published?: Map<string, string | null>): Promise<PluginsSnapshot> {
const [{ plugins, broken }, installs] = await Promise.all([discoverPlugins(), listPluginInstalls()]); const [{ plugins, broken }, installs] = await Promise.all([discoverPlugins(), listPluginInstalls()]);
const byName = new Map(installs.map((row) => [row.appName, row])); const byName = new Map(installs.map((row) => [row.appName, row]));
@@ -72,7 +78,7 @@ export async function snapshotPlugins(): Promise<PluginsSnapshot> {
plugin, plugin,
install, install,
outdated: !!install && install.version !== plugin.manifest.version, outdated: !!install && install.version !== plugin.manifest.version,
release: await releaseState(plugin.dir), release: await releaseState(plugin.dir, published?.get(plugin.appName) ?? null),
}; };
}), }),
); );
+55
View File
@@ -94,6 +94,61 @@ describe('releaseState', () => {
expect(s.updateable).toBe(true); expect(s.updateable).toBe(true);
}); });
/*
The catalogue path. These are the ones that matter now: a host no longer fetches to find out what the
newest release is, so every assertion below is about the store's tag winning over local refs — and
about the fallback still being there when it has nothing to say.
*/
test('the published tag decides, without the checkout having ever fetched it', async () => {
const { work, origin, clone } = pair('published-newer');
// The clone is at v1 and has NEVER seen v2 — no fetch, no remote ref for it. Exactly the state a
// host is in a minute after somebody else tagged a release.
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
const s = await releaseState(clone, 'v2');
expect(s.tagAtHead).toBe('v1');
expect(s.newestTag).toBe('v2');
expect(s.updateable).toBe(true);
expect(s.reason).toBe('new-release');
});
test('the published tag matching HEAD is up to date', async () => {
const { work, origin, clone } = pair('published-same');
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
const s = await releaseState(clone, 'v1');
expect(s.updateable).toBe(false);
expect(s.reason).toBe('at-latest');
});
test('an untagged checkout is updateable AND says what it would land on', async () => {
const { clone } = pair('published-untagged');
const s = await releaseState(clone, 'v9');
expect(s.tagAtHead).toBeNull();
// The old code returned null here, so the button read "Update" with nothing after it.
expect(s.newestTag).toBe('v9');
expect(s.updateable).toBe(true);
expect(s.reason).toBe('untagged');
});
test('no published tag falls back to the git rule rather than to a guess', async () => {
const { work, origin, clone } = pair('no-entry');
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
// Null, as it is for a plugin nobody registered or a store that could not be reached.
const s = await releaseState(clone, null);
expect(s.tagAtHead).toBe('v1');
expect(s.newestTag).toBe('v1');
expect(s.updateable).toBe(false);
});
test('a directory that is not a checkout is not updateable', async () => { test('a directory that is not a checkout is not updateable', async () => {
const d = join(root, 'plain'); const d = join(root, 'plain');
mkdirSync(d, { recursive: true }); mkdirSync(d, { recursive: true });
+39 -65
View File
@@ -29,18 +29,32 @@ import { join } from 'node:path';
* is that tag, this checkout is at the latest release. If it is a different one, the remote has moved on. * is that tag, this checkout is at the latest release. If it is a different one, the remote has moved on.
* That holds for any tag names anyone ever chooses, including ones that do not sort. * That holds for any tag names anyone ever chooses, including ones that do not sort.
* *
* ── The answer is never awaited on the network ── * ── Where "the newest tag" comes from: the catalogue, not this machine's refs ──
* *
* Every answer comes from refs already on disk. This is called once per plugin on a request that renders * Every answer here is local, and this is called once per plugin on a request that renders a list —
* a list, and twenty-one network round trips would make the plugins screen feel broken. * twenty-one network round trips would make the plugins screen feel broken. That left one hard problem:
* refs on disk go stale. Tags arrived at clone and at the fetch `pullPlugin` does, so you learned an
* update existed BY UPDATING. A machine that had not touched a plugin since August compared this
* morning's release against a snapshot from before it existed and said, truthfully and uselessly, that
* it was at the latest.
* *
* But refs on disk go stale, and for a long time nothing refreshed them: tags arrived at clone and at * That was patched with a rate-limited background `git fetch` per checkout, which worked and was the
* the fetch `pullPlugin` does, so you only learned an update existed BY UPDATING. A machine that had not * wrong shape: every host in the fleet polling twenty-one repositories to re-derive a fact one place
* updated a plugin since August compared this morning's release against a snapshot from before it * already knows. The marketplace publishes releases now — see `servers/marketplace/sync.ts` — so the
* existed and said, truthfully and uselessly, that it was at the latest. * newest release is simply asked of the catalogue, which every host already fetches for the store.
* *
* So `refreshRemote` runs a fetch in the background and this request answers from what is there now. The * published tag (catalogue) vs tag at HEAD (this checkout) → differ, so there is an update
* NEXT read is current. Rate-limited per checkout and run one at a time — see the note on it. *
* DIFFERENCE, not ordering. `2026.08.27.01` parses fine until somebody tags `v2`, and then it fails
* silently. Two strings that are not equal is a comparison that cannot rot.
*
* ── The git rule stays, for a plugin the store cannot speak for ──
*
* A checkout of something unpublished — a plugin being written, one never registered — has no catalogue
* entry, and the fallback below is the old rule against local remote-tracking refs. Those refs are no
* longer refreshed here, so for that case this can miss a tag until something else fetches. That is the
* accepted cost: the alternative was every machine polling every repository forever to cover a case that
* only arises on a developer's own box, where `git fetch` is a thing they were about to run anyway.
*/ */
export type ReleaseState = { export type ReleaseState = {
@@ -53,65 +67,18 @@ export type ReleaseState = {
reason: 'untagged' | 'new-release' | 'at-latest' | 'not-a-checkout'; reason: 'untagged' | 'new-release' | 'at-latest' | 'not-a-checkout';
}; };
/**
* A background `git fetch --tags`, at most one per checkout per REFRESH_MS.
*
* ── Not awaited, on purpose ──
*
* The caller is rendering a list. Waiting would put the network on the critical path of a page that has
* to feel instant, which is the reason this file never fetched in the first place. The request answers
* from the refs it has; the fetch it starts is for the next one.
*
* The cost of that is one stale view after a release, corrected by a refresh. The alternative — never
* fetching — is a badge that is wrong until somebody updates for an unrelated reason.
*
* ── One at a time ──
*
* Twenty-one plugins would otherwise spawn twenty-one gits at once, every ten minutes, on machines that
* are not all fast. Chained rather than pooled because there is no hurry: nothing is waiting on it.
*
* Failure is silence. A checkout with no remote, an unreachable host, a repository that has moved — none
* of those should surface as anything, because the list still has a true answer without them.
*/
/*
Two minutes, and the number is chosen for how releases are actually watched.
It was ten. The way this gets used is: ship a plugin, open the page, look for the badge — and ten
minutes of that is indistinguishable from the feature not working. A release tagged at 12:42 was
invisible at 12:45 because the last fetch happened at 12:37, which is correct behaviour and no comfort.
The cost is bounded by the page being open: nothing fetches when nobody is looking, and the fetches are
chained, so the worst case is one small round trip per plugin every two minutes while somebody watches
the plugins screen.
*/
const REFRESH_MS = 2 * 60 * 1000;
const fetchedAt = new Map<string, number>();
let queue: Promise<unknown> = Promise.resolve();
export function refreshRemote(dir: string): void {
const last = fetchedAt.get(dir) ?? 0;
if (Date.now() - last < REFRESH_MS) return;
// Stamped BEFORE the fetch runs, so a slow one cannot be started again by the next twenty requests.
fetchedAt.set(dir, Date.now());
queue = queue.then(async () => {
const proc = Bun.spawn(['git', '-C', dir, 'fetch', '--tags', '--quiet', 'origin'], {
stdout: 'ignore',
stderr: 'ignore',
// A host that hangs must not hold the queue: every plugin behind it would stop refreshing too.
signal: AbortSignal.timeout(20_000),
});
await proc.exited.catch(() => {});
});
}
const run = async (dir: string, args: string[]): Promise<string | null> => { const run = async (dir: string, args: string[]): Promise<string | null> => {
const proc = Bun.spawn(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' }); const proc = Bun.spawn(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
const out = await new Response(proc.stdout).text(); const out = await new Response(proc.stdout).text();
return (await proc.exited) === 0 ? out.trim() || null : null; return (await proc.exited) === 0 ? out.trim() || null : null;
}; };
export async function releaseState(dir: string): Promise<ReleaseState> { /**
* @param publishedTag The tag the marketplace says is this plugin's newest release, or null when the
* catalogue has nothing to say — an unregistered plugin, one never tagged, or a store that could not
* be reached. Null falls through to the git rule rather than to a guess.
*/
export async function releaseState(dir: string, publishedTag: string | null = null): Promise<ReleaseState> {
const none = { tagAtHead: null, newestTag: null }; const none = { tagAtHead: null, newestTag: null };
if (!existsSync(join(dir, '.git'))) { if (!existsSync(join(dir, '.git'))) {
// Copied in by hand or bundled. Nothing to compare, and nothing to pull either — `pullPlugin` says // Copied in by hand or bundled. Nothing to compare, and nothing to pull either — `pullPlugin` says
@@ -119,12 +86,19 @@ export async function releaseState(dir: string): Promise<ReleaseState> {
return { ...none, updateable: false, reason: 'not-a-checkout' }; return { ...none, updateable: false, reason: 'not-a-checkout' };
} }
// Started, not awaited: this answer comes from the refs on disk and the fetch is for the next one.
refreshRemote(dir);
// `--exact-match`, so this is "the tag AT this commit", not "the last tag before it". Fails, and // `--exact-match`, so this is "the tag AT this commit", not "the last tag before it". Fails, and
// therefore returns null, when HEAD is between releases — which is the first half of the rule. // therefore returns null, when HEAD is between releases — which is the first half of the rule.
const tagAtHead = await run(dir, ['describe', '--exact-match', '--tags', 'HEAD']); const tagAtHead = await run(dir, ['describe', '--exact-match', '--tags', 'HEAD']);
if (publishedTag) {
// Reported even when HEAD is untagged, so the UI can say what pressing Update lands on. The old
// code returned a bare null here and the button read "Update" with nothing after it.
if (!tagAtHead) return { tagAtHead: null, newestTag: publishedTag, updateable: true, reason: 'untagged' };
return publishedTag === tagAtHead
? { tagAtHead, newestTag: publishedTag, updateable: false, reason: 'at-latest' }
: { tagAtHead, newestTag: publishedTag, updateable: true, reason: 'new-release' };
}
if (!tagAtHead) return { ...none, updateable: true, reason: 'untagged' }; if (!tagAtHead) return { ...none, updateable: true, reason: 'untagged' };
// Asked of the REMOTE branch, not of HEAD: a tag pushed after this checkout was made is not reachable // Asked of the REMOTE branch, not of HEAD: a tag pushed after this checkout was made is not reachable
+5 -1
View File
@@ -1,4 +1,5 @@
import { existsSync, readdirSync } from 'node:fs'; import { existsSync, readdirSync } from 'node:fs';
import { publishedTags } from './marketplace';
import { releaseState } from './release'; import { releaseState } from './release';
import { join } from 'node:path'; import { join } from 'node:path';
import { PLUGINS_DIR } from './discover'; import { PLUGINS_DIR } from './discover';
@@ -144,7 +145,10 @@ export async function updatePlugin(
Checked BEFORE the pull, because a pull is what would make the answer stale. Checked BEFORE the pull, because a pull is what would make the answer stale.
*/ */
const state = await releaseState(join(PLUGINS_DIR, appName)); // Against the CATALOGUE, like the badge. Without it this asked local remote-tracking refs, which are
// no longer refreshed in the background — so a plugin at the latest tag it knows about would have
// short-circuited here and told the owner it was up to date without checking anything.
const state = await releaseState(join(PLUGINS_DIR, appName), (await publishedTags()).get(appName) ?? null);
if (!state.updateable && state.reason === 'at-latest') { if (!state.updateable && state.reason === 'at-latest') {
steps.push(`already at ${state.tagAtHead}, the newest release`); steps.push(`already at ${state.tagAtHead}, the newest release`);
onStep?.(steps[steps.length - 1]!); onStep?.(steps[steps.length - 1]!);
@@ -534,9 +534,11 @@ export const PluginDetail = () => {
how you teach someone to stop pressing it. */} how you teach someone to stop pressing it. */}
{plugin.updateable && ( {plugin.updateable && (
<Button disabled={busy} onClick={() => run('update', plugin.appName)}> <Button disabled={busy} onClick={() => run('update', plugin.appName)}>
{plugin.releaseReason === 'new-release' && plugin.newestTag {/* Named whenever a release is known, not only on `new-release`. An untagged checkout
? `Update to ${plugin.newestTag}` reports `untagged`, and it used to lose the destination with it — the button said
: 'Update'} "Update" and the owner had to guess where it landed. The marketplace knows the newest
release even for a checkout that has never fetched it. */}
{plugin.newestTag ? `Update to ${plugin.newestTag}` : 'Update'}
</Button> </Button>
)} )}
{/* Said out loud rather than shown as an absent button. "No Update button" and "up to date" {/* Said out loud rather than shown as an absent button. "No Update button" and "up to date"