the marketplace publishes releases, not working trees

The sync read `platform/plugins/` — the directories this machine has installed — and imported each manifest off the working tree. So publishing meant "get officer-dev's checkout to the release, then run the script", and nothing showed when that had not happened. Live proof at the time of writing: av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26 were all in the store, and none of the four has ever been tagged. It is also why the job could not go on a timer. A cron over it published whatever was in a working tree, on a schedule. Now every registered plugin is mirrored bare into ~/.cache/officer-marketplace/mirrors and read AT ITS NEWEST TAG — manifest, README and file list all out of `git cat-file`. No working tree in the path, so publishing something uncommitted stops being a mistake anyone can make. docs/marketplace.md proposed reading manifests over Gitea's API, which needed a sandbox or a committed manifest.json. Git gives the same guarantee with neither, and the trust model is unchanged: we import our own repositories' code. Third-party submissions still need one of those two. The list of plugins now comes from the `plugins` registry table, which the docs always called the authoritative half and which was in fact a by-product of what happened to be installed. That is what lets `finances` be published without being installed here, and it spans orgs — fofos-admin lives under fofos/, which an org listing would have missed. Adding one is `--add <clone url>`. A plugin with no tag is left alone and named in the report, not deleted. It has made no release, so there is nothing true to say about it; dropping its row would be a decision about somebody's plugin dressed up as a refresh. Also: `release_tag` on plugin_metadata, because the tag is the fact when a manifest version disagrees with it; the sync warns on a mismatch. And icon URLs pin to the tag — `/raw/HEAD/` served whatever was on the default branch, via a 303, so a plugin could show one icon and ship another. Verified: 21/25 published from a release, 4 named as untagged, all 14 tag-pinned icons answer 200. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 13:05:50 +01:00
parent 8d39586de5
commit 78e89ebcec
4 changed files with 385 additions and 164 deletions
@@ -0,0 +1,287 @@
+import { existsSync, mkdirSync } from 'node:fs';
+import { homedir, tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+  listRegistry,
+  upsertMetadata,
+  upsertPlugin,
+} from '../../databases/marketplace_db/src/plugins/queries';
+
+/**
+ * Fill the catalogue from what each plugin has RELEASED.
+ *
+ * ── The bug this file exists to remove ──
+ *
+ * Until 2026-09-03 the sync read `platform/plugins/` — the directories this machine has INSTALLED — and
+ * imported each `manifest.ts` off the working tree. So "publish" meant "make sure officer-dev's checkout
+ * happens to be at the release, then run the script", and nobody could see when it was not. On the day
+ * this was rewritten the live catalogue was advertising four plugins from untagged working copies:
+ * av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of
+ * which had ever been tagged. The store was describing somebody's afternoon.
+ *
+ * That is also why the old script could not be put on a timer: a cron over it would have published
+ * whatever was sitting in a working tree, on a schedule.
+ *
+ * ── What replaces it: a bare mirror, read at a tag ──
+ *
+ * Every plugin in the registry is mirrored (`git clone --mirror`) into a cache directory, refreshed, and
+ * read AT ITS NEWEST TAG — the manifest, the README, the file list, all out of `git cat-file`. There is
+ * no working tree anywhere in this path, so publishing something uncommitted is not a mistake that can
+ * be made. A mirror is cheap: the whole set clones in a couple of seconds and refreshes in well under
+ * one.
+ *
+ * `docs/marketplace.md` proposed instead reading manifests over Gitea's API, which needed either a
+ * sandbox to evaluate a fetched string or a committed `manifest.json` for every plugin. Git gives the
+ * same guarantee — content at a named release, independent of any checkout — with neither, and the trust
+ * model is unchanged from today: we import our own repositories' code.
+ *
+ * `[still a blocker for third-party submissions]` It is unchanged BECAUSE it is unchanged. Importing a
+ * stranger's manifest is code execution no matter which ref it came from. Submissions still need a
+ * sandbox or a published `manifest.json`.
+ *
+ * ── Which plugins, and where from: the registry, and only the registry ──
+ *
+ * The list comes from the `plugins` table, which `docs/marketplace.md` has always called the
+ * authoritative, hand-curated half. It was not actually being used that way — the old sync discovered
+ * plugins by listing a directory, so the registry was a by-product of whatever officer-dev had
+ * installed. Now it decides, which means:
+ *
+ *   · a plugin nobody has installed here can still be published (`finances` is in exactly that state),
+ *   · a plugin lives wherever its registry row says, across orgs — `fofos-admin` is under `fofos/`,
+ *     not `plugins/`, so an org listing would have missed it,
+ *   · and adding one is a deliberate act: `marketplace-sync.ts --add <https clone url>`.
+ *
+ * ── A plugin with no tag is left ALONE, not removed ──
+ *
+ * It has never made a release, so this file has nothing true to say about it and says nothing: the
+ * existing metadata row stays, and the run reports it. Deleting the row would drop it out of the store
+ * on the day the rule changed, which is a decision about somebody's plugin dressed up as a refresh.
+ * Tagging it is the fix, and the report is there to ask for one.
+ */
+
+export type SyncOutcome =
+  | { appName: string; status: 'synced'; tag: string; version: string }
+  | { appName: string; status: 'unreleased' }
+  | { appName: string; status: 'failed'; error: string };
+
+/*
+  Where the mirrors live.
+
+  A fixed path under the user's cache directory, NOT one derived from the environment. This runs three
+  ways — by hand with `--env-file=.env`, from the webhook inside `officer-marketplace`, and from cron —
+  and those three do not have the same environment. A path that moved with `DATA_PATH` would have each
+  of them re-cloning twenty-five repositories into its own copy, and the "is it already mirrored" check
+  would answer differently depending on who asked.
+
+  Not in the source tree, because it is a cache of remote repositories and a `.gitignore` entry is a
+  worse way to keep several hundred megabytes out of a repo than not putting them there.
+*/
+const MIRROR_DIR =
+  process.env.MARKETPLACE_MIRROR_DIR ?? join(homedir(), '.cache', 'officer-marketplace', 'mirrors');
+
+const GIT_TIMEOUT_MS = 60_000;
+
+type Git = { ok: true; out: string } | { ok: false; err: string };
+
+async function git(args: string[], cwd?: string): Promise<Git> {
+  // A missing `cwd` makes posix_spawn fail with ENOENT, which reads as "git is not installed" and sent
+  // a first run of this looking for git for ten minutes. It means the mirror is not there yet.
+  if (cwd && !existsSync(cwd)) return { ok: false, err: `${cwd} does not exist` };
+
+  const proc = Bun.spawn(['git', ...args], {
+    cwd,
+    stdout: 'pipe',
+    stderr: 'pipe',
+    // Never inherited: a mirror fetch that hits a host asking for a password would otherwise sit on a
+    // terminal prompt forever, and in the webhook server there is no terminal to sit on.
+    env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GIT_ASKPASS: '/bin/true' },
+    signal: AbortSignal.timeout(GIT_TIMEOUT_MS),
+  });
+  const [out, err] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]);
+  return (await proc.exited) === 0 ? { ok: true, out: out.trim() } : { ok: false, err: err.trim() || 'git failed' };
+}
+
+/** The mirror for a plugin, cloned if this is the first time and refreshed if it is not. */
+async function mirror(appName: string, repoUrl: string): Promise<string> {
+  mkdirSync(MIRROR_DIR, { recursive: true });
+  const dir = join(MIRROR_DIR, `${appName}.git`);
+
+  // Re-cloned rather than repaired when it is not a healthy bare repo: an interrupted clone leaves a
+  // directory that every later command fails against, and there is nothing in it worth keeping.
+  const existing = await git(['rev-parse', '--is-bare-repository'], dir);
+  if (!existing.ok || existing.out !== 'true') {
+    if (existsSync(dir)) await Bun.$`rm -rf ${dir}`.quiet().nothrow();
+    const cloned = await git(['clone', '--mirror', '--quiet', repoUrl, dir]);
+    if (!cloned.ok) throw new Error(`clone failed: ${cloned.err}`);
+    return dir;
+  }
+
+  // The registry is allowed to move a plugin. Setting the URL every time is one command and removes the
+  // failure where a repo is relocated and the mirror keeps refreshing from where it used to be.
+  await git(['remote', 'set-url', 'origin', repoUrl], dir);
+
+  // `--prune --prune-tags`: a deleted or MOVED tag has to disappear here too. Without it, a tag that was
+  // pushed by mistake and deleted would stay in the mirror forever and could go on being published as
+  // the newest release — a store advertising a version its repository no longer has.
+  const fetched = await git(['fetch', '--quiet', '--prune', '--prune-tags', '--tags', 'origin'], dir);
+  if (!fetched.ok) throw new Error(`fetch failed: ${fetched.err}`);
+  return dir;
+}
+
+/**
+ * The newest release, or null for a plugin that has never made one.
+ *
+ * Asked of git rather than by sorting version strings, which is the same rule `servers/plugins/release.ts`
+ * follows and for the same reason: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
+ * then fails silently. `--abbrev=0` gives the closest tag reachable from HEAD, and a mirror's HEAD is the
+ * remote's default branch, so this is "the newest release on the branch the repository points at".
+ */
+async function newestTag(dir: string): Promise<string | null> {
+  const described = await git(['describe', '--tags', '--abbrev=0', 'HEAD'], dir);
+  return described.ok ? described.out : null;
+}
+
+/** A file at a ref, or null when it is not in that tree. */
+async function fileAt(dir: string, ref: string, path: string): Promise<string | null> {
+  const shown = await git(['cat-file', '-p', `${ref}:${path}`], dir);
+  return shown.ok ? shown.out : null;
+}
+
+/**
+ * The manifest at a release, as an object.
+ *
+ * Written to a temp file and imported, because a manifest is TypeScript. Keyed by the COMMIT, so the
+ * module cache is correct rather than merely convenient: the same commit is the same file, and a tag
+ * moved to a different commit gets a different path and a fresh import. That matters in the webhook
+ * server, which is long-lived and would otherwise serve the first version of a manifest it ever read.
+ *
+ * The `import type { PluginManifest }` at the top of every manifest is erased by the transpiler, so the
+ * file has no runtime dependency on the platform's path aliases and imports fine from anywhere.
+ */
+async function manifestAt(dir: string, ref: string, appName: string, commit: string): Promise<Record<string, unknown>> {
+  const source = await fileAt(dir, ref, 'manifest.ts');
+  if (source === null) throw new Error(`no manifest.ts at ${ref}`);
+
+  const scratch = join(tmpdir(), 'officer-marketplace-manifests');
+  mkdirSync(scratch, { recursive: true });
+  const path = join(scratch, `${appName}-${commit}.ts`);
+  await Bun.write(path, source);
+
+  const mod = (await import(path)) as { manifest?: Record<string, unknown> };
+  if (!mod.manifest) throw new Error(`manifest.ts at ${ref} exports no \`manifest\``);
+  return mod.manifest;
+}
+
+/**
+ * A raw-file URL for a Gitea repository, PINNED TO THE RELEASE.
+ *
+ * It used to be `/raw/HEAD/…`, which works only because Gitea 303s it to `/raw/branch/HEAD/…`; the
+ * artwork it served was whatever was on the default branch, so a plugin could show one icon in the store
+ * and ship another. `/raw/tag/<tag>/…` answers 200 directly and shows the icon that belongs to the
+ * version being advertised.
+ *
+ * `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
+ * be part of the registry rather than inferred from the URL.
+ */
+function rawUrl(repoUrl: string, tag: string, path: string): string {
+  return `${repoUrl.replace(/\.git$/, '')}/raw/tag/${encodeURIComponent(tag)}/${path}`;
+}
+
+async function syncRegistered(row: { id: number; appName: string; repoUrl: string }): Promise<SyncOutcome> {
+  const { appName, repoUrl } = row;
+  const dir = await mirror(appName, repoUrl);
+
+  const tag = await newestTag(dir);
+  if (!tag) return { appName, status: 'unreleased' };
+
+  const commitAt = await git(['rev-list', '-n', '1', tag], dir);
+  if (!commitAt.ok) throw new Error(`cannot resolve ${tag}`);
+  const commit = commitAt.out;
+
+  const manifest = await manifestAt(dir, tag, appName, commit);
+
+  // What the plugin SHIPS, read from the tree at the release rather than from a declaration — the same
+  // rule discovery follows on disk, so a catalogue entry cannot claim a sidecar the release does not
+  // have.
+  const listed = await git(['ls-tree', '-r', '--name-only', tag], dir);
+  const files = new Set(listed.ok ? listed.out.split('\n') : []);
+  const has = {
+    api: files.has('api/router.ts'),
+    schema: files.has('db/schema.ts'),
+    sidecar: files.has('sidecar/index.ts') || files.has('sidecar/index.mjs'),
+    web: [...files].some((f) => f.startsWith('web/')),
+  };
+
+  const readme = (await fileAt(dir, tag, 'README.md')) ?? (await fileAt(dir, tag, 'readme.md'));
+
+  const version = String(manifest.version ?? tag);
+  if (version !== tag) {
+    // Not an error and not corrected here. The tag is what was released and the manifest is what the
+    // plugin will report about itself once installed; when they disagree, an update badge that compares
+    // versions goes quiet. Saying so is the only way anyone finds out.
+    console.warn(`[marketplace-sync] ${appName}: tag ${tag} but manifest says ${version} — they should match`);
+  }
+
+  const developer = String(manifest.publisher ?? 'unknown');
+  const pluginId = await upsertPlugin({ appName, developer, repoUrl });
+
+  await upsertMetadata(pluginId, {
+    version,
+    releaseTag: tag,
+    platform: String(manifest.platform ?? '>=1.0.0'),
+    label: String(manifest.label ?? appName),
+    summary: String(manifest.summary ?? ''),
+    // Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
+    // vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
+    // the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
+    // quietly disagreeing with the repository it claims to mirror.
+    tags: Array.isArray(manifest.tags) ? manifest.tags : [],
+    color: (manifest.color as string) ?? null,
+    tile: (manifest.tile as string) ?? null,
+    iconUrl: files.has('assets/icon.png') ? rawUrl(repoUrl, tag, 'assets/icon.png') : null,
+    icon: (manifest.icon as string) ?? null,
+    permissions: manifest.permissions ?? [],
+    osDependencies: manifest.osDependencies ?? null,
+    has,
+    readme,
+    sourceCommit: commit,
+  });
+
+  return { appName, status: 'synced', tag, version };
+}
+
+/** One plugin by app name. Unknown to the registry is a failure, not a silent no-op. */
+export async function syncPlugin(appName: string): Promise<SyncOutcome> {
+  const row = (await listRegistry()).find((r) => r.appName === appName);
+  if (!row) return { appName, status: 'failed', error: 'not in the registry' };
+  return syncRegistered(row).catch((err) => ({
+    appName,
+    status: 'failed' as const,
+    error: err instanceof Error ? err.message : String(err),
+  }));
+}
+
+/**
+ * Every registered plugin, one at a time.
+ *
+ * Sequential on purpose. Twenty-four concurrent fetches against one Gitea is a thundering herd for a job
+ * nothing is waiting on, and the whole set finishes in about a second anyway.
+ *
+ * Unlisted rows are included: `listed` hides a plugin from the public API, usually after a bad release,
+ * and it has to be able to come back with current metadata rather than with whatever was true when it
+ * was pulled.
+ */
+export async function syncAll(): Promise<SyncOutcome[]> {
+  const rows = await listRegistry();
+  const out: SyncOutcome[] = [];
+  for (const row of rows) {
+    out.push(
+      await syncRegistered(row).catch((err) => ({
+        appName: row.appName,
+        status: 'failed' as const,
+        error: err instanceof Error ? err.message : String(err),
+      })),
+    );
+  }
+  return out;
+}