marketplace.officer.dev: the catalogue becomes a service

The plugins page read a TypeScript constant. It now reads a public API backed by its own database, and falls back to that constant when it cannot. ── Four pieces ── `src/databases/marketplace_db` — a second Postgres database, `officer_marketplace`. Two tables, split by owner and lifetime: `plugins` is the registry (app name, developer, clone URL) and is authoritative; `plugin_metadata` is a CACHE of what each repository says about itself, so the website can render without cloning three repos per page load. Keeping them apart is what makes the cache safe to be wrong — a refresh can never lose the fact that a plugin exists. `scripts/marketplace-sync.ts` — fills it from the plugin repositories. Reads each `manifest.ts` (imported, since a manifest is TypeScript), derives `has` from the directory rather than trusting a declaration, and rewrites SSH remotes to HTTPS: `example` and `offscale` had `git@gitea…` origins, which would have handed every installer a URL that only works on the machine that published it. `src/servers/marketplace/server.tsx` — its own process, its own port, two GET routes and no others. Not part of the platform server: it has no users, no sessions and no request it answers differently per caller, and mounting a public unauthenticated surface inside the process holding mail, a wallet and passkeys means writing permission-gate exemptions that later turn out to be too wide. `src/apps/marketplace` — the public site. A window, deliberately: nothing here installs anything, because installing means cloning a repo and starting a process on someone else's machine. Every plugin page ends at "install it from your own Officer". ── The platform side is the part that had to be careful ── `fetchCatalogue()` now answers live → cached → bundled, in that order, and never throws. The plugins page is where an owner manages plugins they ALREADY have, so the store being down must not take that page with it. Network responses are validated per entry — not against a hostile marketplace, which has already won if the owner installs what it lists, but against a wrong one: a proxy serving HTML, a half-deployed API, a field renamed on one side. A non-https `source` is refused outright because that string reaches `git clone`. 8 tests on the degradation path. Verified end to end against the running service: live catalogue, cached answer surviving the store dying, bundled fallback. ── One thing found on the way ── `bunfig.toml` and CLAUDE.md both document `bun install --no-frozen-lockfile` as the way to change a dependency. Bun 1.3.14 has no such flag — only `--frozen-lockfile` — so the documented escape hatch does not work. Sidestepped by not making marketplace_db a workspace member (relative imports, zero lockfile change), and `bun install` is frozen-clean again. Worth fixing the docs. tsgo clean, 873 pass / 7 fail unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 04:07:49 +00:00
parent 8c66042e42
commit 921fbff9fe
15 changed files with 1202 additions and 10 deletions
@@ -0,0 +1,98 @@
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import { clearCatalogueCache, fetchCatalogue } from './marketplace';
+
+// The three answers — live, cached, bundled — and the rule that none of them is an exception.
+//
+// The plugins page is blocked on this call, and an owner's INSTALLED plugins are managed on that same
+// page. So a marketplace that is down must degrade to something true rather than throw: the failure
+// mode being defended against is "officer.dev has a bad afternoon and nobody can uninstall anything".
+
+const realFetch = globalThis.fetch;
+
+const entry = (appName: string) => ({
+  appName,
+  publisher: 'officerdev',
+  version: '1.0.0',
+  platform: '>=1.0.0',
+  label: appName,
+  summary: 'a plugin',
+  color: '#fff',
+  source: `https://gitea.officer.dev/plugins/${appName}.git`,
+  permissions: [],
+  has: { api: true, schema: false, sidecar: false, web: true },
+});
+
+const respondWith = (body: unknown, status = 200) => {
+  globalThis.fetch = (async () =>
+    new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } })) as typeof fetch;
+};
+
+const failWith = (message: string) => {
+  globalThis.fetch = (async () => {
+    throw new Error(message);
+  }) as typeof fetch;
+};
+
+beforeEach(() => clearCatalogueCache());
+afterEach(() => {
+  globalThis.fetch = realFetch;
+  clearCatalogueCache();
+});
+
+describe('fetchCatalogue', () => {
+  test('serves what the marketplace returns', async () => {
+    respondWith([entry('music'), entry('offscale')]);
+    expect((await fetchCatalogue()).map((e) => e.appName)).toEqual(['music', 'offscale']);
+  });
+
+  test('falls back to the bundled catalogue when the store is unreachable', async () => {
+    failWith('getaddrinfo ENOTFOUND');
+    const entries = await fetchCatalogue();
+    // Not asserting the exact list — that changes as plugins are published. The property is that a
+    // machine with no network still has a usable store.
+    expect(entries.length).toBeGreaterThan(0);
+    expect(entries.every((e) => e.source.startsWith('https://'))).toBe(true);
+  });
+
+  test('a cached answer outlives the store going down', async () => {
+    respondWith([entry('music')]);
+    await fetchCatalogue();
+
+    // The cache is deliberately NOT cleared here — surviving the store going down is the whole point.
+    failWith('connection refused');
+    const entries = await fetchCatalogue();
+    expect(entries.map((e) => e.appName)).toEqual(['music']);
+  });
+
+  test('an HTTP error is a failure, not a catalogue', async () => {
+    respondWith({ error: 'catalogue unavailable' }, 503);
+    const entries = await fetchCatalogue();
+    expect(entries.some((e) => e.appName === 'music')).toBe(true); // the bundled list, not the error body
+  });
+
+  test('a proxy serving HTML instead of JSON does not empty the store', async () => {
+    globalThis.fetch = (async () => new Response('<html>502 Bad Gateway</html>', { status: 200 })) as typeof fetch;
+    const entries = await fetchCatalogue();
+    expect(entries.length).toBeGreaterThan(0);
+  });
+
+  test('malformed entries are dropped individually, not fatally', async () => {
+    respondWith([entry('music'), { appName: 'broken' }, entry('example')]);
+    expect((await fetchCatalogue()).map((e) => e.appName)).toEqual(['music', 'example']);
+  });
+
+  test('a non-https source is refused — it drives git clone', async () => {
+    respondWith([{ ...entry('evil'), source: 'file:///etc/passwd' }, entry('music')]);
+    expect((await fetchCatalogue()).map((e) => e.appName)).toEqual(['music']);
+  });
+
+  test('an empty catalogue is a valid answer and is cached as one', async () => {
+    respondWith([]);
+    expect(await fetchCatalogue()).toEqual([]);
+
+    // Cached: a store that legitimately lists nothing must not fall through to the bundled list, or an
+    // owner who unpublished everything would keep seeing plugins that are gone.
+    failWith('down');
+    expect(await fetchCatalogue()).toEqual([]);
+  });
+});
@@ -75,13 +75,20 @@ export type CatalogueEntry = {
 };

 /**
- * The stand-in catalogue.
+ * What this machine knows without asking anyone.
 *
- * One entry, and it is not a fixture: `plugins/music` is a real public repository that a real
- * `git clone` reaches with no credentials. The install path is therefore exercised end to end even
- * though the catalogue itself is a constant.
+ * This WAS the catalogue until marketplace.officer.dev existed. It is now the floor under it: the
+ * answer when the store cannot be reached and nothing has been cached yet.
+ *
+ * Kept rather than deleted, and kept CURRENT, because the alternative is a first-boot machine with no
+ * internet showing an empty store — which is indistinguishable from a store with nothing in it. Three
+ * entries is a small price for "the plugins page always has something true on it".
+ *
+ * It will drift from the real catalogue, and that is acceptable in exactly one direction: it may be
+ * missing something new. It must never describe a plugin that no longer exists, because that produces
+ * an install button that fails at clone time.
 */
-const CATALOGUE: CatalogueEntry[] = [
+const BUNDLED: CatalogueEntry[] = [
  {
    appName: 'music',
    publisher: 'officerdev',
@@ -156,19 +163,108 @@ const CATALOGUE: CatalogueEntry[] = [
  },
 ];

+// ── Reaching the real one ──
+
+/** Where the catalogue lives. Overridable so a fork, an air-gapped install or a test can point elsewhere. */
+const MARKETPLACE_URL = process.env.MARKETPLACE_URL ?? 'https://marketplace.officer.dev';
+
+/**
+ * How long a good answer is reused before asking again.
+ *
+ * Long enough that opening the plugins page repeatedly does not hammer the store, short enough that a
+ * plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a
+ * restart is always a fresh look.
+ */
+const CACHE_TTL_MS = 5 * 60 * 1000;
+
+/**
+ * How long to wait for the store before giving up on it.
+ *
+ * The plugins page is blocked on this call, so the ceiling is "how long may that page take to render
+ * when marketplace.officer.dev is unreachable". Four seconds is already at the edge of what a page can
+ * spend before it feels broken — and a DNS black hole would otherwise hang it for the OS timeout.
+ */
+const FETCH_TIMEOUT_MS = 4000;
+
+let cache: { entries: CatalogueEntry[]; at: number } | null = null;
+
+/**
+ * A catalogue entry drives `git clone`, so nothing arrives from the network untyped.
+ *
+ * This is not defence against a hostile marketplace — a store that serves a malicious `source` has
+ * already won, since the owner asked to install what it lists. It is defence against a WRONG one: a
+ * half-deployed API, a proxy serving an error page as JSON, a field renamed on one side. Every one of
+ * those otherwise reaches the install path as `undefined` and fails somewhere far less legible.
+ *
+ * Entries that fail are dropped individually rather than failing the whole response: one malformed
+ * plugin should not empty the store.
+ */
+function isEntry(value: unknown): value is CatalogueEntry {
+  if (!value || typeof value !== 'object') return false;
+  const e = value as Record<string, unknown>;
+  return (
+    typeof e.appName === 'string' &&
+    e.appName.length > 0 &&
+    typeof e.source === 'string' &&
+    e.source.startsWith('https://') &&
+    typeof e.label === 'string' &&
+    typeof e.summary === 'string' &&
+    Array.isArray(e.permissions) &&
+    typeof e.has === 'object' &&
+    e.has !== null
+  );
+}
+
 /**
 * Everything the marketplace offers.
 *
- * Async and allowed to fail from the start, because the real one will be a network call. A catalogue
- * that cannot be reached must leave the plugins page working with what is on disk — an owner who cannot
- * see the store should still be able to manage what they have already installed.
+ * Three answers in preference order — live, cached, bundled — and the page cannot tell which it got.
+ * That is deliberate: the plugins page's job is to show what can be installed, and it can always show
+ * something true. The alternative, an error state on the store being unreachable, makes an owner's own
+ * installed plugins unmanageable because a website is down.
+ *
+ * Never throws. `catalogueEntry` below is on the install path, and an install that fails because the
+ * catalogue lookup threw would be reported as "that plugin does not exist".
 */
 export async function fetchCatalogue(): Promise<CatalogueEntry[]> {
-  return CATALOGUE;
+  if (cache && Date.now() - cache.at < CACHE_TTL_MS) return cache.entries;
+
+  try {
+    const res = await fetch(`${MARKETPLACE_URL}/api/plugins`, {
+      signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
+      headers: { accept: 'application/json' },
+    });
+    if (!res.ok) throw new Error(`marketplace responded ${res.status}`);
+
+    const body: unknown = await res.json();
+    if (!Array.isArray(body)) throw new Error('marketplace returned a non-array');
+
+    const entries = body.filter(isEntry);
+    const dropped = body.length - entries.length;
+    if (dropped > 0) console.warn(`[marketplace] dropped ${dropped} malformed entr(y/ies)`);
+
+    // An empty valid response is cached like any other: a store may legitimately list nothing, and
+    // treating empty as failure would pin a stale cache forever the first time it happened.
+    cache = { entries, at: Date.now() };
+    return entries;
+  } catch (err) {
+    const reason = err instanceof Error ? err.message : String(err);
+    if (cache) {
+      console.warn(`[marketplace] unreachable (${reason}) — serving the cached catalogue`);
+      return cache.entries;
+    }
+    console.warn(`[marketplace] unreachable (${reason}) — serving the bundled catalogue`);
+    return BUNDLED;
+  }
 }

 /** One entry by app name, or null. Used by install to find where to clone from. */
 export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
-  const all = await fetchCatalogue().catch(() => [] as CatalogueEntry[]);
+  const all = await fetchCatalogue().catch(() => BUNDLED);
  return all.find((e) => e.appName === appName) ?? null;
 }
+
+/** Testing seam. The cache is process-lifetime, so a test that does not clear it leaks into the next. */
+export function clearCatalogueCache(): void {
+  cache = null;
+}