marketplace.officer.dev: the catalogue becomes a service
The plugins page read a TypeScript constant. It now reads a public API backed by its own database, and falls back to that constant when it cannot. ── Four pieces ── `src/databases/marketplace_db` — a second Postgres database, `officer_marketplace`. Two tables, split by owner and lifetime: `plugins` is the registry (app name, developer, clone URL) and is authoritative; `plugin_metadata` is a CACHE of what each repository says about itself, so the website can render without cloning three repos per page load. Keeping them apart is what makes the cache safe to be wrong — a refresh can never lose the fact that a plugin exists. `scripts/marketplace-sync.ts` — fills it from the plugin repositories. Reads each `manifest.ts` (imported, since a manifest is TypeScript), derives `has` from the directory rather than trusting a declaration, and rewrites SSH remotes to HTTPS: `example` and `offscale` had `git@gitea…` origins, which would have handed every installer a URL that only works on the machine that published it. `src/servers/marketplace/server.tsx` — its own process, its own port, two GET routes and no others. Not part of the platform server: it has no users, no sessions and no request it answers differently per caller, and mounting a public unauthenticated surface inside the process holding mail, a wallet and passkeys means writing permission-gate exemptions that later turn out to be too wide. `src/apps/marketplace` — the public site. A window, deliberately: nothing here installs anything, because installing means cloning a repo and starting a process on someone else's machine. Every plugin page ends at "install it from your own Officer". ── The platform side is the part that had to be careful ── `fetchCatalogue()` now answers live → cached → bundled, in that order, and never throws. The plugins page is where an owner manages plugins they ALREADY have, so the store being down must not take that page with it. Network responses are validated per entry — not against a hostile marketplace, which has already won if the owner installs what it lists, but against a wrong one: a proxy serving HTML, a half-deployed API, a field renamed on one side. A non-https `source` is refused outright because that string reaches `git clone`. 8 tests on the degradation path. Verified end to end against the running service: live catalogue, cached answer surviving the store dying, bundled fallback. ── One thing found on the way ── `bunfig.toml` and CLAUDE.md both document `bun install --no-frozen-lockfile` as the way to change a dependency. Bun 1.3.14 has no such flag — only `--frozen-lockfile` — so the documented escape hatch does not work. Sidestepped by not making marketplace_db a workspace member (relative imports, zero lockfile change), and `bun install` is frozen-clean again. Worth fixing the docs. tsgo clean, 873 pass / 7 fail unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,35 @@
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
import { defineConfig } from 'drizzle-kit';
|
||||
|
||||
// Same .env loading as officer_db's config: drizzle-kit runs as its own process from this directory and
|
||||
// inherits nothing from the server.
|
||||
const envPath = resolve(__dirname, '../../../.env');
|
||||
try {
|
||||
const text = readFileSync(envPath, 'utf-8');
|
||||
for (const line of text.split('\n')) {
|
||||
const match = line.match(/^(\w+)=(.*)$/);
|
||||
if (match) {
|
||||
const [, key, val] = match;
|
||||
if (!process.env[key!]) process.env[key!] = val!.replace(/^["']|["']$/g, '');
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
|
||||
// Derived exactly the way the runtime derives it — see src/db.ts. Duplicated rather than imported
|
||||
// because drizzle-kit loads this file outside the workspace's module resolution.
|
||||
const url = (() => {
|
||||
if (process.env.MARKETPLACE_POSTGRES_URL) return process.env.MARKETPLACE_POSTGRES_URL;
|
||||
const base = process.env.POSTGRES_URL;
|
||||
if (!base) throw new Error('MARKETPLACE_POSTGRES_URL or POSTGRES_URL is required');
|
||||
const parsed = new URL(base);
|
||||
parsed.pathname = '/officer_marketplace';
|
||||
return parsed.toString();
|
||||
})();
|
||||
|
||||
export default defineConfig({
|
||||
schema: './src/schema.ts',
|
||||
out: './migrations',
|
||||
dialect: 'postgresql',
|
||||
dbCredentials: { url },
|
||||
});
|
||||
@@ -0,0 +1,45 @@
|
||||
import { drizzle } from 'drizzle-orm/postgres-js';
|
||||
import postgres from 'postgres';
|
||||
import * as schema from './schema';
|
||||
|
||||
/**
|
||||
* The marketplace's own database.
|
||||
*
|
||||
* ── Why a second database rather than more tables in `officer` ──
|
||||
*
|
||||
* They answer to different people. `officer` is one owner's machine: their accounts, their dashboards,
|
||||
* what THEY have installed. The marketplace is a public catalogue that is the same for everyone and is
|
||||
* served by a process with no platform login, no session and no user. A plugin being listed has nothing
|
||||
* to do with any particular install, and `plugin_installs` in the platform's database has nothing to do
|
||||
* with the store.
|
||||
*
|
||||
* The practical half: this database is the one thing here that can be dropped and rebuilt from the
|
||||
* repositories it indexes, and a public HTTP surface reads it. Keeping that behind its own connection
|
||||
* means the catalogue API cannot reach a single row of anybody's mail, wallet or passkeys — a property
|
||||
* that is worth more than the convenience of one connection string.
|
||||
*
|
||||
* ── Where the URL comes from ──
|
||||
*
|
||||
* `MARKETPLACE_POSTGRES_URL` when set. Otherwise the platform's `POSTGRES_URL` with the database name
|
||||
* swapped for `officer_marketplace`, because the two live on the same server in every deployment that
|
||||
* exists — and a derived default is one fewer thing for `officer-setup.sh` to write correctly. Setting
|
||||
* the variable explicitly is what a split across servers would need, and it costs nothing to support.
|
||||
*/
|
||||
const MARKETPLACE_DB_NAME = 'officer_marketplace';
|
||||
|
||||
export function marketplaceUrl(): string {
|
||||
const explicit = process.env.MARKETPLACE_POSTGRES_URL;
|
||||
if (explicit) return explicit;
|
||||
|
||||
const base = process.env.POSTGRES_URL;
|
||||
if (!base) throw new Error('MARKETPLACE_POSTGRES_URL or POSTGRES_URL is required');
|
||||
|
||||
// Swap only the path. Parsing rather than string-replacing, so a password containing the database
|
||||
// name — or a name that appears in the host — cannot be rewritten by accident.
|
||||
const url = new URL(base);
|
||||
url.pathname = `/${MARKETPLACE_DB_NAME}`;
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
const client = postgres(marketplaceUrl());
|
||||
export const marketplaceDb = drizzle(client, { schema });
|
||||
@@ -0,0 +1,3 @@
|
||||
export { marketplaceDb } from './db';
|
||||
export * as schema from './schema';
|
||||
export * from './plugins/queries';
|
||||
@@ -0,0 +1,134 @@
|
||||
import { eq } from 'drizzle-orm';
|
||||
import { marketplaceDb } from '../db';
|
||||
import { pluginMetadata, plugins } from './schema';
|
||||
|
||||
/**
|
||||
* One catalogue entry, in the shape the platform already consumes.
|
||||
*
|
||||
* This is `CatalogueEntry` from `servers/plugins/marketplace.ts`, and the duplication is deliberate: it
|
||||
* is the WIRE FORMAT, and the two sides are different programs that will be deployed at different times.
|
||||
* Importing the platform's type here would make a change to it a silent break in a running website, and
|
||||
* a marketplace that must be redeployed in lockstep with every install is not a marketplace.
|
||||
*/
|
||||
export type CatalogueEntry = {
|
||||
appName: string;
|
||||
publisher: string;
|
||||
version: string;
|
||||
platform: string;
|
||||
label: string;
|
||||
summary: string;
|
||||
color: string;
|
||||
tile?: 'badge' | 'bare';
|
||||
iconUrl?: string;
|
||||
icon?: string;
|
||||
source: string;
|
||||
permissions: unknown[];
|
||||
osDependencies?: unknown[];
|
||||
has: { api: boolean; schema: boolean; sidecar: boolean; web: boolean };
|
||||
/** Only on the single-plugin endpoint. A list of full READMEs is a page nobody asked for. */
|
||||
readme?: string;
|
||||
};
|
||||
|
||||
type Row = {
|
||||
appName: string;
|
||||
developer: string;
|
||||
repoUrl: string;
|
||||
metadata: typeof pluginMetadata.$inferSelect | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* A registry row plus its cached metadata, as the API returns it.
|
||||
*
|
||||
* Returns null when there is no metadata yet — a registered plugin that has never been synced is real,
|
||||
* but there is nothing to render, and a half-filled tile is worse than an absent one. `listCatalogue`
|
||||
* drops those; the count of dropped rows is what a sync failure looks like from outside.
|
||||
*/
|
||||
function toCatalogueEntry(row: Row): CatalogueEntry | null {
|
||||
const m = row.metadata;
|
||||
if (!m) return null;
|
||||
return {
|
||||
appName: row.appName,
|
||||
publisher: row.developer,
|
||||
version: m.version,
|
||||
platform: m.platform,
|
||||
label: m.label,
|
||||
summary: m.summary,
|
||||
color: m.color ?? '#94a3b8',
|
||||
...(m.tile ? { tile: m.tile as 'badge' | 'bare' } : {}),
|
||||
...(m.iconUrl ? { iconUrl: m.iconUrl } : {}),
|
||||
...(m.icon ? { icon: m.icon } : {}),
|
||||
source: row.repoUrl,
|
||||
permissions: (m.permissions as unknown[]) ?? [],
|
||||
...(m.osDependencies ? { osDependencies: m.osDependencies as unknown[] } : {}),
|
||||
has: m.has as CatalogueEntry['has'],
|
||||
};
|
||||
}
|
||||
|
||||
/** Everything listed, newest registration last. Unlisted rows never leave this function. */
|
||||
export async function listCatalogue(): Promise<CatalogueEntry[]> {
|
||||
const rows = await marketplaceDb
|
||||
.select({
|
||||
appName: plugins.appName,
|
||||
developer: plugins.developer,
|
||||
repoUrl: plugins.repoUrl,
|
||||
metadata: pluginMetadata,
|
||||
})
|
||||
.from(plugins)
|
||||
.leftJoin(pluginMetadata, eq(pluginMetadata.pluginId, plugins.id))
|
||||
.where(eq(plugins.listed, true))
|
||||
.orderBy(plugins.appName);
|
||||
|
||||
return rows.map(toCatalogueEntry).filter((entry): entry is CatalogueEntry => entry !== null);
|
||||
}
|
||||
|
||||
/** One plugin, with its README. Null for an unknown or unlisted app name. */
|
||||
export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
|
||||
const [row] = await marketplaceDb
|
||||
.select({
|
||||
appName: plugins.appName,
|
||||
developer: plugins.developer,
|
||||
repoUrl: plugins.repoUrl,
|
||||
metadata: pluginMetadata,
|
||||
})
|
||||
.from(plugins)
|
||||
.leftJoin(pluginMetadata, eq(pluginMetadata.pluginId, plugins.id))
|
||||
.where(eq(plugins.appName, appName))
|
||||
.limit(1);
|
||||
|
||||
if (!row || !row.metadata) return null;
|
||||
const entry = toCatalogueEntry(row);
|
||||
if (!entry) return null;
|
||||
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
|
||||
}
|
||||
|
||||
// ── Writes. Used by the sync script, never by the public API ──
|
||||
|
||||
type RegisterParams = { appName: string; developer: string; repoUrl: string };
|
||||
|
||||
/** Register a plugin, or update where it lives. Returns its id. */
|
||||
export async function upsertPlugin({ appName, developer, repoUrl }: RegisterParams): Promise<number> {
|
||||
const [row] = await marketplaceDb
|
||||
.insert(plugins)
|
||||
.values({ appName, developer, repoUrl })
|
||||
.onConflictDoUpdate({
|
||||
target: plugins.appName,
|
||||
set: { developer, repoUrl, updatedAt: new Date() },
|
||||
})
|
||||
.returning({ id: plugins.id });
|
||||
|
||||
if (!row) throw new Error(`upsertPlugin: no row returned for ${appName}`);
|
||||
return row.id;
|
||||
}
|
||||
|
||||
export type MetadataParams = Omit<typeof pluginMetadata.$inferInsert, 'id' | 'pluginId' | 'fetchedAt'>;
|
||||
|
||||
/** Replace a plugin's cached metadata. One row per plugin — see the schema for why. */
|
||||
export async function upsertMetadata(pluginId: number, data: MetadataParams): Promise<void> {
|
||||
await marketplaceDb
|
||||
.insert(pluginMetadata)
|
||||
.values({ ...data, pluginId })
|
||||
.onConflictDoUpdate({
|
||||
target: pluginMetadata.pluginId,
|
||||
set: { ...data, fetchedAt: new Date() },
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
import { boolean, integer, jsonb, pgTable, serial, text, timestamp, uniqueIndex } from 'drizzle-orm/pg-core';
|
||||
|
||||
// What the marketplace offers, and what it knows about each thing it offers.
|
||||
//
|
||||
// ── Two tables, because they have different owners and different lifetimes ──
|
||||
//
|
||||
// `plugins` is the REGISTRY: the app name, who wrote it, where to clone it from. Every row is put there
|
||||
// deliberately by whoever runs the marketplace, and nothing outside this database may change one. It is
|
||||
// small, hand-curated, and the only part that is authoritative.
|
||||
//
|
||||
// `plugin_metadata` is a CACHE of what that repository says about itself — its label, its icon, the
|
||||
// permissions it will ask for, its README. All of it already exists in the plugin's own `manifest.ts`,
|
||||
// which is the real source of truth and stays that way. This table exists so the website can render a
|
||||
// catalogue without cloning three repositories on every page load.
|
||||
//
|
||||
// Keeping them apart is what makes the cache safe to be wrong. A metadata row can be deleted, rebuilt
|
||||
// from the repository, or found to be stale, and none of that can lose the fact that the plugin exists
|
||||
// or where it lives. Merging them would make a refresh a write to the registry, which is exactly the
|
||||
// kind of thing that turns a failed fetch into a plugin disappearing from the store.
|
||||
//
|
||||
// ── The platform re-reads everything after cloning, deliberately ──
|
||||
//
|
||||
// Nothing here is trusted by an install. The catalogue is what the owner sees BEFORE consenting — the
|
||||
// permissions it will ask for, the host binaries it will install — and after the clone the plugin's own
|
||||
// manifest is what the platform reads. A metadata row that disagrees with the repository is a marketplace
|
||||
// bug, never a second source of truth.
|
||||
|
||||
/**
|
||||
* A plugin the marketplace lists. Identity and provenance only.
|
||||
*
|
||||
* `repoUrl` is HTTPS, never SSH: the marketplace serves one URL to every machine, and an `ssh://` clone
|
||||
* assumes a key on that host for that user — true where the plugin was published and false everywhere
|
||||
* else. Anonymous HTTPS also keeps credentials out of the install path entirely.
|
||||
*/
|
||||
export const plugins = pgTable(
|
||||
'plugins',
|
||||
{
|
||||
id: serial('id').primaryKey(),
|
||||
/** The directory it clones into and its id everywhere else. Matches the plugin's own `appName`. */
|
||||
appName: text('app_name').notNull(),
|
||||
/** Who publishes it. A name today; an account when submissions exist. */
|
||||
developer: text('developer').notNull(),
|
||||
repoUrl: text('repo_url').notNull(),
|
||||
/**
|
||||
* Off hides it from the public API without deleting it or its metadata. A plugin taken down for a
|
||||
* bad release has to come back with its history, and a delete makes that a re-registration.
|
||||
*/
|
||||
listed: boolean('listed').notNull().default(true),
|
||||
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
|
||||
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
(table) => [uniqueIndex('uq_marketplace_plugins_app_name').on(table.appName)],
|
||||
);
|
||||
|
||||
/**
|
||||
* What a plugin's repository says about itself, as of the last sync.
|
||||
*
|
||||
* One row per plugin — this is a snapshot, not a history. Versions are a separate feature and would be a
|
||||
* separate table; pretending this one holds them by letting rows accumulate would give a store with no
|
||||
* way to say which row is current.
|
||||
*
|
||||
* The JSON columns hold the manifest's own shapes (`PluginPermission[]`, `OsDependency[]`, the `has`
|
||||
* flags) rather than being flattened into columns. They are read as a unit, rendered as a unit, and
|
||||
* their shape is owned by `plugins/manifest.ts` in the platform — flattening would mean a migration here
|
||||
* every time a manifest gains a field, for no query anyone runs.
|
||||
*/
|
||||
export const pluginMetadata = pgTable(
|
||||
'plugin_metadata',
|
||||
{
|
||||
id: serial('id').primaryKey(),
|
||||
pluginId: integer('plugin_id')
|
||||
.notNull()
|
||||
.references(() => plugins.id, { onDelete: 'cascade' }),
|
||||
|
||||
version: text('version').notNull(),
|
||||
/** The platform range it declares support for, e.g. `>=1.0.0`. */
|
||||
platform: text('platform').notNull(),
|
||||
label: text('label').notNull(),
|
||||
summary: text('summary').notNull(),
|
||||
color: text('color'),
|
||||
/** `badge` or `bare` — whether the tile draws a coloured square behind the icon. */
|
||||
tile: text('tile'),
|
||||
/** Absolute URL to artwork. A store must show an icon for something the machine has never seen. */
|
||||
iconUrl: text('icon_url'),
|
||||
/** A lucide glyph NAME, for a plugin that ships no artwork. `iconUrl` wins where both are given. */
|
||||
icon: text('icon'),
|
||||
|
||||
permissions: jsonb('permissions').notNull(),
|
||||
osDependencies: jsonb('os_dependencies'),
|
||||
/** `{ api, schema, sidecar, web }` — what the plugin ships. */
|
||||
has: jsonb('has').notNull(),
|
||||
|
||||
/** The repository's README, rendered on the plugin's page. Null until one exists. */
|
||||
readme: text('readme'),
|
||||
|
||||
/** The commit this snapshot was taken from, so a stale row can be told from an unchanged one. */
|
||||
sourceCommit: text('source_commit'),
|
||||
fetchedAt: timestamp('fetched_at', { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
(table) => [uniqueIndex('uq_marketplace_plugin_metadata_plugin').on(table.pluginId)],
|
||||
);
|
||||
@@ -0,0 +1,5 @@
|
||||
// drizzle-kit's view of the marketplace database.
|
||||
//
|
||||
// Unlike `officer_db`, there is no generated plugin barrel to keep out of the runtime graph: this
|
||||
// database has one owner and a fixed set of tables. Nothing on this machine may be absent from it.
|
||||
export * from './plugins/schema';
|
||||
Reference in New Issue
Block a user