the marketplace publishes releases, not working trees
The sync read `platform/plugins/` — the directories this machine has installed — and imported each manifest off the working tree. So publishing meant "get officer-dev's checkout to the release, then run the script", and nothing showed when that had not happened. Live proof at the time of writing: av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26 were all in the store, and none of the four has ever been tagged. It is also why the job could not go on a timer. A cron over it published whatever was in a working tree, on a schedule. Now every registered plugin is mirrored bare into ~/.cache/officer-marketplace/mirrors and read AT ITS NEWEST TAG — manifest, README and file list all out of `git cat-file`. No working tree in the path, so publishing something uncommitted stops being a mistake anyone can make. docs/marketplace.md proposed reading manifests over Gitea's API, which needed a sandbox or a committed manifest.json. Git gives the same guarantee with neither, and the trust model is unchanged: we import our own repositories' code. Third-party submissions still need one of those two. The list of plugins now comes from the `plugins` registry table, which the docs always called the authoritative half and which was in fact a by-product of what happened to be installed. That is what lets `finances` be published without being installed here, and it spans orgs — fofos-admin lives under fofos/, which an org listing would have missed. Adding one is `--add <clone url>`. A plugin with no tag is left alone and named in the report, not deleted. It has made no release, so there is nothing true to say about it; dropping its row would be a decision about somebody's plugin dressed up as a refresh. Also: `release_tag` on plugin_metadata, because the tag is the fact when a manifest version disagrees with it; the sync warns on a mismatch. And icon URLs pin to the tag — `/raw/HEAD/` served whatever was on the default branch, via a 303, so a plugin could show one icon and ship another. Verified: 21/25 published from a release, 4 named as untagged, all 14 tag-pinned icons answer 200. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+60
-164
@@ -1,43 +1,30 @@
|
|||||||
#!/usr/bin/env bun
|
#!/usr/bin/env bun
|
||||||
import { existsSync, readFileSync } from 'node:fs';
|
import { upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
|
||||||
import { basename, join, resolve } from 'node:path';
|
import { syncAll, syncPlugin } from '../src/servers/marketplace/sync';
|
||||||
import { upsertMetadata, upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
|
|
||||||
import { discoverPlugins } from '../src/servers/plugins/discover';
|
|
||||||
|
|
||||||
// Fill the marketplace database from the plugin repositories.
|
/**
|
||||||
//
|
* Fill the marketplace catalogue from the plugins' releases.
|
||||||
// ── What this reads, and why it is not the marketplace's own opinion ──
|
*
|
||||||
//
|
* bun --env-file=.env scripts/marketplace-sync.ts every registered plugin
|
||||||
// Every field except "where to clone from" comes out of the plugin's `manifest.ts` and its README. The
|
* bun --env-file=.env scripts/marketplace-sync.ts photos just one
|
||||||
// marketplace stores a COPY so the website can render without cloning three repositories per page load;
|
* bun --env-file=.env scripts/marketplace-sync.ts --add <url> register a new one, then sync it
|
||||||
// the manifest stays the source of truth, and the platform re-reads it after cloning. A row here that
|
*
|
||||||
// disagrees with the repository is a stale cache, not a second opinion — which is why re-running this is
|
* All the reasoning is in `src/servers/marketplace/sync.ts` — this file is the command line around it.
|
||||||
// always safe and never merges: it overwrites.
|
* It is a thin wrapper on purpose: the same functions are what the Gitea webhook calls, so a plugin
|
||||||
//
|
* published by hand and one published by pushing a tag go through exactly the same code.
|
||||||
// ── Reading the manifest by importing it ──
|
*
|
||||||
//
|
* Re-running is always safe. It overwrites and never merges.
|
||||||
// A manifest is TypeScript, not JSON, so it is imported and evaluated rather than parsed. That is code
|
*/
|
||||||
// execution, and it is fine for OUR repositories and not fine as a general mechanism.
|
|
||||||
//
|
|
||||||
// `[blocker for third-party submissions]` A stranger's plugin cannot be imported by the machine that
|
|
||||||
// serves the catalogue. Whoever builds submissions has to either evaluate manifests in a sandbox or make
|
|
||||||
// authors publish a `manifest.json` alongside. Recorded rather than solved: nobody can submit yet.
|
|
||||||
//
|
|
||||||
// ── Local clones, for now ──
|
|
||||||
//
|
|
||||||
// It syncs from `plugins/` on this machine rather than by cloning each repo into a temp directory. That
|
|
||||||
// keeps tonight's scope to "get real data into the database" — and it means the machine running the sync
|
|
||||||
// must have the plugins checked out. See `docs/marketplace.md` → "Sync" for the version that clones.
|
|
||||||
|
|
||||||
type Sync = { appName: string; developer: string; repoUrl: string };
|
const args = process.argv.slice(2);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A clone URL the whole internet can use.
|
* A clone URL the whole internet can use.
|
||||||
*
|
*
|
||||||
* `plugins/example` and `plugins/offscale` have `git@gitea.officer.dev:plugins/<name>.git` remotes,
|
* Several plugin repos have `git@gitea.officer.dev:plugins/<name>.git` remotes, because whoever set them
|
||||||
* because whoever set them up had a key on that host. Publishing that to a catalogue would hand every
|
* up had a key on that host. Registering one of those would hand every installer a URL that only works
|
||||||
* installer a URL that only works on the machine it was published from — the install fails at
|
* on the machine it was published from — the install fails at authentication, which reads as "the plugin
|
||||||
* authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong".
|
* is broken" rather than "the catalogue is wrong".
|
||||||
*/
|
*/
|
||||||
export function toHttpsRemote(remote: string): string | null {
|
export function toHttpsRemote(remote: string): string | null {
|
||||||
const trimmed = remote.trim();
|
const trimmed = remote.trim();
|
||||||
@@ -60,142 +47,51 @@ export function toHttpsRemote(remote: string): string | null {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
const git = (dir: string, args: string[]): string | null => {
|
if (args[0] === '--add') {
|
||||||
const result = Bun.spawnSync(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
|
const raw = args[1];
|
||||||
if (result.exitCode !== 0) return null;
|
if (!raw) {
|
||||||
return new TextDecoder().decode(result.stdout).trim() || null;
|
console.error('usage: marketplace-sync.ts --add <clone url>');
|
||||||
};
|
process.exit(1);
|
||||||
|
|
||||||
/*
|
|
||||||
Which plugins directory to publish FROM.
|
|
||||||
|
|
||||||
There is one: `platform/plugins/`, holding what this machine has INSTALLED — updated by
|
|
||||||
`plugins:update`, which is a git pull, so it sits at whatever version was last installed.
|
|
||||||
|
|
||||||
Publishing from it means the catalogue advertises the version this machine happens to be running
|
|
||||||
rather than the version that was released. That is exactly what happened on 2026-08-27: 21 plugins
|
|
||||||
were tagged and the store went on showing 1.0.0.
|
|
||||||
|
|
||||||
A second directory — `officer-dev/plugins/`, every repo in the org cloned side by side — was created
|
|
||||||
to fix that, and never did: its entries were SYMLINKS into this one, so both paths read the same
|
|
||||||
working copy and published the same version. It was removed on 2026-08-28.
|
|
||||||
|
|
||||||
**The problem is still open, and a second checkout was never the answer.** Each directory here is a
|
|
||||||
real git repo with its own tags, so the released manifest can be read out of the tag itself —
|
|
||||||
`git show <newest-tag>:manifest.ts` — without caring what is checked out. That is the fix; it is not
|
|
||||||
written yet, and until it is, this script advertises what is installed.
|
|
||||||
*/
|
|
||||||
const PLUGINS_DIR = process.env.PLUGINS_DIR ?? resolve(import.meta.dir, '..', 'plugins');
|
|
||||||
|
|
||||||
async function syncOne(dir: string): Promise<Sync | null> {
|
|
||||||
const appName = basename(dir);
|
|
||||||
const manifestPath = join(dir, 'manifest.ts');
|
|
||||||
if (!existsSync(manifestPath)) {
|
|
||||||
console.warn(`[marketplace-sync] ${appName}: no manifest.ts — skipped`);
|
|
||||||
return null;
|
|
||||||
}
|
}
|
||||||
|
const repoUrl = toHttpsRemote(raw);
|
||||||
const remote = git(dir, ['remote', 'get-url', 'origin']);
|
|
||||||
const repoUrl = remote ? toHttpsRemote(remote) : null;
|
|
||||||
if (!repoUrl) {
|
if (!repoUrl) {
|
||||||
console.warn(`[marketplace-sync] ${appName}: no usable https remote (${remote ?? 'none'}) — skipped`);
|
console.error(`[marketplace-sync] ${raw} is not a clone URL this can turn into https`);
|
||||||
return null;
|
process.exit(1);
|
||||||
|
}
|
||||||
|
const appName = repoUrl.replace(/\.git$/, '').split('/').pop() ?? '';
|
||||||
|
if (!/^[a-z0-9][a-z0-9-]{0,63}$/.test(appName)) {
|
||||||
|
console.error(`[marketplace-sync] "${appName}" is not a usable app name`);
|
||||||
|
process.exit(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
const { manifest } = (await import(manifestPath)) as { manifest: Record<string, unknown> };
|
// Registered with a placeholder developer, then immediately synced — which reads the real publisher
|
||||||
if (!manifest) {
|
// out of the released manifest and overwrites it. A repository with no tag stays registered and
|
||||||
console.warn(`[marketplace-sync] ${appName}: manifest.ts exports no \`manifest\` — skipped`);
|
// unpublished, which is the correct state for one that has not made a release yet.
|
||||||
return null;
|
await upsertPlugin({ appName, developer: 'unknown', repoUrl });
|
||||||
|
console.log(`[marketplace-sync] registered ${appName} → ${repoUrl}`);
|
||||||
|
const one = await syncPlugin(appName);
|
||||||
|
report([one]);
|
||||||
|
process.exit(one.status === 'failed' ? 1 : 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
const results = args[0] ? [await syncPlugin(args[0])] : await syncAll();
|
||||||
|
report(results);
|
||||||
|
process.exit(results.some((r) => r.status === 'failed') ? 1 : 0);
|
||||||
|
|
||||||
|
function report(results: Awaited<ReturnType<typeof syncAll>>): void {
|
||||||
|
for (const r of results) {
|
||||||
|
if (r.status === 'synced') console.log(`[marketplace-sync] ${r.appName} ← ${r.tag}`);
|
||||||
|
else if (r.status === 'unreleased') console.warn(`[marketplace-sync] ${r.appName}: no tag — nothing published`);
|
||||||
|
else console.error(`[marketplace-sync] ${r.appName} failed: ${r.error}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// What the plugin SHIPS, read from the directory rather than declared — the same rule discovery
|
const synced = results.filter((r) => r.status === 'synced').length;
|
||||||
// follows, so a catalogue entry cannot claim a sidecar the repository does not have.
|
const unreleased = results.filter((r) => r.status === 'unreleased').map((r) => r.appName);
|
||||||
const has = {
|
const failed = results.filter((r) => r.status === 'failed').map((r) => r.appName);
|
||||||
api: existsSync(join(dir, 'api', 'router.ts')),
|
|
||||||
schema: existsSync(join(dir, 'db', 'schema.ts')),
|
|
||||||
sidecar: existsSync(join(dir, 'sidecar', 'index.ts')) || existsSync(join(dir, 'sidecar', 'index.mjs')),
|
|
||||||
web: existsSync(join(dir, 'web')),
|
|
||||||
};
|
|
||||||
|
|
||||||
const readmePath = ['README.md', 'readme.md'].map((n) => join(dir, n)).find((p) => existsSync(p));
|
console.log(`[marketplace-sync] ${synced}/${results.length} published from a release`);
|
||||||
const readme = readmePath ? readFileSync(readmePath, 'utf-8') : null;
|
// Named rather than counted. "3 unreleased" is a number nobody acts on; the names are a to-do list,
|
||||||
|
// and every one of them is a plugin whose store entry is frozen until somebody tags it.
|
||||||
const developer = String(manifest.publisher ?? 'unknown');
|
if (unreleased.length) console.warn(`[marketplace-sync] never tagged, so not updated: ${unreleased.join(', ')}`);
|
||||||
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
|
if (failed.length) console.error(`[marketplace-sync] failed: ${failed.join(', ')}`);
|
||||||
|
|
||||||
await upsertMetadata(pluginId, {
|
|
||||||
version: String(manifest.version ?? '0.0.0'),
|
|
||||||
platform: String(manifest.platform ?? '>=1.0.0'),
|
|
||||||
label: String(manifest.label ?? appName),
|
|
||||||
summary: String(manifest.summary ?? ''),
|
|
||||||
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
|
|
||||||
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
|
|
||||||
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
|
|
||||||
// quietly disagreeing with the repository it claims to mirror.
|
|
||||||
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
|
|
||||||
color: (manifest.color as string) ?? null,
|
|
||||||
tile: (manifest.tile as string) ?? null,
|
|
||||||
// Artwork lives in the repo, so the URL is built from the remote rather than stored by hand. Only
|
|
||||||
// claimed when the file is actually there — a broken image is worse than a glyph.
|
|
||||||
iconUrl: existsSync(join(dir, 'assets', 'icon.png')) ? rawUrl(repoUrl, 'assets/icon.png') : null,
|
|
||||||
icon: (manifest.icon as string) ?? null,
|
|
||||||
permissions: manifest.permissions ?? [],
|
|
||||||
osDependencies: manifest.osDependencies ?? null,
|
|
||||||
has,
|
|
||||||
readme,
|
|
||||||
sourceCommit: git(dir, ['rev-parse', 'HEAD']),
|
|
||||||
});
|
|
||||||
|
|
||||||
console.log(
|
|
||||||
`[marketplace-sync] ${appName} ← ${repoUrl}${readme ? '' : ' (no README)'}${has.web ? '' : ' (no web/)'}`,
|
|
||||||
);
|
|
||||||
return { appName, developer, repoUrl };
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* A raw-file URL for a Gitea repository.
|
|
||||||
*
|
|
||||||
* `HEAD`, not a branch name. This said `branch/main` and the comment here warned that "a repo whose
|
|
||||||
* default branch is `master` gets a 404" — which then happened: four plugin repositories were created
|
|
||||||
* on `master` and every one of them served a broken image in the marketplace on any server that reads
|
|
||||||
* the remote catalogue. A warning in a comment is not a guard.
|
|
||||||
*
|
|
||||||
* Gitea resolves `HEAD` to whatever the default branch is, so the question stops being asked. Verified
|
|
||||||
* against both spellings.
|
|
||||||
*
|
|
||||||
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
|
|
||||||
* be part of the registry rather than inferred from the URL.
|
|
||||||
*/
|
|
||||||
function rawUrl(repoUrl: string, path: string): string {
|
|
||||||
return `${repoUrl.replace(/\.git$/, '')}/raw/HEAD/${path}`;
|
|
||||||
}
|
|
||||||
|
|
||||||
const dirs = existsSync(PLUGINS_DIR)
|
|
||||||
? [...new Bun.Glob('*/').scanSync({ cwd: PLUGINS_DIR, onlyFiles: false })]
|
|
||||||
// Not a plugin: the shared install target for plugin npm packages, which lives here because bun
|
|
||||||
// resolves node_modules up the tree. `discoverPlugins` skips it for the same reason.
|
|
||||||
.filter((d) => d.replace(/\/$/, '') !== 'node_modules')
|
|
||||||
.map((d) => join(PLUGINS_DIR, d))
|
|
||||||
: [];
|
|
||||||
|
|
||||||
if (dirs.length === 0) {
|
|
||||||
console.log(`[marketplace-sync] no plugin directories under ${PLUGINS_DIR} — nothing to sync`);
|
|
||||||
process.exit(0);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Discovery is not used to FIND them — a plugin that fails discovery is exactly the kind of thing a
|
|
||||||
// catalogue should still be able to describe — but its broken list is worth surfacing here.
|
|
||||||
const { broken } = await discoverPlugins().catch(() => ({ broken: [] as { appName: string; error: string }[] }));
|
|
||||||
for (const { appName, error } of broken) console.warn(`[marketplace-sync] ${appName} is broken on disk: ${error}`);
|
|
||||||
|
|
||||||
const synced: Sync[] = [];
|
|
||||||
for (const dir of dirs) {
|
|
||||||
const result = await syncOne(dir).catch((err) => {
|
|
||||||
console.error(`[marketplace-sync] ${basename(dir)} failed: ${err instanceof Error ? err.message : err}`);
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
if (result) synced.push(result);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`[marketplace-sync] ${synced.length}/${dirs.length} plugin(s) in the catalogue`);
|
|
||||||
process.exit(0);
|
|
||||||
|
|||||||
@@ -14,6 +14,14 @@ export type CatalogueEntry = {
|
|||||||
appName: string;
|
appName: string;
|
||||||
publisher: string;
|
publisher: string;
|
||||||
version: string;
|
version: string;
|
||||||
|
/**
|
||||||
|
* The git tag this entry was published from.
|
||||||
|
*
|
||||||
|
* Optional on the wire, not in the database, and the two mean different things. Here it is absent for
|
||||||
|
* a row that predates release-based publishing OR for a catalogue too old to send the field at all —
|
||||||
|
* so a consumer must treat "no tag" as "cannot tell" and fall back to whatever it knew before.
|
||||||
|
*/
|
||||||
|
releaseTag?: string;
|
||||||
platform: string;
|
platform: string;
|
||||||
label: string;
|
label: string;
|
||||||
summary: string;
|
summary: string;
|
||||||
@@ -52,6 +60,7 @@ function toCatalogueEntry(row: Row): CatalogueEntry | null {
|
|||||||
appName: row.appName,
|
appName: row.appName,
|
||||||
publisher: row.developer,
|
publisher: row.developer,
|
||||||
version: m.version,
|
version: m.version,
|
||||||
|
...(m.releaseTag ? { releaseTag: m.releaseTag } : {}),
|
||||||
platform: m.platform,
|
platform: m.platform,
|
||||||
label: m.label,
|
label: m.label,
|
||||||
summary: m.summary,
|
summary: m.summary,
|
||||||
@@ -106,6 +115,22 @@ export async function catalogueEntry(appName: string): Promise<CatalogueEntry |
|
|||||||
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
|
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every registered plugin — identity and clone URL, no metadata.
|
||||||
|
*
|
||||||
|
* This is what the sync iterates, and it is the reason the registry is called authoritative: the
|
||||||
|
* catalogue lists what somebody deliberately registered, not what happens to be checked out on the
|
||||||
|
* machine that runs the job. Unlisted rows are included; hiding a plugin from the public API is not the
|
||||||
|
* same as no longer tracking it, and one taken down for a bad release has to be able to come back
|
||||||
|
* current.
|
||||||
|
*/
|
||||||
|
export async function listRegistry(): Promise<{ id: number; appName: string; repoUrl: string }[]> {
|
||||||
|
return marketplaceDb
|
||||||
|
.select({ id: plugins.id, appName: plugins.appName, repoUrl: plugins.repoUrl })
|
||||||
|
.from(plugins)
|
||||||
|
.orderBy(plugins.appName);
|
||||||
|
}
|
||||||
|
|
||||||
// ── Writes. Used by the sync script, never by the public API ──
|
// ── Writes. Used by the sync script, never by the public API ──
|
||||||
|
|
||||||
type RegisterParams = { appName: string; developer: string; repoUrl: string };
|
type RegisterParams = { appName: string; developer: string; repoUrl: string };
|
||||||
|
|||||||
@@ -73,6 +73,19 @@ export const pluginMetadata = pgTable(
|
|||||||
.references(() => plugins.id, { onDelete: 'cascade' }),
|
.references(() => plugins.id, { onDelete: 'cascade' }),
|
||||||
|
|
||||||
version: text('version').notNull(),
|
version: text('version').notNull(),
|
||||||
|
/**
|
||||||
|
* The git tag this row was read from — what "published" actually means.
|
||||||
|
*
|
||||||
|
* `version` above is the manifest's own claim, and the two are the same string every time somebody
|
||||||
|
* tags correctly. They are kept apart because when they DISAGREE the tag is the fact: it is what was
|
||||||
|
* pushed, and a manifest whose version nobody bumped would otherwise make a real release invisible
|
||||||
|
* to anything comparing versions.
|
||||||
|
*
|
||||||
|
* Nullable because rows existed before releases were the unit of publishing. A null one is a
|
||||||
|
* snapshot taken from a working tree by the old sync, and every consumer should read it as "unknown"
|
||||||
|
* rather than "no release".
|
||||||
|
*/
|
||||||
|
releaseTag: text('release_tag'),
|
||||||
/** The platform range it declares support for, e.g. `>=1.0.0`. */
|
/** The platform range it declares support for, e.g. `>=1.0.0`. */
|
||||||
platform: text('platform').notNull(),
|
platform: text('platform').notNull(),
|
||||||
label: text('label').notNull(),
|
label: text('label').notNull(),
|
||||||
|
|||||||
@@ -0,0 +1,287 @@
|
|||||||
|
import { existsSync, mkdirSync } from 'node:fs';
|
||||||
|
import { homedir, tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import {
|
||||||
|
listRegistry,
|
||||||
|
upsertMetadata,
|
||||||
|
upsertPlugin,
|
||||||
|
} from '../../databases/marketplace_db/src/plugins/queries';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fill the catalogue from what each plugin has RELEASED.
|
||||||
|
*
|
||||||
|
* ── The bug this file exists to remove ──
|
||||||
|
*
|
||||||
|
* Until 2026-09-03 the sync read `platform/plugins/` — the directories this machine has INSTALLED — and
|
||||||
|
* imported each `manifest.ts` off the working tree. So "publish" meant "make sure officer-dev's checkout
|
||||||
|
* happens to be at the release, then run the script", and nobody could see when it was not. On the day
|
||||||
|
* this was rewritten the live catalogue was advertising four plugins from untagged working copies:
|
||||||
|
* av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of
|
||||||
|
* which had ever been tagged. The store was describing somebody's afternoon.
|
||||||
|
*
|
||||||
|
* That is also why the old script could not be put on a timer: a cron over it would have published
|
||||||
|
* whatever was sitting in a working tree, on a schedule.
|
||||||
|
*
|
||||||
|
* ── What replaces it: a bare mirror, read at a tag ──
|
||||||
|
*
|
||||||
|
* Every plugin in the registry is mirrored (`git clone --mirror`) into a cache directory, refreshed, and
|
||||||
|
* read AT ITS NEWEST TAG — the manifest, the README, the file list, all out of `git cat-file`. There is
|
||||||
|
* no working tree anywhere in this path, so publishing something uncommitted is not a mistake that can
|
||||||
|
* be made. A mirror is cheap: the whole set clones in a couple of seconds and refreshes in well under
|
||||||
|
* one.
|
||||||
|
*
|
||||||
|
* `docs/marketplace.md` proposed instead reading manifests over Gitea's API, which needed either a
|
||||||
|
* sandbox to evaluate a fetched string or a committed `manifest.json` for every plugin. Git gives the
|
||||||
|
* same guarantee — content at a named release, independent of any checkout — with neither, and the trust
|
||||||
|
* model is unchanged from today: we import our own repositories' code.
|
||||||
|
*
|
||||||
|
* `[still a blocker for third-party submissions]` It is unchanged BECAUSE it is unchanged. Importing a
|
||||||
|
* stranger's manifest is code execution no matter which ref it came from. Submissions still need a
|
||||||
|
* sandbox or a published `manifest.json`.
|
||||||
|
*
|
||||||
|
* ── Which plugins, and where from: the registry, and only the registry ──
|
||||||
|
*
|
||||||
|
* The list comes from the `plugins` table, which `docs/marketplace.md` has always called the
|
||||||
|
* authoritative, hand-curated half. It was not actually being used that way — the old sync discovered
|
||||||
|
* plugins by listing a directory, so the registry was a by-product of whatever officer-dev had
|
||||||
|
* installed. Now it decides, which means:
|
||||||
|
*
|
||||||
|
* · a plugin nobody has installed here can still be published (`finances` is in exactly that state),
|
||||||
|
* · a plugin lives wherever its registry row says, across orgs — `fofos-admin` is under `fofos/`,
|
||||||
|
* not `plugins/`, so an org listing would have missed it,
|
||||||
|
* · and adding one is a deliberate act: `marketplace-sync.ts --add <https clone url>`.
|
||||||
|
*
|
||||||
|
* ── A plugin with no tag is left ALONE, not removed ──
|
||||||
|
*
|
||||||
|
* It has never made a release, so this file has nothing true to say about it and says nothing: the
|
||||||
|
* existing metadata row stays, and the run reports it. Deleting the row would drop it out of the store
|
||||||
|
* on the day the rule changed, which is a decision about somebody's plugin dressed up as a refresh.
|
||||||
|
* Tagging it is the fix, and the report is there to ask for one.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type SyncOutcome =
|
||||||
|
| { appName: string; status: 'synced'; tag: string; version: string }
|
||||||
|
| { appName: string; status: 'unreleased' }
|
||||||
|
| { appName: string; status: 'failed'; error: string };
|
||||||
|
|
||||||
|
/*
|
||||||
|
Where the mirrors live.
|
||||||
|
|
||||||
|
A fixed path under the user's cache directory, NOT one derived from the environment. This runs three
|
||||||
|
ways — by hand with `--env-file=.env`, from the webhook inside `officer-marketplace`, and from cron —
|
||||||
|
and those three do not have the same environment. A path that moved with `DATA_PATH` would have each
|
||||||
|
of them re-cloning twenty-five repositories into its own copy, and the "is it already mirrored" check
|
||||||
|
would answer differently depending on who asked.
|
||||||
|
|
||||||
|
Not in the source tree, because it is a cache of remote repositories and a `.gitignore` entry is a
|
||||||
|
worse way to keep several hundred megabytes out of a repo than not putting them there.
|
||||||
|
*/
|
||||||
|
const MIRROR_DIR =
|
||||||
|
process.env.MARKETPLACE_MIRROR_DIR ?? join(homedir(), '.cache', 'officer-marketplace', 'mirrors');
|
||||||
|
|
||||||
|
const GIT_TIMEOUT_MS = 60_000;
|
||||||
|
|
||||||
|
type Git = { ok: true; out: string } | { ok: false; err: string };
|
||||||
|
|
||||||
|
async function git(args: string[], cwd?: string): Promise<Git> {
|
||||||
|
// A missing `cwd` makes posix_spawn fail with ENOENT, which reads as "git is not installed" and sent
|
||||||
|
// a first run of this looking for git for ten minutes. It means the mirror is not there yet.
|
||||||
|
if (cwd && !existsSync(cwd)) return { ok: false, err: `${cwd} does not exist` };
|
||||||
|
|
||||||
|
const proc = Bun.spawn(['git', ...args], {
|
||||||
|
cwd,
|
||||||
|
stdout: 'pipe',
|
||||||
|
stderr: 'pipe',
|
||||||
|
// Never inherited: a mirror fetch that hits a host asking for a password would otherwise sit on a
|
||||||
|
// terminal prompt forever, and in the webhook server there is no terminal to sit on.
|
||||||
|
env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GIT_ASKPASS: '/bin/true' },
|
||||||
|
signal: AbortSignal.timeout(GIT_TIMEOUT_MS),
|
||||||
|
});
|
||||||
|
const [out, err] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]);
|
||||||
|
return (await proc.exited) === 0 ? { ok: true, out: out.trim() } : { ok: false, err: err.trim() || 'git failed' };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The mirror for a plugin, cloned if this is the first time and refreshed if it is not. */
|
||||||
|
async function mirror(appName: string, repoUrl: string): Promise<string> {
|
||||||
|
mkdirSync(MIRROR_DIR, { recursive: true });
|
||||||
|
const dir = join(MIRROR_DIR, `${appName}.git`);
|
||||||
|
|
||||||
|
// Re-cloned rather than repaired when it is not a healthy bare repo: an interrupted clone leaves a
|
||||||
|
// directory that every later command fails against, and there is nothing in it worth keeping.
|
||||||
|
const existing = await git(['rev-parse', '--is-bare-repository'], dir);
|
||||||
|
if (!existing.ok || existing.out !== 'true') {
|
||||||
|
if (existsSync(dir)) await Bun.$`rm -rf ${dir}`.quiet().nothrow();
|
||||||
|
const cloned = await git(['clone', '--mirror', '--quiet', repoUrl, dir]);
|
||||||
|
if (!cloned.ok) throw new Error(`clone failed: ${cloned.err}`);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The registry is allowed to move a plugin. Setting the URL every time is one command and removes the
|
||||||
|
// failure where a repo is relocated and the mirror keeps refreshing from where it used to be.
|
||||||
|
await git(['remote', 'set-url', 'origin', repoUrl], dir);
|
||||||
|
|
||||||
|
// `--prune --prune-tags`: a deleted or MOVED tag has to disappear here too. Without it, a tag that was
|
||||||
|
// pushed by mistake and deleted would stay in the mirror forever and could go on being published as
|
||||||
|
// the newest release — a store advertising a version its repository no longer has.
|
||||||
|
const fetched = await git(['fetch', '--quiet', '--prune', '--prune-tags', '--tags', 'origin'], dir);
|
||||||
|
if (!fetched.ok) throw new Error(`fetch failed: ${fetched.err}`);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The newest release, or null for a plugin that has never made one.
|
||||||
|
*
|
||||||
|
* Asked of git rather than by sorting version strings, which is the same rule `servers/plugins/release.ts`
|
||||||
|
* follows and for the same reason: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
|
||||||
|
* then fails silently. `--abbrev=0` gives the closest tag reachable from HEAD, and a mirror's HEAD is the
|
||||||
|
* remote's default branch, so this is "the newest release on the branch the repository points at".
|
||||||
|
*/
|
||||||
|
async function newestTag(dir: string): Promise<string | null> {
|
||||||
|
const described = await git(['describe', '--tags', '--abbrev=0', 'HEAD'], dir);
|
||||||
|
return described.ok ? described.out : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A file at a ref, or null when it is not in that tree. */
|
||||||
|
async function fileAt(dir: string, ref: string, path: string): Promise<string | null> {
|
||||||
|
const shown = await git(['cat-file', '-p', `${ref}:${path}`], dir);
|
||||||
|
return shown.ok ? shown.out : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The manifest at a release, as an object.
|
||||||
|
*
|
||||||
|
* Written to a temp file and imported, because a manifest is TypeScript. Keyed by the COMMIT, so the
|
||||||
|
* module cache is correct rather than merely convenient: the same commit is the same file, and a tag
|
||||||
|
* moved to a different commit gets a different path and a fresh import. That matters in the webhook
|
||||||
|
* server, which is long-lived and would otherwise serve the first version of a manifest it ever read.
|
||||||
|
*
|
||||||
|
* The `import type { PluginManifest }` at the top of every manifest is erased by the transpiler, so the
|
||||||
|
* file has no runtime dependency on the platform's path aliases and imports fine from anywhere.
|
||||||
|
*/
|
||||||
|
async function manifestAt(dir: string, ref: string, appName: string, commit: string): Promise<Record<string, unknown>> {
|
||||||
|
const source = await fileAt(dir, ref, 'manifest.ts');
|
||||||
|
if (source === null) throw new Error(`no manifest.ts at ${ref}`);
|
||||||
|
|
||||||
|
const scratch = join(tmpdir(), 'officer-marketplace-manifests');
|
||||||
|
mkdirSync(scratch, { recursive: true });
|
||||||
|
const path = join(scratch, `${appName}-${commit}.ts`);
|
||||||
|
await Bun.write(path, source);
|
||||||
|
|
||||||
|
const mod = (await import(path)) as { manifest?: Record<string, unknown> };
|
||||||
|
if (!mod.manifest) throw new Error(`manifest.ts at ${ref} exports no \`manifest\``);
|
||||||
|
return mod.manifest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A raw-file URL for a Gitea repository, PINNED TO THE RELEASE.
|
||||||
|
*
|
||||||
|
* It used to be `/raw/HEAD/…`, which works only because Gitea 303s it to `/raw/branch/HEAD/…`; the
|
||||||
|
* artwork it served was whatever was on the default branch, so a plugin could show one icon in the store
|
||||||
|
* and ship another. `/raw/tag/<tag>/…` answers 200 directly and shows the icon that belongs to the
|
||||||
|
* version being advertised.
|
||||||
|
*
|
||||||
|
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
|
||||||
|
* be part of the registry rather than inferred from the URL.
|
||||||
|
*/
|
||||||
|
function rawUrl(repoUrl: string, tag: string, path: string): string {
|
||||||
|
return `${repoUrl.replace(/\.git$/, '')}/raw/tag/${encodeURIComponent(tag)}/${path}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function syncRegistered(row: { id: number; appName: string; repoUrl: string }): Promise<SyncOutcome> {
|
||||||
|
const { appName, repoUrl } = row;
|
||||||
|
const dir = await mirror(appName, repoUrl);
|
||||||
|
|
||||||
|
const tag = await newestTag(dir);
|
||||||
|
if (!tag) return { appName, status: 'unreleased' };
|
||||||
|
|
||||||
|
const commitAt = await git(['rev-list', '-n', '1', tag], dir);
|
||||||
|
if (!commitAt.ok) throw new Error(`cannot resolve ${tag}`);
|
||||||
|
const commit = commitAt.out;
|
||||||
|
|
||||||
|
const manifest = await manifestAt(dir, tag, appName, commit);
|
||||||
|
|
||||||
|
// What the plugin SHIPS, read from the tree at the release rather than from a declaration — the same
|
||||||
|
// rule discovery follows on disk, so a catalogue entry cannot claim a sidecar the release does not
|
||||||
|
// have.
|
||||||
|
const listed = await git(['ls-tree', '-r', '--name-only', tag], dir);
|
||||||
|
const files = new Set(listed.ok ? listed.out.split('\n') : []);
|
||||||
|
const has = {
|
||||||
|
api: files.has('api/router.ts'),
|
||||||
|
schema: files.has('db/schema.ts'),
|
||||||
|
sidecar: files.has('sidecar/index.ts') || files.has('sidecar/index.mjs'),
|
||||||
|
web: [...files].some((f) => f.startsWith('web/')),
|
||||||
|
};
|
||||||
|
|
||||||
|
const readme = (await fileAt(dir, tag, 'README.md')) ?? (await fileAt(dir, tag, 'readme.md'));
|
||||||
|
|
||||||
|
const version = String(manifest.version ?? tag);
|
||||||
|
if (version !== tag) {
|
||||||
|
// Not an error and not corrected here. The tag is what was released and the manifest is what the
|
||||||
|
// plugin will report about itself once installed; when they disagree, an update badge that compares
|
||||||
|
// versions goes quiet. Saying so is the only way anyone finds out.
|
||||||
|
console.warn(`[marketplace-sync] ${appName}: tag ${tag} but manifest says ${version} — they should match`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const developer = String(manifest.publisher ?? 'unknown');
|
||||||
|
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
|
||||||
|
|
||||||
|
await upsertMetadata(pluginId, {
|
||||||
|
version,
|
||||||
|
releaseTag: tag,
|
||||||
|
platform: String(manifest.platform ?? '>=1.0.0'),
|
||||||
|
label: String(manifest.label ?? appName),
|
||||||
|
summary: String(manifest.summary ?? ''),
|
||||||
|
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
|
||||||
|
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
|
||||||
|
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
|
||||||
|
// quietly disagreeing with the repository it claims to mirror.
|
||||||
|
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
|
||||||
|
color: (manifest.color as string) ?? null,
|
||||||
|
tile: (manifest.tile as string) ?? null,
|
||||||
|
iconUrl: files.has('assets/icon.png') ? rawUrl(repoUrl, tag, 'assets/icon.png') : null,
|
||||||
|
icon: (manifest.icon as string) ?? null,
|
||||||
|
permissions: manifest.permissions ?? [],
|
||||||
|
osDependencies: manifest.osDependencies ?? null,
|
||||||
|
has,
|
||||||
|
readme,
|
||||||
|
sourceCommit: commit,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { appName, status: 'synced', tag, version };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One plugin by app name. Unknown to the registry is a failure, not a silent no-op. */
|
||||||
|
export async function syncPlugin(appName: string): Promise<SyncOutcome> {
|
||||||
|
const row = (await listRegistry()).find((r) => r.appName === appName);
|
||||||
|
if (!row) return { appName, status: 'failed', error: 'not in the registry' };
|
||||||
|
return syncRegistered(row).catch((err) => ({
|
||||||
|
appName,
|
||||||
|
status: 'failed' as const,
|
||||||
|
error: err instanceof Error ? err.message : String(err),
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every registered plugin, one at a time.
|
||||||
|
*
|
||||||
|
* Sequential on purpose. Twenty-four concurrent fetches against one Gitea is a thundering herd for a job
|
||||||
|
* nothing is waiting on, and the whole set finishes in about a second anyway.
|
||||||
|
*
|
||||||
|
* Unlisted rows are included: `listed` hides a plugin from the public API, usually after a bad release,
|
||||||
|
* and it has to be able to come back with current metadata rather than with whatever was true when it
|
||||||
|
* was pulled.
|
||||||
|
*/
|
||||||
|
export async function syncAll(): Promise<SyncOutcome[]> {
|
||||||
|
const rows = await listRegistry();
|
||||||
|
const out: SyncOutcome[] = [];
|
||||||
|
for (const row of rows) {
|
||||||
|
out.push(
|
||||||
|
await syncRegistered(row).catch((err) => ({
|
||||||
|
appName: row.appName,
|
||||||
|
status: 'failed' as const,
|
||||||
|
error: err instanceof Error ? err.message : String(err),
|
||||||
|
})),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user