officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

the marketplace publishes releases, not working trees

Browse Source 
The sync read `platform/plugins/` — the directories this machine has installed — and imported each
manifest off the working tree. So publishing meant "get officer-dev's checkout to the release, then
run the script", and nothing showed when that had not happened. Live proof at the time of writing:
av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26 were all in
the store, and none of the four has ever been tagged.

It is also why the job could not go on a timer. A cron over it published whatever was in a working
tree, on a schedule.

Now every registered plugin is mirrored bare into ~/.cache/officer-marketplace/mirrors and read AT ITS
NEWEST TAG — manifest, README and file list all out of `git cat-file`. No working tree in the path, so
publishing something uncommitted stops being a mistake anyone can make.

docs/marketplace.md proposed reading manifests over Gitea's API, which needed a sandbox or a committed
manifest.json. Git gives the same guarantee with neither, and the trust model is unchanged: we import
our own repositories' code. Third-party submissions still need one of those two.

The list of plugins now comes from the `plugins` registry table, which the docs always called the
authoritative half and which was in fact a by-product of what happened to be installed. That is what
lets `finances` be published without being installed here, and it spans orgs — fofos-admin lives under
fofos/, which an org listing would have missed. Adding one is `--add <clone url>`.

A plugin with no tag is left alone and named in the report, not deleted. It has made no release, so
there is nothing true to say about it; dropping its row would be a decision about somebody's plugin
dressed up as a refresh.

Also: `release_tag` on plugin_metadata, because the tag is the fact when a manifest version disagrees
with it; the sync warns on a mismatch. And icon URLs pin to the tag — `/raw/HEAD/` served whatever was
on the default branch, via a 303, so a plugin could show one icon and ship another.

Verified: 21/25 published from a release, 4 named as untagged, all 14 tag-pinned icons answer 200.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-09-03 13:05:50 +01:00
co-authored by Claude Opus 5
parent 8d39586de5
commit 78e89ebcec
4 changed files with 385 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/marketplace-sync.ts
+60 -164
View File
@@ -1,43 +1,30 @@
#!/usr/bin/env bun
import { existsSync, readFileSync } from 'node:fs';
import { basename, join, resolve } from 'node:path';
import { upsertMetadata, upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
import { discoverPlugins } from '../src/servers/plugins/discover';
import { upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
import { syncAll, syncPlugin } from '../src/servers/marketplace/sync';
// Fill the marketplace database from the plugin repositories.
//
// ── What this reads, and why it is not the marketplace's own opinion ──
//
// Every field except "where to clone from" comes out of the plugin's `manifest.ts` and its README. The
// marketplace stores a COPY so the website can render without cloning three repositories per page load;
// the manifest stays the source of truth, and the platform re-reads it after cloning. A row here that
// disagrees with the repository is a stale cache, not a second opinion — which is why re-running this is
// always safe and never merges: it overwrites.
//
// ── Reading the manifest by importing it ──
//
// A manifest is TypeScript, not JSON, so it is imported and evaluated rather than parsed. That is code
// execution, and it is fine for OUR repositories and not fine as a general mechanism.
//
// `[blocker for third-party submissions]` A stranger's plugin cannot be imported by the machine that
// serves the catalogue. Whoever builds submissions has to either evaluate manifests in a sandbox or make
// authors publish a `manifest.json` alongside. Recorded rather than solved: nobody can submit yet.
//
// ── Local clones, for now ──
//
// It syncs from `plugins/` on this machine rather than by cloning each repo into a temp directory. That
// keeps tonight's scope to "get real data into the database" — and it means the machine running the sync
// must have the plugins checked out. See `docs/marketplace.md` → "Sync" for the version that clones.
/**
* Fill the marketplace catalogue from the plugins' releases.
*
* bun --env-file=.env scripts/marketplace-sync.ts every registered plugin
* bun --env-file=.env scripts/marketplace-sync.ts photos just one
* bun --env-file=.env scripts/marketplace-sync.ts --add <url> register a new one, then sync it
*
* All the reasoning is in `src/servers/marketplace/sync.ts` — this file is the command line around it.
* It is a thin wrapper on purpose: the same functions are what the Gitea webhook calls, so a plugin
* published by hand and one published by pushing a tag go through exactly the same code.
*
* Re-running is always safe. It overwrites and never merges.
*/
type Sync = { appName: string; developer: string; repoUrl: string };
const args = process.argv.slice(2);
/**
* A clone URL the whole internet can use.
*
* `plugins/example` and `plugins/offscale` have `git@gitea.officer.dev:plugins/<name>.git` remotes,
* because whoever set them up had a key on that host. Publishing that to a catalogue would hand every
* installer a URL that only works on the machine it was published from — the install fails at
* authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong".
* Several plugin repos have `git@gitea.officer.dev:plugins/<name>.git` remotes, because whoever set them
* up had a key on that host. Registering one of those would hand every installer a URL that only works
* on the machine it was published from — the install fails at authentication, which reads as "the plugin
* is broken" rather than "the catalogue is wrong".
*/
export function toHttpsRemote(remote: string): string | null {
const trimmed = remote.trim();
@@ -60,142 +47,51 @@ export function toHttpsRemote(remote: string): string | null {
return null;
}
const git = (dir: string, args: string[]): string | null => {
const result = Bun.spawnSync(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
if (result.exitCode !== 0) return null;
return new TextDecoder().decode(result.stdout).trim() || null;
};
/*
Which plugins directory to publish FROM.
There is one: `platform/plugins/`, holding what this machine has INSTALLED — updated by
`plugins:update`, which is a git pull, so it sits at whatever version was last installed.
Publishing from it means the catalogue advertises the version this machine happens to be running
rather than the version that was released. That is exactly what happened on 2026-08-27: 21 plugins
were tagged and the store went on showing 1.0.0.
A second directory — `officer-dev/plugins/`, every repo in the org cloned side by side — was created
to fix that, and never did: its entries were SYMLINKS into this one, so both paths read the same
working copy and published the same version. It was removed on 2026-08-28.
**The problem is still open, and a second checkout was never the answer.** Each directory here is a
real git repo with its own tags, so the released manifest can be read out of the tag itself —
`git show <newest-tag>:manifest.ts` — without caring what is checked out. That is the fix; it is not
written yet, and until it is, this script advertises what is installed.
*/
const PLUGINS_DIR = process.env.PLUGINS_DIR ?? resolve(import.meta.dir, '..', 'plugins');
async function syncOne(dir: string): Promise<Sync | null> {
const appName = basename(dir);
const manifestPath = join(dir, 'manifest.ts');
if (!existsSync(manifestPath)) {
console.warn(`[marketplace-sync] ${appName}: no manifest.ts — skipped`);
return null;
if (args[0] === '--add') {
const raw = args[1];
if (!raw) {
console.error('usage: marketplace-sync.ts --add <clone url>');
process.exit(1);
}
const remote = git(dir, ['remote', 'get-url', 'origin']);
const repoUrl = remote ? toHttpsRemote(remote) : null;
const repoUrl = toHttpsRemote(raw);
if (!repoUrl) {
console.warn(`[marketplace-sync] ${appName}: no usable https remote (${remote ?? 'none'}) — skipped`);
return null;
console.error(`[marketplace-sync] ${raw} is not a clone URL this can turn into https`);
process.exit(1);
}
const appName = repoUrl.replace(/\.git$/, '').split('/').pop() ?? '';
if (!/^[a-z0-9][a-z0-9-]{0,63}$/.test(appName)) {
console.error(`[marketplace-sync] "${appName}" is not a usable app name`);
process.exit(1);
}
const { manifest } = (await import(manifestPath)) as { manifest: Record<string, unknown> };
if (!manifest) {
console.warn(`[marketplace-sync] ${appName}: manifest.ts exports no \`manifest\` — skipped`);
return null;
// Registered with a placeholder developer, then immediately synced — which reads the real publisher
// out of the released manifest and overwrites it. A repository with no tag stays registered and
// unpublished, which is the correct state for one that has not made a release yet.
await upsertPlugin({ appName, developer: 'unknown', repoUrl });
console.log(`[marketplace-sync] registered ${appName} → ${repoUrl}`);
const one = await syncPlugin(appName);
report([one]);
process.exit(one.status === 'failed' ? 1 : 0);
}
const results = args[0] ? [await syncPlugin(args[0])] : await syncAll();
report(results);
process.exit(results.some((r) => r.status === 'failed') ? 1 : 0);
function report(results: Awaited<ReturnType<typeof syncAll>>): void {
for (const r of results) {
if (r.status === 'synced') console.log(`[marketplace-sync] ${r.appName} ← ${r.tag}`);
else if (r.status === 'unreleased') console.warn(`[marketplace-sync] ${r.appName}: no tag — nothing published`);
else console.error(`[marketplace-sync] ${r.appName} failed: ${r.error}`);
}
// What the plugin SHIPS, read from the directory rather than declared — the same rule discovery
// follows, so a catalogue entry cannot claim a sidecar the repository does not have.
const has = {
api: existsSync(join(dir, 'api', 'router.ts')),
schema: existsSync(join(dir, 'db', 'schema.ts')),
sidecar: existsSync(join(dir, 'sidecar', 'index.ts')) || existsSync(join(dir, 'sidecar', 'index.mjs')),
web: existsSync(join(dir, 'web')),
};
const synced = results.filter((r) => r.status === 'synced').length;
const unreleased = results.filter((r) => r.status === 'unreleased').map((r) => r.appName);
const failed = results.filter((r) => r.status === 'failed').map((r) => r.appName);
const readmePath = ['README.md', 'readme.md'].map((n) => join(dir, n)).find((p) => existsSync(p));
const readme = readmePath ? readFileSync(readmePath, 'utf-8') : null;
const developer = String(manifest.publisher ?? 'unknown');
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
await upsertMetadata(pluginId, {
version: String(manifest.version ?? '0.0.0'),
platform: String(manifest.platform ?? '>=1.0.0'),
label: String(manifest.label ?? appName),
summary: String(manifest.summary ?? ''),
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
// quietly disagreeing with the repository it claims to mirror.
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
color: (manifest.color as string) ?? null,
tile: (manifest.tile as string) ?? null,
// Artwork lives in the repo, so the URL is built from the remote rather than stored by hand. Only
// claimed when the file is actually there — a broken image is worse than a glyph.
iconUrl: existsSync(join(dir, 'assets', 'icon.png')) ? rawUrl(repoUrl, 'assets/icon.png') : null,
icon: (manifest.icon as string) ?? null,
permissions: manifest.permissions ?? [],
osDependencies: manifest.osDependencies ?? null,
has,
readme,
sourceCommit: git(dir, ['rev-parse', 'HEAD']),
});
console.log(
`[marketplace-sync] ${appName} ← ${repoUrl}${readme ? '' : ' (no README)'}${has.web ? '' : ' (no web/)'}`,
);
return { appName, developer, repoUrl };
console.log(`[marketplace-sync] ${synced}/${results.length} published from a release`);
// Named rather than counted. "3 unreleased" is a number nobody acts on; the names are a to-do list,
// and every one of them is a plugin whose store entry is frozen until somebody tags it.
if (unreleased.length) console.warn(`[marketplace-sync] never tagged, so not updated: ${unreleased.join(', ')}`);
if (failed.length) console.error(`[marketplace-sync] failed: ${failed.join(', ')}`);
}
/**
* A raw-file URL for a Gitea repository.
*
* `HEAD`, not a branch name. This said `branch/main` and the comment here warned that "a repo whose
* default branch is `master` gets a 404" — which then happened: four plugin repositories were created
* on `master` and every one of them served a broken image in the marketplace on any server that reads
* the remote catalogue. A warning in a comment is not a guard.
*
* Gitea resolves `HEAD` to whatever the default branch is, so the question stops being asked. Verified
* against both spellings.
*
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
* be part of the registry rather than inferred from the URL.
*/
function rawUrl(repoUrl: string, path: string): string {
return `${repoUrl.replace(/\.git$/, '')}/raw/HEAD/${path}`;
}
const dirs = existsSync(PLUGINS_DIR)
? [...new Bun.Glob('*/').scanSync({ cwd: PLUGINS_DIR, onlyFiles: false })]
// Not a plugin: the shared install target for plugin npm packages, which lives here because bun
// resolves node_modules up the tree. `discoverPlugins` skips it for the same reason.
.filter((d) => d.replace(/\/$/, '') !== 'node_modules')
.map((d) => join(PLUGINS_DIR, d))
: [];
if (dirs.length === 0) {
console.log(`[marketplace-sync] no plugin directories under ${PLUGINS_DIR} — nothing to sync`);
process.exit(0);
}
// Discovery is not used to FIND them — a plugin that fails discovery is exactly the kind of thing a
// catalogue should still be able to describe — but its broken list is worth surfacing here.
const { broken } = await discoverPlugins().catch(() => ({ broken: [] as { appName: string; error: string }[] }));
for (const { appName, error } of broken) console.warn(`[marketplace-sync] ${appName} is broken on disk: ${error}`);
const synced: Sync[] = [];
for (const dir of dirs) {
const result = await syncOne(dir).catch((err) => {
console.error(`[marketplace-sync] ${basename(dir)} failed: ${err instanceof Error ? err.message : err}`);
return null;
});
if (result) synced.push(result);
}
console.log(`[marketplace-sync] ${synced.length}/${dirs.length} plugin(s) in the catalogue`);
process.exit(0);