publishing a plugin is pushing a tag

Three mechanisms where there should be one. The catalogue was filled by hand, and each host separately
polled twenty-one repositories with git to work out whether a release existed — a second source of
truth for a fact one place already knew. Shipping photos 2026.09.03.02 correctly and watching alpha
not notice for ten minutes is what started this.

The timer. scripts/marketplace-cron.sh, every minute, on the machine that publishes. Safe now only
because the previous commit made the sync read tags instead of working trees; over the old one this
would have published somebody's afternoon on a schedule. Nothing is written unless the release commit
differs from the one already published, so a tick with nothing to do is one query, one fetch per repo
and no writes — about 2.7s. One timestamped summary line per tick regardless, because a fresh timestamp
is the only thing that separates "nothing to publish" from "the timer died"; a named line only for a
plugin that moved.

The badge. `updateable` now compares the tag at HEAD with the tag the catalogue says is newest —
difference, not ordering, since parsing 2026.08.27.01 works right up until somebody tags v2 and then
fails silently. refreshRemote() and its two-minute window are gone with the polling they existed to
paper over. The git rule stays as the fallback for a plugin the store cannot speak for: an unregistered
checkout, or an unreachable store. That case now reads whatever refs the last pull left, which is the
accepted cost of not making every machine poll every repository forever.

fetchCatalogue()'s cache went from five minutes to one. It only drew tiles before; now the wait for a
release to appear would have been the sync's minute plus its five.

Also: the Update button names its destination whenever a release is known, not only on `new-release`.
An untagged checkout reports `untagged` and used to lose the tag with it, so the button said "Update"
and the owner had to guess where it landed — the marketplace knows, even for a checkout that has never
fetched it.

Verified: forced a stale row, the timer republished it inside 60s and logged one line. Badge decisions
checked against the live catalogue for seven plugins, including the two shapes that matter — at a tag
the store agrees with, and untagged with a release known only to the store. 9 release tests pass, 100%
of release.ts covered, four of them new and all against real repositories.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-03 13:13:43 +01:00
co-authored by Claude Opus 5
parent 78e89ebcec
commit 19bbbab907
14 changed files with 360 additions and 119 deletions
@@ -115,6 +115,14 @@ export async function catalogueEntry(appName: string): Promise<CatalogueEntry |
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
}
/**
* A registry row, plus the one piece of metadata the sync needs before deciding to do any work.
*
* `sourceCommit` is what the last sync published, and it is here rather than fetched per plugin so a
* run that has nothing to do costs one query instead of twenty-five. Null means never synced.
*/
export type RegistryRow = { id: number; appName: string; repoUrl: string; sourceCommit: string | null };
/**
* Every registered plugin — identity and clone URL, no metadata.
*
@@ -124,10 +132,16 @@ export async function catalogueEntry(appName: string): Promise<CatalogueEntry |
* same as no longer tracking it, and one taken down for a bad release has to be able to come back
* current.
*/
export async function listRegistry(): Promise<{ id: number; appName: string; repoUrl: string }[]> {
export async function listRegistry(): Promise<RegistryRow[]> {
return marketplaceDb
.select({ id: plugins.id, appName: plugins.appName, repoUrl: plugins.repoUrl })
.select({
id: plugins.id,
appName: plugins.appName,
repoUrl: plugins.repoUrl,
sourceCommit: pluginMetadata.sourceCommit,
})
.from(plugins)
.leftJoin(pluginMetadata, eq(pluginMetadata.pluginId, plugins.id))
.orderBy(plugins.appName);
}
+4 -2
View File
@@ -4,7 +4,7 @@ import { isSuperAdmin } from '../../super-admin';
import { mountPrefix } from '../../plugins/manifest';
import { snapshotPlugins } from '../../plugins/mount';
import { manualInstallHint, reportDependencies } from '../../plugins/os-deps';
import { fetchCatalogue, fetchReadme } from '../../plugins/marketplace';
import { fetchCatalogue, fetchReadme, publishedTags } from '../../plugins/marketplace';
import { findOrphanTables } from '../../plugins/orphans';
import { fetchPlugin } from '../../plugins/fetch';
import { discoverPlugins } from '../../plugins/discover';
@@ -57,7 +57,9 @@ pluginsRouter.get('/catalogue/:appName/readme', async (ctx) => {
});
pluginsRouter.get('/', async (ctx) => {
const { states, broken } = await snapshotPlugins();
// The catalogue, for the update badge — cached, never throws, and falls back to the git rule per
// plugin when the store has nothing to say. See `plugins/release.ts`.
const { states, broken } = await snapshotPlugins(await publishedTags());
// Asked per plugin rather than once, because `pm2 jlist` is a fork and most plugins have no sidecar to
// ask about. A plugin that is installed and enabled but whose process is not online is the state worth
// rendering differently — it is the difference between "off" and "broken".
+8 -3
View File
@@ -18,9 +18,14 @@ import { catalogueEntry, listCatalogue } from '../../databases/marketplace_db/sr
*
* ── Read-only, and structurally so ──
*
* There are two GET routes and no others. Registering a plugin is `scripts/marketplace-sync.ts`, run by
* hand against the database. That is not a temporary shortcut standing in for a write API: while
* submissions do not exist, a public write surface is all risk and no feature.
* There are two GET routes and no others. Everything that WRITES the catalogue — registering a plugin,
* publishing a release — is `scripts/marketplace-sync.ts` against the database, on the one machine that
* publishes. That is not a temporary shortcut standing in for a write API: while submissions do not
* exist, a public write surface is all risk and no feature.
*
* It stayed that way when publishing went onto a timer, which is the moment it would have been easy to
* add a `POST /api/sync` for a Gitea webhook. A minute of delay is cheaper than a write route on the one
* process here that has no login.
*
* ── CORS is wide open, deliberately ──
*
+19 -5
View File
@@ -3,6 +3,7 @@ import { homedir, tmpdir } from 'node:os';
import { join } from 'node:path';
import {
listRegistry,
type RegistryRow,
upsertMetadata,
upsertPlugin,
} from '../../databases/marketplace_db/src/plugins/queries';
@@ -61,6 +62,8 @@ import {
export type SyncOutcome =
| { appName: string; status: 'synced'; tag: string; version: string }
/** The release on the remote is the one already published. Nothing was written. */
| { appName: string; status: 'unchanged'; tag: string }
| { appName: string; status: 'unreleased' }
| { appName: string; status: 'failed'; error: string };
@@ -187,7 +190,14 @@ function rawUrl(repoUrl: string, tag: string, path: string): string {
return `${repoUrl.replace(/\.git$/, '')}/raw/tag/${encodeURIComponent(tag)}/${path}`;
}
async function syncRegistered(row: { id: number; appName: string; repoUrl: string }): Promise<SyncOutcome> {
/*
`force` re-publishes a release that is already published.
Needed because "unchanged" is judged on the plugin's commit, and that is only half the input: the other
half is THIS file. Change how a field is derived — as the switch to tag-pinned icon URLs did — and every
row is stale while every commit still matches. Then the run to fix them all is `--force`.
*/
async function syncRegistered(row: RegistryRow, force: boolean): Promise<SyncOutcome> {
const { appName, repoUrl } = row;
const dir = await mirror(appName, repoUrl);
@@ -198,6 +208,10 @@ async function syncRegistered(row: { id: number; appName: string; repoUrl: strin
if (!commitAt.ok) throw new Error(`cannot resolve ${tag}`);
const commit = commitAt.out;
// Before the manifest is read, not after: this is what makes a per-minute run affordable. Twenty-five
// repositories with nothing new cost one query, twenty-five fetches of nothing, and no writes at all.
if (!force && row.sourceCommit === commit) return { appName, status: 'unchanged', tag };
const manifest = await manifestAt(dir, tag, appName, commit);
// What the plugin SHIPS, read from the tree at the release rather than from a declaration — the same
@@ -251,10 +265,10 @@ async function syncRegistered(row: { id: number; appName: string; repoUrl: strin
}
/** One plugin by app name. Unknown to the registry is a failure, not a silent no-op. */
export async function syncPlugin(appName: string): Promise<SyncOutcome> {
export async function syncPlugin(appName: string, force = false): Promise<SyncOutcome> {
const row = (await listRegistry()).find((r) => r.appName === appName);
if (!row) return { appName, status: 'failed', error: 'not in the registry' };
return syncRegistered(row).catch((err) => ({
return syncRegistered(row, force).catch((err) => ({
appName,
status: 'failed' as const,
error: err instanceof Error ? err.message : String(err),
@@ -271,12 +285,12 @@ export async function syncPlugin(appName: string): Promise<SyncOutcome> {
* and it has to be able to come back with current metadata rather than with whatever was true when it
* was pulled.
*/
export async function syncAll(): Promise<SyncOutcome[]> {
export async function syncAll(force = false): Promise<SyncOutcome[]> {
const rows = await listRegistry();
const out: SyncOutcome[] = [];
for (const row of rows) {
out.push(
await syncRegistered(row).catch((err) => ({
await syncRegistered(row, force).catch((err) => ({
appName: row.appName,
status: 'failed' as const,
error: err instanceof Error ? err.message : String(err),
+34 -1
View File
@@ -39,6 +39,15 @@ export type CatalogueEntry = {
appName: string;
publisher: string;
version: string;
/**
* The git tag this entry was published from — the newest RELEASE, as opposed to whatever a manifest
* claims about itself.
*
* Optional, and absent means "cannot tell" rather than "no release": an older catalogue does not send
* the field at all, and a plugin that has never been tagged has nothing to put in it. Both send the
* badge back to the git rule.
*/
releaseTag?: string;
platform: string;
label: string;
summary: string;
@@ -556,7 +565,15 @@ const MARKETPLACE_URL = process.env.MARKETPLACE_URL ?? 'https://marketplace.offi
* plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a
* restart is always a fresh look.
*/
const CACHE_TTL_MS = 5 * 60 * 1000;
/*
One minute, and the number changed when the badge started depending on it.
Five was fine while this only drew the store's tiles; it is not fine now that "is there an update" is
answered from the catalogue, because the wait for a release to appear became the sync's minute PLUS
this one. The cost is one small request per process per minute, and only while somebody is looking at
a page that lists plugins.
*/
const CACHE_TTL_MS = 60 * 1000;
/**
* How long to wait for the store before giving up on it.
@@ -639,6 +656,22 @@ export async function fetchCatalogue(): Promise<CatalogueEntry[]> {
}
}
/**
* What the store says the newest release of each plugin is, keyed by app name.
*
* This is the fact the update badge is built on. It replaces every host in the fleet running `git fetch`
* against twenty-one repositories to re-derive something the catalogue already knows — see
* `plugins/release.ts` for the rule and for what happens to a plugin that has no entry here.
*
* A missing key and a null value mean the same thing to a caller and are kept apart for the reader: no
* entry at all (unregistered, or a store that could not be reached) versus an entry with no release yet.
* Both fall through to the git rule, which is the only honest answer when the store cannot speak.
*/
export async function publishedTags(): Promise<Map<string, string | null>> {
const entries = await fetchCatalogue().catch(() => []);
return new Map(entries.map((e) => [e.appName, e.releaseTag ?? null]));
}
/** One entry by app name, or null. Used by install to find where to clone from. */
export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
const all = await fetchCatalogue().catch(() => BUNDLED);
+8 -2
View File
@@ -59,7 +59,13 @@ export type PluginsSnapshot = {
* the tree while the row stayed, and there is nothing to mount, describe or offer. The row is left in the
* database on purpose — deleting it here would turn "somebody moved the checkout" into silent data loss.
*/
export async function snapshotPlugins(): Promise<PluginsSnapshot> {
/**
* @param published Newest released tag per plugin, from the marketplace — `publishedTags()`. Passed by
* the one caller that renders update badges, and by nothing else on purpose: every other caller wants
* "what is on disk", and boot is among them. Making this function fetch the catalogue itself would put
* a network request with a timeout in front of mounting plugins on a machine with no internet.
*/
export async function snapshotPlugins(published?: Map<string, string | null>): Promise<PluginsSnapshot> {
const [{ plugins, broken }, installs] = await Promise.all([discoverPlugins(), listPluginInstalls()]);
const byName = new Map(installs.map((row) => [row.appName, row]));
@@ -72,7 +78,7 @@ export async function snapshotPlugins(): Promise<PluginsSnapshot> {
plugin,
install,
outdated: !!install && install.version !== plugin.manifest.version,
release: await releaseState(plugin.dir),
release: await releaseState(plugin.dir, published?.get(plugin.appName) ?? null),
};
}),
);
+55
View File
@@ -94,6 +94,61 @@ describe('releaseState', () => {
expect(s.updateable).toBe(true);
});
/*
The catalogue path. These are the ones that matter now: a host no longer fetches to find out what the
newest release is, so every assertion below is about the store's tag winning over local refs — and
about the fallback still being there when it has nothing to say.
*/
test('the published tag decides, without the checkout having ever fetched it', async () => {
const { work, origin, clone } = pair('published-newer');
// The clone is at v1 and has NEVER seen v2 — no fetch, no remote ref for it. Exactly the state a
// host is in a minute after somebody else tagged a release.
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
const s = await releaseState(clone, 'v2');
expect(s.tagAtHead).toBe('v1');
expect(s.newestTag).toBe('v2');
expect(s.updateable).toBe(true);
expect(s.reason).toBe('new-release');
});
test('the published tag matching HEAD is up to date', async () => {
const { work, origin, clone } = pair('published-same');
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
const s = await releaseState(clone, 'v1');
expect(s.updateable).toBe(false);
expect(s.reason).toBe('at-latest');
});
test('an untagged checkout is updateable AND says what it would land on', async () => {
const { clone } = pair('published-untagged');
const s = await releaseState(clone, 'v9');
expect(s.tagAtHead).toBeNull();
// The old code returned null here, so the button read "Update" with nothing after it.
expect(s.newestTag).toBe('v9');
expect(s.updateable).toBe(true);
expect(s.reason).toBe('untagged');
});
test('no published tag falls back to the git rule rather than to a guess', async () => {
const { work, origin, clone } = pair('no-entry');
publish(work, origin, 'two', 'v1');
git(clone, 'fetch', '-q', '--tags', 'origin');
git(clone, 'checkout', '-q', 'v1');
// Null, as it is for a plugin nobody registered or a store that could not be reached.
const s = await releaseState(clone, null);
expect(s.tagAtHead).toBe('v1');
expect(s.newestTag).toBe('v1');
expect(s.updateable).toBe(false);
});
test('a directory that is not a checkout is not updateable', async () => {
const d = join(root, 'plain');
mkdirSync(d, { recursive: true });
+39 -65
View File
@@ -29,18 +29,32 @@ import { join } from 'node:path';
* is that tag, this checkout is at the latest release. If it is a different one, the remote has moved on.
* That holds for any tag names anyone ever chooses, including ones that do not sort.
*
* ── The answer is never awaited on the network ──
* ── Where "the newest tag" comes from: the catalogue, not this machine's refs ──
*
* Every answer comes from refs already on disk. This is called once per plugin on a request that renders
* a list, and twenty-one network round trips would make the plugins screen feel broken.
* Every answer here is local, and this is called once per plugin on a request that renders a list —
* twenty-one network round trips would make the plugins screen feel broken. That left one hard problem:
* refs on disk go stale. Tags arrived at clone and at the fetch `pullPlugin` does, so you learned an
* update existed BY UPDATING. A machine that had not touched a plugin since August compared this
* morning's release against a snapshot from before it existed and said, truthfully and uselessly, that
* it was at the latest.
*
* But refs on disk go stale, and for a long time nothing refreshed them: tags arrived at clone and at
* the fetch `pullPlugin` does, so you only learned an update existed BY UPDATING. A machine that had not
* updated a plugin since August compared this morning's release against a snapshot from before it
* existed and said, truthfully and uselessly, that it was at the latest.
* That was patched with a rate-limited background `git fetch` per checkout, which worked and was the
* wrong shape: every host in the fleet polling twenty-one repositories to re-derive a fact one place
* already knows. The marketplace publishes releases now — see `servers/marketplace/sync.ts` — so the
* newest release is simply asked of the catalogue, which every host already fetches for the store.
*
* So `refreshRemote` runs a fetch in the background and this request answers from what is there now. The
* NEXT read is current. Rate-limited per checkout and run one at a time — see the note on it.
* published tag (catalogue) vs tag at HEAD (this checkout) → differ, so there is an update
*
* DIFFERENCE, not ordering. `2026.08.27.01` parses fine until somebody tags `v2`, and then it fails
* silently. Two strings that are not equal is a comparison that cannot rot.
*
* ── The git rule stays, for a plugin the store cannot speak for ──
*
* A checkout of something unpublished — a plugin being written, one never registered — has no catalogue
* entry, and the fallback below is the old rule against local remote-tracking refs. Those refs are no
* longer refreshed here, so for that case this can miss a tag until something else fetches. That is the
* accepted cost: the alternative was every machine polling every repository forever to cover a case that
* only arises on a developer's own box, where `git fetch` is a thing they were about to run anyway.
*/
export type ReleaseState = {
@@ -53,65 +67,18 @@ export type ReleaseState = {
reason: 'untagged' | 'new-release' | 'at-latest' | 'not-a-checkout';
};
/**
* A background `git fetch --tags`, at most one per checkout per REFRESH_MS.
*
* ── Not awaited, on purpose ──
*
* The caller is rendering a list. Waiting would put the network on the critical path of a page that has
* to feel instant, which is the reason this file never fetched in the first place. The request answers
* from the refs it has; the fetch it starts is for the next one.
*
* The cost of that is one stale view after a release, corrected by a refresh. The alternative — never
* fetching — is a badge that is wrong until somebody updates for an unrelated reason.
*
* ── One at a time ──
*
* Twenty-one plugins would otherwise spawn twenty-one gits at once, every ten minutes, on machines that
* are not all fast. Chained rather than pooled because there is no hurry: nothing is waiting on it.
*
* Failure is silence. A checkout with no remote, an unreachable host, a repository that has moved — none
* of those should surface as anything, because the list still has a true answer without them.
*/
/*
Two minutes, and the number is chosen for how releases are actually watched.
It was ten. The way this gets used is: ship a plugin, open the page, look for the badge — and ten
minutes of that is indistinguishable from the feature not working. A release tagged at 12:42 was
invisible at 12:45 because the last fetch happened at 12:37, which is correct behaviour and no comfort.
The cost is bounded by the page being open: nothing fetches when nobody is looking, and the fetches are
chained, so the worst case is one small round trip per plugin every two minutes while somebody watches
the plugins screen.
*/
const REFRESH_MS = 2 * 60 * 1000;
const fetchedAt = new Map<string, number>();
let queue: Promise<unknown> = Promise.resolve();
export function refreshRemote(dir: string): void {
const last = fetchedAt.get(dir) ?? 0;
if (Date.now() - last < REFRESH_MS) return;
// Stamped BEFORE the fetch runs, so a slow one cannot be started again by the next twenty requests.
fetchedAt.set(dir, Date.now());
queue = queue.then(async () => {
const proc = Bun.spawn(['git', '-C', dir, 'fetch', '--tags', '--quiet', 'origin'], {
stdout: 'ignore',
stderr: 'ignore',
// A host that hangs must not hold the queue: every plugin behind it would stop refreshing too.
signal: AbortSignal.timeout(20_000),
});
await proc.exited.catch(() => {});
});
}
const run = async (dir: string, args: string[]): Promise<string | null> => {
const proc = Bun.spawn(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
const out = await new Response(proc.stdout).text();
return (await proc.exited) === 0 ? out.trim() || null : null;
};
export async function releaseState(dir: string): Promise<ReleaseState> {
/**
* @param publishedTag The tag the marketplace says is this plugin's newest release, or null when the
* catalogue has nothing to say — an unregistered plugin, one never tagged, or a store that could not
* be reached. Null falls through to the git rule rather than to a guess.
*/
export async function releaseState(dir: string, publishedTag: string | null = null): Promise<ReleaseState> {
const none = { tagAtHead: null, newestTag: null };
if (!existsSync(join(dir, '.git'))) {
// Copied in by hand or bundled. Nothing to compare, and nothing to pull either — `pullPlugin` says
@@ -119,12 +86,19 @@ export async function releaseState(dir: string): Promise<ReleaseState> {
return { ...none, updateable: false, reason: 'not-a-checkout' };
}
// Started, not awaited: this answer comes from the refs on disk and the fetch is for the next one.
refreshRemote(dir);
// `--exact-match`, so this is "the tag AT this commit", not "the last tag before it". Fails, and
// therefore returns null, when HEAD is between releases — which is the first half of the rule.
const tagAtHead = await run(dir, ['describe', '--exact-match', '--tags', 'HEAD']);
if (publishedTag) {
// Reported even when HEAD is untagged, so the UI can say what pressing Update lands on. The old
// code returned a bare null here and the button read "Update" with nothing after it.
if (!tagAtHead) return { tagAtHead: null, newestTag: publishedTag, updateable: true, reason: 'untagged' };
return publishedTag === tagAtHead
? { tagAtHead, newestTag: publishedTag, updateable: false, reason: 'at-latest' }
: { tagAtHead, newestTag: publishedTag, updateable: true, reason: 'new-release' };
}
if (!tagAtHead) return { ...none, updateable: true, reason: 'untagged' };
// Asked of the REMOTE branch, not of HEAD: a tag pushed after this checkout was made is not reachable
+5 -1
View File
@@ -1,4 +1,5 @@
import { existsSync, readdirSync } from 'node:fs';
import { publishedTags } from './marketplace';
import { releaseState } from './release';
import { join } from 'node:path';
import { PLUGINS_DIR } from './discover';
@@ -144,7 +145,10 @@ export async function updatePlugin(
Checked BEFORE the pull, because a pull is what would make the answer stale.
*/
const state = await releaseState(join(PLUGINS_DIR, appName));
// Against the CATALOGUE, like the badge. Without it this asked local remote-tracking refs, which are
// no longer refreshed in the background — so a plugin at the latest tag it knows about would have
// short-circuited here and told the owner it was up to date without checking anything.
const state = await releaseState(join(PLUGINS_DIR, appName), (await publishedTags()).get(appName) ?? null);
if (!state.updateable && state.reason === 'at-latest') {
steps.push(`already at ${state.tagAtHead}, the newest release`);
onStep?.(steps[steps.length - 1]!);
@@ -534,9 +534,11 @@ export const PluginDetail = () => {
how you teach someone to stop pressing it. */}
{plugin.updateable && (
<Button disabled={busy} onClick={() => run('update', plugin.appName)}>
{plugin.releaseReason === 'new-release' && plugin.newestTag
? `Update to ${plugin.newestTag}`
: 'Update'}
{/* Named whenever a release is known, not only on `new-release`. An untagged checkout
reports `untagged`, and it used to lose the destination with it — the button said
"Update" and the owner had to guess where it landed. The marketplace knows the newest
release even for a checkout that has never fetched it. */}
{plugin.newestTag ? `Update to ${plugin.newestTag}` : 'Update'}
</Button>
)}
{/* Said out loud rather than shown as an absent button. "No Update button" and "up to date"