Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
19bbbab907 | ||
|
|
78e89ebcec |
@@ -29,17 +29,21 @@ Ranked by what actually blocks something.
|
|||||||
|
|
||||||
## Staleness
|
## Staleness
|
||||||
|
|
||||||
- [ ] **Nothing refreshes the catalogue automatically.** Publishing a plugin update changes nothing until
|
- [x] **Nothing refreshes the catalogue automatically.** Done 2026-09-03: `scripts/marketplace-cron.sh`
|
||||||
somebody runs `bun --env-file=.env scripts/marketplace-sync.ts`. A Gitea webhook or a cron is the
|
every minute. A Gitea webhook would make it a second instead, at the price of a public write route
|
||||||
obvious next step.
|
on the one process here with no login — not worth it for a minute.
|
||||||
|
|
||||||
- [ ] **The sync reads local clones, not the remotes.** The machine running it must have every plugin
|
- [x] **The sync reads local clones, not the remotes.** Done 2026-09-03. Bare mirrors, read at the newest
|
||||||
checked out, so the catalogue describes that machine's copies. The version that clones each repo
|
tag, driven by the registry rather than by a directory listing. The catalogue no longer describes
|
||||||
into a temp directory is a small change.
|
any machine's working copies.
|
||||||
|
|
||||||
- [ ] **`rawUrl()` assumes Gitea and a `main` branch.** A GitHub-hosted plugin, or one whose default branch
|
- [ ] **`rawUrl()` assumes Gitea.** A GitHub-hosted plugin gets a 404 for its icon. The forge belongs in
|
||||||
is `master`, gets a 404 for its icon. The forge belongs in the registry rather than inferred from a
|
the registry rather than inferred from a URL shape. (The `main`-vs-`master` half of this is gone —
|
||||||
URL shape.
|
URLs pin to the release tag now.)
|
||||||
|
|
||||||
|
- [ ] **Four plugins have never been tagged**, so the store still shows rows the old sync took from a
|
||||||
|
working tree: av-club, kanban, openseo, fofos-admin. They are frozen until somebody tags them, and
|
||||||
|
the sync names them on every run somebody asks for.
|
||||||
|
|
||||||
## Scale
|
## Scale
|
||||||
|
|
||||||
|
|||||||
+82
-17
@@ -10,8 +10,9 @@ from. **The website installs nothing** — installing means cloning a repository
|
|||||||
starting a process on someone else's machine, so it happens entirely inside the owner's own Officer.
|
starting a process on someone else's machine, so it happens entirely inside the owner's own Officer.
|
||||||
|
|
||||||
```
|
```
|
||||||
gitea.officer.dev/plugins/* the repositories (source of truth)
|
gitea.officer.dev/** the repositories (source of truth)
|
||||||
│ scripts/marketplace-sync.ts (manual, reads manifest.ts + README.md)
|
│ bare mirrors, read AT THE NEWEST TAG — src/servers/marketplace/sync.ts
|
||||||
|
│ every minute, scripts/marketplace-cron.sh
|
||||||
▼
|
▼
|
||||||
officer_marketplace (postgres) plugins + plugin_metadata
|
officer_marketplace (postgres) plugins + plugin_metadata
|
||||||
│ src/servers/marketplace/server.tsx :9100
|
│ src/servers/marketplace/server.tsx :9100
|
||||||
@@ -20,6 +21,7 @@ marketplace.officer.dev GET /api/plugins, /api/plugins/:appName
|
|||||||
│ fetchCatalogue() live → cached → bundled
|
│ fetchCatalogue() live → cached → bundled
|
||||||
▼
|
▼
|
||||||
an Officer install /marketplace → clone → push schema → mount
|
an Officer install /marketplace → clone → push schema → mount
|
||||||
|
and the update badge — see "Publishing" below
|
||||||
```
|
```
|
||||||
|
|
||||||
## The parts
|
## The parts
|
||||||
@@ -78,19 +80,87 @@ disappearing from the store.
|
|||||||
the clone, the plugin's own manifest is what the platform reads. A metadata row that disagrees with the
|
the clone, the plugin's own manifest is what the platform reads. A metadata row that disagrees with the
|
||||||
repository is a marketplace bug, never a second source of truth.
|
repository is a marketplace bug, never a second source of truth.
|
||||||
|
|
||||||
## Sync
|
## Publishing
|
||||||
|
|
||||||
|
**Push a tag. That is the whole procedure.** A timer on officer-dev picks it up within a minute and every
|
||||||
|
Officer install sees it on the next page load.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bun --env-file=.env scripts/marketplace-sync.ts
|
bun --env-file=.env scripts/marketplace-sync.ts # all of them, now
|
||||||
|
bun --env-file=.env scripts/marketplace-sync.ts photos # one
|
||||||
|
bun --env-file=.env scripts/marketplace-sync.ts --add https://gitea.officer.dev/plugins/finances.git
|
||||||
|
bun --env-file=.env scripts/marketplace-sync.ts --force # re-publish rows this file's own logic changed
|
||||||
```
|
```
|
||||||
|
|
||||||
Reads every directory under `plugins/`, imports its `manifest.ts`, derives `has` from the directory
|
### It reads releases, never a working tree
|
||||||
rather than from a declaration, reads `README.md`, and upserts. Re-running is always safe: it overwrites,
|
|
||||||
never merges.
|
|
||||||
|
|
||||||
It rewrites SSH remotes to HTTPS. `example` and `offscale` had `git@gitea…` origins, which would have
|
Every registered plugin is mirrored bare into `~/.cache/officer-marketplace/mirrors` and read at its
|
||||||
handed every installer a URL that only works on the machine that published it — the install fails at
|
newest tag — manifest, README and file list all out of `git cat-file`. There is no working tree in the
|
||||||
authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong".
|
path, so publishing something uncommitted is not a mistake that can be made.
|
||||||
|
|
||||||
|
That is a change, and it was load-bearing. Until 2026-09-03 the sync imported `manifest.ts` off
|
||||||
|
`platform/plugins/` — the directories THIS machine has installed — so publishing meant "get officer-dev's
|
||||||
|
checkout to the release, then run the script", and nothing showed when that had not happened. On the day
|
||||||
|
it was rewritten the live store was advertising four plugins from untagged working copies: av-club
|
||||||
|
2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of which had ever
|
||||||
|
been tagged. It is also why the job could not go on a timer before: a cron over it published whatever was
|
||||||
|
in a working tree, on a schedule.
|
||||||
|
|
||||||
|
### The registry decides which plugins, and where from
|
||||||
|
|
||||||
|
The list comes from the `plugins` table — the half this document has always called authoritative, and
|
||||||
|
which was in fact a by-product of what happened to be checked out. Now it decides. A plugin nobody has
|
||||||
|
installed here can still be published, and a plugin lives wherever its row says: `fofos-admin` is under
|
||||||
|
`fofos/`, not `plugins/`, which an org listing would have missed. Registering one is `--add`.
|
||||||
|
|
||||||
|
### A plugin with no tag is left alone
|
||||||
|
|
||||||
|
It has never made a release, so the sync has nothing true to say about it: the existing row stays and the
|
||||||
|
run names it. Deleting it would drop somebody's plugin out of the store on the day the rule changed.
|
||||||
|
Tagging it is the fix, and the report is the reminder.
|
||||||
|
|
||||||
|
### `release_tag`, and why it is not `version`
|
||||||
|
|
||||||
|
`version` is the manifest's own claim; `release_tag` is what was pushed. They match every time somebody
|
||||||
|
tags correctly, and when they do not, the tag is the fact — a manifest whose version nobody bumped would
|
||||||
|
otherwise make a real release invisible to anything comparing versions. The sync warns on a mismatch.
|
||||||
|
|
||||||
|
Icon URLs pin to the tag too. `/raw/HEAD/…` served whatever was on the default branch, via a 303, so a
|
||||||
|
plugin could show one icon in the store and ship another.
|
||||||
|
|
||||||
|
### The timer
|
||||||
|
|
||||||
|
`scripts/marketplace-cron.sh`, `* * * * *`, logging to `officer-dev/marketplace-sync.log`. One
|
||||||
|
timestamped summary line per tick — always, including the ticks that do nothing, because a fresh
|
||||||
|
timestamp is the only thing that distinguishes "nothing to publish" from "the timer died". A plugin-level
|
||||||
|
line appears only when one actually moved.
|
||||||
|
|
||||||
|
A tick costs about 2.7s with nothing to do: one query, one fetch per repository, no writes. Nothing is
|
||||||
|
written unless the release commit differs from the one already published.
|
||||||
|
|
||||||
|
A Gitea webhook would cut the delay from a minute to a second, at the price of a public write route and a
|
||||||
|
shared secret. A minute was never the problem anybody had — the complaint that started this was a release
|
||||||
|
that never appeared at all.
|
||||||
|
|
||||||
|
### Remotes
|
||||||
|
|
||||||
|
`--add` rewrites SSH to HTTPS. A `git@gitea…` origin in the catalogue hands every installer a URL that
|
||||||
|
only works on the machine that published it, and the install fails at authentication — which reads as
|
||||||
|
"the plugin is broken" rather than "the catalogue is wrong".
|
||||||
|
|
||||||
|
## The update badge reads the catalogue
|
||||||
|
|
||||||
|
`updateable` on `/api/plugins` compares the tag at the checkout's HEAD with the tag the catalogue says is
|
||||||
|
newest. Difference, not ordering: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
|
||||||
|
then it fails silently.
|
||||||
|
|
||||||
|
Before this, each host ran a rate-limited background `git fetch` against all twenty-one repositories to
|
||||||
|
re-derive a fact one place already knew. That is gone. `releaseState()` still holds the git rule as a
|
||||||
|
fallback for a plugin the store cannot speak for — an unregistered checkout, or a store that could not be
|
||||||
|
reached — and in that case its refs are whatever the last `pullPlugin` left, which is the accepted cost.
|
||||||
|
|
||||||
|
`fetchCatalogue()`'s cache dropped from five minutes to one when the badge started depending on it:
|
||||||
|
otherwise the wait for a release to appear was the sync's minute plus the cache's five.
|
||||||
|
|
||||||
## The platform degrades, it does not break
|
## The platform degrades, it does not break
|
||||||
|
|
||||||
@@ -111,13 +181,8 @@ The full list, ranked, is `docs/marketplace-todo.md`. The ones that shape the de
|
|||||||
- **`[blocker]` Third-party submissions cannot use this sync.** It reads a manifest by importing it, which
|
- **`[blocker]` Third-party submissions cannot use this sync.** It reads a manifest by importing it, which
|
||||||
is code execution. Fine for our own repositories; not a mechanism a stranger's plugin can go through.
|
is code execution. Fine for our own repositories; not a mechanism a stranger's plugin can go through.
|
||||||
Submissions need either a sandbox or a published `manifest.json` alongside the TypeScript one.
|
Submissions need either a sandbox or a published `manifest.json` alongside the TypeScript one.
|
||||||
- **`[open]` Sync reads local clones, not the remotes.** The machine running it must have the plugins
|
- **`[open]` `rawUrl()` assumes Gitea.** A GitHub-hosted plugin needs the other URL form. The forge should
|
||||||
checked out. The version that clones each repo into a temp directory is a small change and was left out
|
become part of the registry rather than inferred from the URL.
|
||||||
of the first pass.
|
|
||||||
- **`[open]` `rawUrl()` assumes Gitea and a `main` branch.** A GitHub-hosted plugin or one defaulting to
|
|
||||||
`master` gets a 404 for its icon. The forge should become part of the registry rather than inferred.
|
|
||||||
- **`[open]` No versions table.** `plugin_metadata` is one row per plugin — a snapshot, not a history. A
|
- **`[open]` No versions table.** `plugin_metadata` is one row per plugin — a snapshot, not a history. A
|
||||||
store that can offer "install 1.2 instead of 1.3" needs a different shape.
|
store that can offer "install 1.2 instead of 1.3" needs a different shape.
|
||||||
- **`[open]` Nothing refreshes automatically.** Publishing a plugin update does not change the catalogue
|
|
||||||
until somebody runs the sync. A cron or a Gitea webhook is the obvious next step.
|
|
||||||
- **`[open]` No search, no categories, no counts.** Three plugins do not need them.
|
- **`[open]` No search, no categories, no counts.** Three plugins do not need them.
|
||||||
|
|||||||
Executable
+46
@@ -0,0 +1,46 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Publish every plugin release to the marketplace catalogue. Meant for cron, on the ONE machine that
|
||||||
|
# publishes.
|
||||||
|
#
|
||||||
|
# ── Why a timer at all, when a webhook would be instant ──
|
||||||
|
#
|
||||||
|
# The sync reads bare mirrors at their newest tag, so running it on a schedule cannot publish anything
|
||||||
|
# that was not pushed — which is exactly what made a timer unsafe before, and is no longer true. A Gitea
|
||||||
|
# webhook would cut the delay from a minute to a second, and it would also mean a public write route, a
|
||||||
|
# shared secret, and a second path into the same code. A minute is not the problem anybody had: the
|
||||||
|
# complaint that started this was a release that never appeared at all.
|
||||||
|
#
|
||||||
|
# ── What it costs ──
|
||||||
|
#
|
||||||
|
# About 2.7s per tick with nothing to do — one query, one fetch per repository, no writes. Gitea is on
|
||||||
|
# this same machine, so none of that leaves the box.
|
||||||
|
#
|
||||||
|
# ── The log ──
|
||||||
|
#
|
||||||
|
# One timestamped summary line per tick, ALWAYS, including the ticks that do nothing. A fresh timestamp
|
||||||
|
# is the only thing that distinguishes "nothing to publish" from "the timer died". Plugin-level lines
|
||||||
|
# appear only for a plugin that actually moved, which is what `--quiet` buys.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
PLATFORM="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
LOG="${MARKETPLACE_SYNC_LOG:-$(dirname "$PLATFORM")/marketplace-sync.log}"
|
||||||
|
LOCK="/tmp/officer-marketplace-sync.lock"
|
||||||
|
|
||||||
|
# Cron's PATH is three entries and bun is in none of them.
|
||||||
|
export PATH="$HOME/.bun/bin:/usr/local/bin:/usr/bin:/bin"
|
||||||
|
|
||||||
|
cd "$PLATFORM" || exit 1
|
||||||
|
|
||||||
|
# Skip rather than queue. A tick that finds the previous one still running has nothing to add — the run
|
||||||
|
# in progress will publish whatever this one would have.
|
||||||
|
exec 9>"$LOCK"
|
||||||
|
flock -n 9 || exit 0
|
||||||
|
|
||||||
|
bun --env-file=.env scripts/marketplace-sync.ts --quiet >>"$LOG" 2>&1
|
||||||
|
|
||||||
|
# Bounded without logrotate, because this is one file on one machine and a config file for it would be
|
||||||
|
# more moving parts than the thing it manages.
|
||||||
|
if [ "$(wc -l <"$LOG")" -gt 20000 ]; then
|
||||||
|
tail -n 5000 "$LOG" >"$LOG.tmp" && mv "$LOG.tmp" "$LOG"
|
||||||
|
fi
|
||||||
+74
-161
@@ -1,43 +1,40 @@
|
|||||||
#!/usr/bin/env bun
|
#!/usr/bin/env bun
|
||||||
import { existsSync, readFileSync } from 'node:fs';
|
import { upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
|
||||||
import { basename, join, resolve } from 'node:path';
|
import { syncAll, syncPlugin } from '../src/servers/marketplace/sync';
|
||||||
import { upsertMetadata, upsertPlugin } from '../src/databases/marketplace_db/src/plugins/queries';
|
|
||||||
import { discoverPlugins } from '../src/servers/plugins/discover';
|
|
||||||
|
|
||||||
// Fill the marketplace database from the plugin repositories.
|
/**
|
||||||
//
|
* Fill the marketplace catalogue from the plugins' releases.
|
||||||
// ── What this reads, and why it is not the marketplace's own opinion ──
|
*
|
||||||
//
|
* bun --env-file=.env scripts/marketplace-sync.ts every registered plugin
|
||||||
// Every field except "where to clone from" comes out of the plugin's `manifest.ts` and its README. The
|
* bun --env-file=.env scripts/marketplace-sync.ts photos just one
|
||||||
// marketplace stores a COPY so the website can render without cloning three repositories per page load;
|
* bun --env-file=.env scripts/marketplace-sync.ts --add <url> register a new one, then sync it
|
||||||
// the manifest stays the source of truth, and the platform re-reads it after cloning. A row here that
|
* bun --env-file=.env scripts/marketplace-sync.ts --force re-publish releases already published
|
||||||
// disagrees with the repository is a stale cache, not a second opinion — which is why re-running this is
|
* bun --env-file=.env scripts/marketplace-sync.ts --quiet print only what changed
|
||||||
// always safe and never merges: it overwrites.
|
*
|
||||||
//
|
* All the reasoning is in `src/servers/marketplace/sync.ts` — this file is the command line around it.
|
||||||
// ── Reading the manifest by importing it ──
|
* It is a thin wrapper on purpose: the same functions are what the Gitea webhook calls, so a plugin
|
||||||
//
|
* published by hand and one published by pushing a tag go through exactly the same code.
|
||||||
// A manifest is TypeScript, not JSON, so it is imported and evaluated rather than parsed. That is code
|
*
|
||||||
// execution, and it is fine for OUR repositories and not fine as a general mechanism.
|
* Re-running is always safe. It overwrites and never merges.
|
||||||
//
|
*/
|
||||||
// `[blocker for third-party submissions]` A stranger's plugin cannot be imported by the machine that
|
|
||||||
// serves the catalogue. Whoever builds submissions has to either evaluate manifests in a sandbox or make
|
|
||||||
// authors publish a `manifest.json` alongside. Recorded rather than solved: nobody can submit yet.
|
|
||||||
//
|
|
||||||
// ── Local clones, for now ──
|
|
||||||
//
|
|
||||||
// It syncs from `plugins/` on this machine rather than by cloning each repo into a temp directory. That
|
|
||||||
// keeps tonight's scope to "get real data into the database" — and it means the machine running the sync
|
|
||||||
// must have the plugins checked out. See `docs/marketplace.md` → "Sync" for the version that clones.
|
|
||||||
|
|
||||||
type Sync = { appName: string; developer: string; repoUrl: string };
|
const argv = process.argv.slice(2);
|
||||||
|
/*
|
||||||
|
`--quiet` is what the timer runs: one line per plugin that actually moved, and a summary. Not silence —
|
||||||
|
the summary is always printed, because a log whose last line is old is the only way to tell "nothing to
|
||||||
|
publish" from "the timer died", and that question does come up.
|
||||||
|
*/
|
||||||
|
const quiet = argv.includes('--quiet');
|
||||||
|
const force = argv.includes('--force');
|
||||||
|
const args = argv.filter((a) => a !== '--quiet' && a !== '--force');
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A clone URL the whole internet can use.
|
* A clone URL the whole internet can use.
|
||||||
*
|
*
|
||||||
* `plugins/example` and `plugins/offscale` have `git@gitea.officer.dev:plugins/<name>.git` remotes,
|
* Several plugin repos have `git@gitea.officer.dev:plugins/<name>.git` remotes, because whoever set them
|
||||||
* because whoever set them up had a key on that host. Publishing that to a catalogue would hand every
|
* up had a key on that host. Registering one of those would hand every installer a URL that only works
|
||||||
* installer a URL that only works on the machine it was published from — the install fails at
|
* on the machine it was published from — the install fails at authentication, which reads as "the plugin
|
||||||
* authentication, which reads as "the plugin is broken" rather than "the catalogue is wrong".
|
* is broken" rather than "the catalogue is wrong".
|
||||||
*/
|
*/
|
||||||
export function toHttpsRemote(remote: string): string | null {
|
export function toHttpsRemote(remote: string): string | null {
|
||||||
const trimmed = remote.trim();
|
const trimmed = remote.trim();
|
||||||
@@ -60,142 +57,58 @@ export function toHttpsRemote(remote: string): string | null {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
const git = (dir: string, args: string[]): string | null => {
|
if (args[0] === '--add') {
|
||||||
const result = Bun.spawnSync(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
|
const raw = args[1];
|
||||||
if (result.exitCode !== 0) return null;
|
if (!raw) {
|
||||||
return new TextDecoder().decode(result.stdout).trim() || null;
|
console.error('usage: marketplace-sync.ts --add <clone url>');
|
||||||
};
|
process.exit(1);
|
||||||
|
|
||||||
/*
|
|
||||||
Which plugins directory to publish FROM.
|
|
||||||
|
|
||||||
There is one: `platform/plugins/`, holding what this machine has INSTALLED — updated by
|
|
||||||
`plugins:update`, which is a git pull, so it sits at whatever version was last installed.
|
|
||||||
|
|
||||||
Publishing from it means the catalogue advertises the version this machine happens to be running
|
|
||||||
rather than the version that was released. That is exactly what happened on 2026-08-27: 21 plugins
|
|
||||||
were tagged and the store went on showing 1.0.0.
|
|
||||||
|
|
||||||
A second directory — `officer-dev/plugins/`, every repo in the org cloned side by side — was created
|
|
||||||
to fix that, and never did: its entries were SYMLINKS into this one, so both paths read the same
|
|
||||||
working copy and published the same version. It was removed on 2026-08-28.
|
|
||||||
|
|
||||||
**The problem is still open, and a second checkout was never the answer.** Each directory here is a
|
|
||||||
real git repo with its own tags, so the released manifest can be read out of the tag itself —
|
|
||||||
`git show <newest-tag>:manifest.ts` — without caring what is checked out. That is the fix; it is not
|
|
||||||
written yet, and until it is, this script advertises what is installed.
|
|
||||||
*/
|
|
||||||
const PLUGINS_DIR = process.env.PLUGINS_DIR ?? resolve(import.meta.dir, '..', 'plugins');
|
|
||||||
|
|
||||||
async function syncOne(dir: string): Promise<Sync | null> {
|
|
||||||
const appName = basename(dir);
|
|
||||||
const manifestPath = join(dir, 'manifest.ts');
|
|
||||||
if (!existsSync(manifestPath)) {
|
|
||||||
console.warn(`[marketplace-sync] ${appName}: no manifest.ts — skipped`);
|
|
||||||
return null;
|
|
||||||
}
|
}
|
||||||
|
const repoUrl = toHttpsRemote(raw);
|
||||||
const remote = git(dir, ['remote', 'get-url', 'origin']);
|
|
||||||
const repoUrl = remote ? toHttpsRemote(remote) : null;
|
|
||||||
if (!repoUrl) {
|
if (!repoUrl) {
|
||||||
console.warn(`[marketplace-sync] ${appName}: no usable https remote (${remote ?? 'none'}) — skipped`);
|
console.error(`[marketplace-sync] ${raw} is not a clone URL this can turn into https`);
|
||||||
return null;
|
process.exit(1);
|
||||||
|
}
|
||||||
|
const appName = repoUrl.replace(/\.git$/, '').split('/').pop() ?? '';
|
||||||
|
if (!/^[a-z0-9][a-z0-9-]{0,63}$/.test(appName)) {
|
||||||
|
console.error(`[marketplace-sync] "${appName}" is not a usable app name`);
|
||||||
|
process.exit(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
const { manifest } = (await import(manifestPath)) as { manifest: Record<string, unknown> };
|
// Registered with a placeholder developer, then immediately synced — which reads the real publisher
|
||||||
if (!manifest) {
|
// out of the released manifest and overwrites it. A repository with no tag stays registered and
|
||||||
console.warn(`[marketplace-sync] ${appName}: manifest.ts exports no \`manifest\` — skipped`);
|
// unpublished, which is the correct state for one that has not made a release yet.
|
||||||
return null;
|
await upsertPlugin({ appName, developer: 'unknown', repoUrl });
|
||||||
|
console.log(`[marketplace-sync] registered ${appName} → ${repoUrl}`);
|
||||||
|
const one = await syncPlugin(appName, force);
|
||||||
|
report([one]);
|
||||||
|
process.exit(one.status === 'failed' ? 1 : 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
// What the plugin SHIPS, read from the directory rather than declared — the same rule discovery
|
const results = args[0] ? [await syncPlugin(args[0], force)] : await syncAll(force);
|
||||||
// follows, so a catalogue entry cannot claim a sidecar the repository does not have.
|
report(results);
|
||||||
const has = {
|
process.exit(results.some((r) => r.status === 'failed') ? 1 : 0);
|
||||||
api: existsSync(join(dir, 'api', 'router.ts')),
|
|
||||||
schema: existsSync(join(dir, 'db', 'schema.ts')),
|
|
||||||
sidecar: existsSync(join(dir, 'sidecar', 'index.ts')) || existsSync(join(dir, 'sidecar', 'index.mjs')),
|
|
||||||
web: existsSync(join(dir, 'web')),
|
|
||||||
};
|
|
||||||
|
|
||||||
const readmePath = ['README.md', 'readme.md'].map((n) => join(dir, n)).find((p) => existsSync(p));
|
function report(results: Awaited<ReturnType<typeof syncAll>>): void {
|
||||||
const readme = readmePath ? readFileSync(readmePath, 'utf-8') : null;
|
const stamp = new Date().toISOString();
|
||||||
|
for (const r of results) {
|
||||||
const developer = String(manifest.publisher ?? 'unknown');
|
if (r.status === 'synced') console.log(`[marketplace-sync] ${r.appName} ← ${r.tag}`);
|
||||||
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
|
else if (r.status === 'failed') console.error(`[marketplace-sync] ${r.appName} failed: ${r.error}`);
|
||||||
|
else if (quiet) continue;
|
||||||
await upsertMetadata(pluginId, {
|
else if (r.status === 'unchanged') console.log(`[marketplace-sync] ${r.appName} = ${r.tag}`);
|
||||||
version: String(manifest.version ?? '0.0.0'),
|
else console.warn(`[marketplace-sync] ${r.appName}: no tag — nothing published`);
|
||||||
platform: String(manifest.platform ?? '>=1.0.0'),
|
|
||||||
label: String(manifest.label ?? appName),
|
|
||||||
summary: String(manifest.summary ?? ''),
|
|
||||||
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
|
|
||||||
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
|
|
||||||
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
|
|
||||||
// quietly disagreeing with the repository it claims to mirror.
|
|
||||||
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
|
|
||||||
color: (manifest.color as string) ?? null,
|
|
||||||
tile: (manifest.tile as string) ?? null,
|
|
||||||
// Artwork lives in the repo, so the URL is built from the remote rather than stored by hand. Only
|
|
||||||
// claimed when the file is actually there — a broken image is worse than a glyph.
|
|
||||||
iconUrl: existsSync(join(dir, 'assets', 'icon.png')) ? rawUrl(repoUrl, 'assets/icon.png') : null,
|
|
||||||
icon: (manifest.icon as string) ?? null,
|
|
||||||
permissions: manifest.permissions ?? [],
|
|
||||||
osDependencies: manifest.osDependencies ?? null,
|
|
||||||
has,
|
|
||||||
readme,
|
|
||||||
sourceCommit: git(dir, ['rev-parse', 'HEAD']),
|
|
||||||
});
|
|
||||||
|
|
||||||
console.log(
|
|
||||||
`[marketplace-sync] ${appName} ← ${repoUrl}${readme ? '' : ' (no README)'}${has.web ? '' : ' (no web/)'}`,
|
|
||||||
);
|
|
||||||
return { appName, developer, repoUrl };
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
const synced = results.filter((r) => r.status === 'synced').length;
|
||||||
* A raw-file URL for a Gitea repository.
|
const unchanged = results.filter((r) => r.status === 'unchanged').length;
|
||||||
*
|
const unreleased = results.filter((r) => r.status === 'unreleased').map((r) => r.appName);
|
||||||
* `HEAD`, not a branch name. This said `branch/main` and the comment here warned that "a repo whose
|
const failed = results.filter((r) => r.status === 'failed').map((r) => r.appName);
|
||||||
* default branch is `master` gets a 404" — which then happened: four plugin repositories were created
|
|
||||||
* on `master` and every one of them served a broken image in the marketplace on any server that reads
|
console.log(`[marketplace-sync] ${stamp} — ${synced} published, ${unchanged} unchanged, ${unreleased.length} untagged`);
|
||||||
* the remote catalogue. A warning in a comment is not a guard.
|
// Named rather than counted, and not under `--quiet`: "4 untagged" is a number nobody acts on, but a
|
||||||
*
|
// timer that reprints the same four names every minute is a log nobody reads. The names belong in a
|
||||||
* Gitea resolves `HEAD` to whatever the default branch is, so the question stops being asked. Verified
|
// run somebody asked for.
|
||||||
* against both spellings.
|
if (unreleased.length && !quiet) {
|
||||||
*
|
console.warn(`[marketplace-sync] never tagged, so not updated: ${unreleased.join(', ')}`);
|
||||||
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
|
|
||||||
* be part of the registry rather than inferred from the URL.
|
|
||||||
*/
|
|
||||||
function rawUrl(repoUrl: string, path: string): string {
|
|
||||||
return `${repoUrl.replace(/\.git$/, '')}/raw/HEAD/${path}`;
|
|
||||||
}
|
}
|
||||||
|
if (failed.length) console.error(`[marketplace-sync] failed: ${failed.join(', ')}`);
|
||||||
const dirs = existsSync(PLUGINS_DIR)
|
|
||||||
? [...new Bun.Glob('*/').scanSync({ cwd: PLUGINS_DIR, onlyFiles: false })]
|
|
||||||
// Not a plugin: the shared install target for plugin npm packages, which lives here because bun
|
|
||||||
// resolves node_modules up the tree. `discoverPlugins` skips it for the same reason.
|
|
||||||
.filter((d) => d.replace(/\/$/, '') !== 'node_modules')
|
|
||||||
.map((d) => join(PLUGINS_DIR, d))
|
|
||||||
: [];
|
|
||||||
|
|
||||||
if (dirs.length === 0) {
|
|
||||||
console.log(`[marketplace-sync] no plugin directories under ${PLUGINS_DIR} — nothing to sync`);
|
|
||||||
process.exit(0);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Discovery is not used to FIND them — a plugin that fails discovery is exactly the kind of thing a
|
|
||||||
// catalogue should still be able to describe — but its broken list is worth surfacing here.
|
|
||||||
const { broken } = await discoverPlugins().catch(() => ({ broken: [] as { appName: string; error: string }[] }));
|
|
||||||
for (const { appName, error } of broken) console.warn(`[marketplace-sync] ${appName} is broken on disk: ${error}`);
|
|
||||||
|
|
||||||
const synced: Sync[] = [];
|
|
||||||
for (const dir of dirs) {
|
|
||||||
const result = await syncOne(dir).catch((err) => {
|
|
||||||
console.error(`[marketplace-sync] ${basename(dir)} failed: ${err instanceof Error ? err.message : err}`);
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
if (result) synced.push(result);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`[marketplace-sync] ${synced.length}/${dirs.length} plugin(s) in the catalogue`);
|
|
||||||
process.exit(0);
|
|
||||||
|
|||||||
@@ -14,6 +14,14 @@ export type CatalogueEntry = {
|
|||||||
appName: string;
|
appName: string;
|
||||||
publisher: string;
|
publisher: string;
|
||||||
version: string;
|
version: string;
|
||||||
|
/**
|
||||||
|
* The git tag this entry was published from.
|
||||||
|
*
|
||||||
|
* Optional on the wire, not in the database, and the two mean different things. Here it is absent for
|
||||||
|
* a row that predates release-based publishing OR for a catalogue too old to send the field at all —
|
||||||
|
* so a consumer must treat "no tag" as "cannot tell" and fall back to whatever it knew before.
|
||||||
|
*/
|
||||||
|
releaseTag?: string;
|
||||||
platform: string;
|
platform: string;
|
||||||
label: string;
|
label: string;
|
||||||
summary: string;
|
summary: string;
|
||||||
@@ -52,6 +60,7 @@ function toCatalogueEntry(row: Row): CatalogueEntry | null {
|
|||||||
appName: row.appName,
|
appName: row.appName,
|
||||||
publisher: row.developer,
|
publisher: row.developer,
|
||||||
version: m.version,
|
version: m.version,
|
||||||
|
...(m.releaseTag ? { releaseTag: m.releaseTag } : {}),
|
||||||
platform: m.platform,
|
platform: m.platform,
|
||||||
label: m.label,
|
label: m.label,
|
||||||
summary: m.summary,
|
summary: m.summary,
|
||||||
@@ -106,6 +115,36 @@ export async function catalogueEntry(appName: string): Promise<CatalogueEntry |
|
|||||||
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
|
return { ...entry, ...(row.metadata.readme ? { readme: row.metadata.readme } : {}) };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A registry row, plus the one piece of metadata the sync needs before deciding to do any work.
|
||||||
|
*
|
||||||
|
* `sourceCommit` is what the last sync published, and it is here rather than fetched per plugin so a
|
||||||
|
* run that has nothing to do costs one query instead of twenty-five. Null means never synced.
|
||||||
|
*/
|
||||||
|
export type RegistryRow = { id: number; appName: string; repoUrl: string; sourceCommit: string | null };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every registered plugin — identity and clone URL, no metadata.
|
||||||
|
*
|
||||||
|
* This is what the sync iterates, and it is the reason the registry is called authoritative: the
|
||||||
|
* catalogue lists what somebody deliberately registered, not what happens to be checked out on the
|
||||||
|
* machine that runs the job. Unlisted rows are included; hiding a plugin from the public API is not the
|
||||||
|
* same as no longer tracking it, and one taken down for a bad release has to be able to come back
|
||||||
|
* current.
|
||||||
|
*/
|
||||||
|
export async function listRegistry(): Promise<RegistryRow[]> {
|
||||||
|
return marketplaceDb
|
||||||
|
.select({
|
||||||
|
id: plugins.id,
|
||||||
|
appName: plugins.appName,
|
||||||
|
repoUrl: plugins.repoUrl,
|
||||||
|
sourceCommit: pluginMetadata.sourceCommit,
|
||||||
|
})
|
||||||
|
.from(plugins)
|
||||||
|
.leftJoin(pluginMetadata, eq(pluginMetadata.pluginId, plugins.id))
|
||||||
|
.orderBy(plugins.appName);
|
||||||
|
}
|
||||||
|
|
||||||
// ── Writes. Used by the sync script, never by the public API ──
|
// ── Writes. Used by the sync script, never by the public API ──
|
||||||
|
|
||||||
type RegisterParams = { appName: string; developer: string; repoUrl: string };
|
type RegisterParams = { appName: string; developer: string; repoUrl: string };
|
||||||
|
|||||||
@@ -73,6 +73,19 @@ export const pluginMetadata = pgTable(
|
|||||||
.references(() => plugins.id, { onDelete: 'cascade' }),
|
.references(() => plugins.id, { onDelete: 'cascade' }),
|
||||||
|
|
||||||
version: text('version').notNull(),
|
version: text('version').notNull(),
|
||||||
|
/**
|
||||||
|
* The git tag this row was read from — what "published" actually means.
|
||||||
|
*
|
||||||
|
* `version` above is the manifest's own claim, and the two are the same string every time somebody
|
||||||
|
* tags correctly. They are kept apart because when they DISAGREE the tag is the fact: it is what was
|
||||||
|
* pushed, and a manifest whose version nobody bumped would otherwise make a real release invisible
|
||||||
|
* to anything comparing versions.
|
||||||
|
*
|
||||||
|
* Nullable because rows existed before releases were the unit of publishing. A null one is a
|
||||||
|
* snapshot taken from a working tree by the old sync, and every consumer should read it as "unknown"
|
||||||
|
* rather than "no release".
|
||||||
|
*/
|
||||||
|
releaseTag: text('release_tag'),
|
||||||
/** The platform range it declares support for, e.g. `>=1.0.0`. */
|
/** The platform range it declares support for, e.g. `>=1.0.0`. */
|
||||||
platform: text('platform').notNull(),
|
platform: text('platform').notNull(),
|
||||||
label: text('label').notNull(),
|
label: text('label').notNull(),
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { isSuperAdmin } from '../../super-admin';
|
|||||||
import { mountPrefix } from '../../plugins/manifest';
|
import { mountPrefix } from '../../plugins/manifest';
|
||||||
import { snapshotPlugins } from '../../plugins/mount';
|
import { snapshotPlugins } from '../../plugins/mount';
|
||||||
import { manualInstallHint, reportDependencies } from '../../plugins/os-deps';
|
import { manualInstallHint, reportDependencies } from '../../plugins/os-deps';
|
||||||
import { fetchCatalogue, fetchReadme } from '../../plugins/marketplace';
|
import { fetchCatalogue, fetchReadme, publishedTags } from '../../plugins/marketplace';
|
||||||
import { findOrphanTables } from '../../plugins/orphans';
|
import { findOrphanTables } from '../../plugins/orphans';
|
||||||
import { fetchPlugin } from '../../plugins/fetch';
|
import { fetchPlugin } from '../../plugins/fetch';
|
||||||
import { discoverPlugins } from '../../plugins/discover';
|
import { discoverPlugins } from '../../plugins/discover';
|
||||||
@@ -57,7 +57,9 @@ pluginsRouter.get('/catalogue/:appName/readme', async (ctx) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
pluginsRouter.get('/', async (ctx) => {
|
pluginsRouter.get('/', async (ctx) => {
|
||||||
const { states, broken } = await snapshotPlugins();
|
// The catalogue, for the update badge — cached, never throws, and falls back to the git rule per
|
||||||
|
// plugin when the store has nothing to say. See `plugins/release.ts`.
|
||||||
|
const { states, broken } = await snapshotPlugins(await publishedTags());
|
||||||
// Asked per plugin rather than once, because `pm2 jlist` is a fork and most plugins have no sidecar to
|
// Asked per plugin rather than once, because `pm2 jlist` is a fork and most plugins have no sidecar to
|
||||||
// ask about. A plugin that is installed and enabled but whose process is not online is the state worth
|
// ask about. A plugin that is installed and enabled but whose process is not online is the state worth
|
||||||
// rendering differently — it is the difference between "off" and "broken".
|
// rendering differently — it is the difference between "off" and "broken".
|
||||||
|
|||||||
@@ -18,9 +18,14 @@ import { catalogueEntry, listCatalogue } from '../../databases/marketplace_db/sr
|
|||||||
*
|
*
|
||||||
* ── Read-only, and structurally so ──
|
* ── Read-only, and structurally so ──
|
||||||
*
|
*
|
||||||
* There are two GET routes and no others. Registering a plugin is `scripts/marketplace-sync.ts`, run by
|
* There are two GET routes and no others. Everything that WRITES the catalogue — registering a plugin,
|
||||||
* hand against the database. That is not a temporary shortcut standing in for a write API: while
|
* publishing a release — is `scripts/marketplace-sync.ts` against the database, on the one machine that
|
||||||
* submissions do not exist, a public write surface is all risk and no feature.
|
* publishes. That is not a temporary shortcut standing in for a write API: while submissions do not
|
||||||
|
* exist, a public write surface is all risk and no feature.
|
||||||
|
*
|
||||||
|
* It stayed that way when publishing went onto a timer, which is the moment it would have been easy to
|
||||||
|
* add a `POST /api/sync` for a Gitea webhook. A minute of delay is cheaper than a write route on the one
|
||||||
|
* process here that has no login.
|
||||||
*
|
*
|
||||||
* ── CORS is wide open, deliberately ──
|
* ── CORS is wide open, deliberately ──
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -0,0 +1,301 @@
|
|||||||
|
import { existsSync, mkdirSync } from 'node:fs';
|
||||||
|
import { homedir, tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import {
|
||||||
|
listRegistry,
|
||||||
|
type RegistryRow,
|
||||||
|
upsertMetadata,
|
||||||
|
upsertPlugin,
|
||||||
|
} from '../../databases/marketplace_db/src/plugins/queries';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fill the catalogue from what each plugin has RELEASED.
|
||||||
|
*
|
||||||
|
* ── The bug this file exists to remove ──
|
||||||
|
*
|
||||||
|
* Until 2026-09-03 the sync read `platform/plugins/` — the directories this machine has INSTALLED — and
|
||||||
|
* imported each `manifest.ts` off the working tree. So "publish" meant "make sure officer-dev's checkout
|
||||||
|
* happens to be at the release, then run the script", and nobody could see when it was not. On the day
|
||||||
|
* this was rewritten the live catalogue was advertising four plugins from untagged working copies:
|
||||||
|
* av-club 2026.08.29.01, kanban 2026.08.29.02, openseo 0.1.0 and fofos-admin 2026.08.29.26, none of
|
||||||
|
* which had ever been tagged. The store was describing somebody's afternoon.
|
||||||
|
*
|
||||||
|
* That is also why the old script could not be put on a timer: a cron over it would have published
|
||||||
|
* whatever was sitting in a working tree, on a schedule.
|
||||||
|
*
|
||||||
|
* ── What replaces it: a bare mirror, read at a tag ──
|
||||||
|
*
|
||||||
|
* Every plugin in the registry is mirrored (`git clone --mirror`) into a cache directory, refreshed, and
|
||||||
|
* read AT ITS NEWEST TAG — the manifest, the README, the file list, all out of `git cat-file`. There is
|
||||||
|
* no working tree anywhere in this path, so publishing something uncommitted is not a mistake that can
|
||||||
|
* be made. A mirror is cheap: the whole set clones in a couple of seconds and refreshes in well under
|
||||||
|
* one.
|
||||||
|
*
|
||||||
|
* `docs/marketplace.md` proposed instead reading manifests over Gitea's API, which needed either a
|
||||||
|
* sandbox to evaluate a fetched string or a committed `manifest.json` for every plugin. Git gives the
|
||||||
|
* same guarantee — content at a named release, independent of any checkout — with neither, and the trust
|
||||||
|
* model is unchanged from today: we import our own repositories' code.
|
||||||
|
*
|
||||||
|
* `[still a blocker for third-party submissions]` It is unchanged BECAUSE it is unchanged. Importing a
|
||||||
|
* stranger's manifest is code execution no matter which ref it came from. Submissions still need a
|
||||||
|
* sandbox or a published `manifest.json`.
|
||||||
|
*
|
||||||
|
* ── Which plugins, and where from: the registry, and only the registry ──
|
||||||
|
*
|
||||||
|
* The list comes from the `plugins` table, which `docs/marketplace.md` has always called the
|
||||||
|
* authoritative, hand-curated half. It was not actually being used that way — the old sync discovered
|
||||||
|
* plugins by listing a directory, so the registry was a by-product of whatever officer-dev had
|
||||||
|
* installed. Now it decides, which means:
|
||||||
|
*
|
||||||
|
* · a plugin nobody has installed here can still be published (`finances` is in exactly that state),
|
||||||
|
* · a plugin lives wherever its registry row says, across orgs — `fofos-admin` is under `fofos/`,
|
||||||
|
* not `plugins/`, so an org listing would have missed it,
|
||||||
|
* · and adding one is a deliberate act: `marketplace-sync.ts --add <https clone url>`.
|
||||||
|
*
|
||||||
|
* ── A plugin with no tag is left ALONE, not removed ──
|
||||||
|
*
|
||||||
|
* It has never made a release, so this file has nothing true to say about it and says nothing: the
|
||||||
|
* existing metadata row stays, and the run reports it. Deleting the row would drop it out of the store
|
||||||
|
* on the day the rule changed, which is a decision about somebody's plugin dressed up as a refresh.
|
||||||
|
* Tagging it is the fix, and the report is there to ask for one.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type SyncOutcome =
|
||||||
|
| { appName: string; status: 'synced'; tag: string; version: string }
|
||||||
|
/** The release on the remote is the one already published. Nothing was written. */
|
||||||
|
| { appName: string; status: 'unchanged'; tag: string }
|
||||||
|
| { appName: string; status: 'unreleased' }
|
||||||
|
| { appName: string; status: 'failed'; error: string };
|
||||||
|
|
||||||
|
/*
|
||||||
|
Where the mirrors live.
|
||||||
|
|
||||||
|
A fixed path under the user's cache directory, NOT one derived from the environment. This runs three
|
||||||
|
ways — by hand with `--env-file=.env`, from the webhook inside `officer-marketplace`, and from cron —
|
||||||
|
and those three do not have the same environment. A path that moved with `DATA_PATH` would have each
|
||||||
|
of them re-cloning twenty-five repositories into its own copy, and the "is it already mirrored" check
|
||||||
|
would answer differently depending on who asked.
|
||||||
|
|
||||||
|
Not in the source tree, because it is a cache of remote repositories and a `.gitignore` entry is a
|
||||||
|
worse way to keep several hundred megabytes out of a repo than not putting them there.
|
||||||
|
*/
|
||||||
|
const MIRROR_DIR =
|
||||||
|
process.env.MARKETPLACE_MIRROR_DIR ?? join(homedir(), '.cache', 'officer-marketplace', 'mirrors');
|
||||||
|
|
||||||
|
const GIT_TIMEOUT_MS = 60_000;
|
||||||
|
|
||||||
|
type Git = { ok: true; out: string } | { ok: false; err: string };
|
||||||
|
|
||||||
|
async function git(args: string[], cwd?: string): Promise<Git> {
|
||||||
|
// A missing `cwd` makes posix_spawn fail with ENOENT, which reads as "git is not installed" and sent
|
||||||
|
// a first run of this looking for git for ten minutes. It means the mirror is not there yet.
|
||||||
|
if (cwd && !existsSync(cwd)) return { ok: false, err: `${cwd} does not exist` };
|
||||||
|
|
||||||
|
const proc = Bun.spawn(['git', ...args], {
|
||||||
|
cwd,
|
||||||
|
stdout: 'pipe',
|
||||||
|
stderr: 'pipe',
|
||||||
|
// Never inherited: a mirror fetch that hits a host asking for a password would otherwise sit on a
|
||||||
|
// terminal prompt forever, and in the webhook server there is no terminal to sit on.
|
||||||
|
env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GIT_ASKPASS: '/bin/true' },
|
||||||
|
signal: AbortSignal.timeout(GIT_TIMEOUT_MS),
|
||||||
|
});
|
||||||
|
const [out, err] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]);
|
||||||
|
return (await proc.exited) === 0 ? { ok: true, out: out.trim() } : { ok: false, err: err.trim() || 'git failed' };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The mirror for a plugin, cloned if this is the first time and refreshed if it is not. */
|
||||||
|
async function mirror(appName: string, repoUrl: string): Promise<string> {
|
||||||
|
mkdirSync(MIRROR_DIR, { recursive: true });
|
||||||
|
const dir = join(MIRROR_DIR, `${appName}.git`);
|
||||||
|
|
||||||
|
// Re-cloned rather than repaired when it is not a healthy bare repo: an interrupted clone leaves a
|
||||||
|
// directory that every later command fails against, and there is nothing in it worth keeping.
|
||||||
|
const existing = await git(['rev-parse', '--is-bare-repository'], dir);
|
||||||
|
if (!existing.ok || existing.out !== 'true') {
|
||||||
|
if (existsSync(dir)) await Bun.$`rm -rf ${dir}`.quiet().nothrow();
|
||||||
|
const cloned = await git(['clone', '--mirror', '--quiet', repoUrl, dir]);
|
||||||
|
if (!cloned.ok) throw new Error(`clone failed: ${cloned.err}`);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The registry is allowed to move a plugin. Setting the URL every time is one command and removes the
|
||||||
|
// failure where a repo is relocated and the mirror keeps refreshing from where it used to be.
|
||||||
|
await git(['remote', 'set-url', 'origin', repoUrl], dir);
|
||||||
|
|
||||||
|
// `--prune --prune-tags`: a deleted or MOVED tag has to disappear here too. Without it, a tag that was
|
||||||
|
// pushed by mistake and deleted would stay in the mirror forever and could go on being published as
|
||||||
|
// the newest release — a store advertising a version its repository no longer has.
|
||||||
|
const fetched = await git(['fetch', '--quiet', '--prune', '--prune-tags', '--tags', 'origin'], dir);
|
||||||
|
if (!fetched.ok) throw new Error(`fetch failed: ${fetched.err}`);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The newest release, or null for a plugin that has never made one.
|
||||||
|
*
|
||||||
|
* Asked of git rather than by sorting version strings, which is the same rule `servers/plugins/release.ts`
|
||||||
|
* follows and for the same reason: parsing `2026.08.27.01` works right up until somebody tags `v2`, and
|
||||||
|
* then fails silently. `--abbrev=0` gives the closest tag reachable from HEAD, and a mirror's HEAD is the
|
||||||
|
* remote's default branch, so this is "the newest release on the branch the repository points at".
|
||||||
|
*/
|
||||||
|
async function newestTag(dir: string): Promise<string | null> {
|
||||||
|
const described = await git(['describe', '--tags', '--abbrev=0', 'HEAD'], dir);
|
||||||
|
return described.ok ? described.out : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A file at a ref, or null when it is not in that tree. */
|
||||||
|
async function fileAt(dir: string, ref: string, path: string): Promise<string | null> {
|
||||||
|
const shown = await git(['cat-file', '-p', `${ref}:${path}`], dir);
|
||||||
|
return shown.ok ? shown.out : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The manifest at a release, as an object.
|
||||||
|
*
|
||||||
|
* Written to a temp file and imported, because a manifest is TypeScript. Keyed by the COMMIT, so the
|
||||||
|
* module cache is correct rather than merely convenient: the same commit is the same file, and a tag
|
||||||
|
* moved to a different commit gets a different path and a fresh import. That matters in the webhook
|
||||||
|
* server, which is long-lived and would otherwise serve the first version of a manifest it ever read.
|
||||||
|
*
|
||||||
|
* The `import type { PluginManifest }` at the top of every manifest is erased by the transpiler, so the
|
||||||
|
* file has no runtime dependency on the platform's path aliases and imports fine from anywhere.
|
||||||
|
*/
|
||||||
|
async function manifestAt(dir: string, ref: string, appName: string, commit: string): Promise<Record<string, unknown>> {
|
||||||
|
const source = await fileAt(dir, ref, 'manifest.ts');
|
||||||
|
if (source === null) throw new Error(`no manifest.ts at ${ref}`);
|
||||||
|
|
||||||
|
const scratch = join(tmpdir(), 'officer-marketplace-manifests');
|
||||||
|
mkdirSync(scratch, { recursive: true });
|
||||||
|
const path = join(scratch, `${appName}-${commit}.ts`);
|
||||||
|
await Bun.write(path, source);
|
||||||
|
|
||||||
|
const mod = (await import(path)) as { manifest?: Record<string, unknown> };
|
||||||
|
if (!mod.manifest) throw new Error(`manifest.ts at ${ref} exports no \`manifest\``);
|
||||||
|
return mod.manifest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A raw-file URL for a Gitea repository, PINNED TO THE RELEASE.
|
||||||
|
*
|
||||||
|
* It used to be `/raw/HEAD/…`, which works only because Gitea 303s it to `/raw/branch/HEAD/…`; the
|
||||||
|
* artwork it served was whatever was on the default branch, so a plugin could show one icon in the store
|
||||||
|
* and ship another. `/raw/tag/<tag>/…` answers 200 directly and shows the icon that belongs to the
|
||||||
|
* version being advertised.
|
||||||
|
*
|
||||||
|
* `[open]` Still Gitea-shaped. A GitHub-hosted plugin needs the other form, and that wants the forge to
|
||||||
|
* be part of the registry rather than inferred from the URL.
|
||||||
|
*/
|
||||||
|
function rawUrl(repoUrl: string, tag: string, path: string): string {
|
||||||
|
return `${repoUrl.replace(/\.git$/, '')}/raw/tag/${encodeURIComponent(tag)}/${path}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
`force` re-publishes a release that is already published.
|
||||||
|
|
||||||
|
Needed because "unchanged" is judged on the plugin's commit, and that is only half the input: the other
|
||||||
|
half is THIS file. Change how a field is derived — as the switch to tag-pinned icon URLs did — and every
|
||||||
|
row is stale while every commit still matches. Then the run to fix them all is `--force`.
|
||||||
|
*/
|
||||||
|
async function syncRegistered(row: RegistryRow, force: boolean): Promise<SyncOutcome> {
|
||||||
|
const { appName, repoUrl } = row;
|
||||||
|
const dir = await mirror(appName, repoUrl);
|
||||||
|
|
||||||
|
const tag = await newestTag(dir);
|
||||||
|
if (!tag) return { appName, status: 'unreleased' };
|
||||||
|
|
||||||
|
const commitAt = await git(['rev-list', '-n', '1', tag], dir);
|
||||||
|
if (!commitAt.ok) throw new Error(`cannot resolve ${tag}`);
|
||||||
|
const commit = commitAt.out;
|
||||||
|
|
||||||
|
// Before the manifest is read, not after: this is what makes a per-minute run affordable. Twenty-five
|
||||||
|
// repositories with nothing new cost one query, twenty-five fetches of nothing, and no writes at all.
|
||||||
|
if (!force && row.sourceCommit === commit) return { appName, status: 'unchanged', tag };
|
||||||
|
|
||||||
|
const manifest = await manifestAt(dir, tag, appName, commit);
|
||||||
|
|
||||||
|
// What the plugin SHIPS, read from the tree at the release rather than from a declaration — the same
|
||||||
|
// rule discovery follows on disk, so a catalogue entry cannot claim a sidecar the release does not
|
||||||
|
// have.
|
||||||
|
const listed = await git(['ls-tree', '-r', '--name-only', tag], dir);
|
||||||
|
const files = new Set(listed.ok ? listed.out.split('\n') : []);
|
||||||
|
const has = {
|
||||||
|
api: files.has('api/router.ts'),
|
||||||
|
schema: files.has('db/schema.ts'),
|
||||||
|
sidecar: files.has('sidecar/index.ts') || files.has('sidecar/index.mjs'),
|
||||||
|
web: [...files].some((f) => f.startsWith('web/')),
|
||||||
|
};
|
||||||
|
|
||||||
|
const readme = (await fileAt(dir, tag, 'README.md')) ?? (await fileAt(dir, tag, 'readme.md'));
|
||||||
|
|
||||||
|
const version = String(manifest.version ?? tag);
|
||||||
|
if (version !== tag) {
|
||||||
|
// Not an error and not corrected here. The tag is what was released and the manifest is what the
|
||||||
|
// plugin will report about itself once installed; when they disagree, an update badge that compares
|
||||||
|
// versions goes quiet. Saying so is the only way anyone finds out.
|
||||||
|
console.warn(`[marketplace-sync] ${appName}: tag ${tag} but manifest says ${version} — they should match`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const developer = String(manifest.publisher ?? 'unknown');
|
||||||
|
const pluginId = await upsertPlugin({ appName, developer, repoUrl });
|
||||||
|
|
||||||
|
await upsertMetadata(pluginId, {
|
||||||
|
version,
|
||||||
|
releaseTag: tag,
|
||||||
|
platform: String(manifest.platform ?? '>=1.0.0'),
|
||||||
|
label: String(manifest.label ?? appName),
|
||||||
|
summary: String(manifest.summary ?? ''),
|
||||||
|
// Taken verbatim, NOT validated here. `manifestProblems` already refuses a tag outside the
|
||||||
|
// vocabulary at discovery, so anything reaching this point is either valid or belongs to a plugin
|
||||||
|
// the platform would not load anyway — and a sync that silently dropped a tag would be a catalogue
|
||||||
|
// quietly disagreeing with the repository it claims to mirror.
|
||||||
|
tags: Array.isArray(manifest.tags) ? manifest.tags : [],
|
||||||
|
color: (manifest.color as string) ?? null,
|
||||||
|
tile: (manifest.tile as string) ?? null,
|
||||||
|
iconUrl: files.has('assets/icon.png') ? rawUrl(repoUrl, tag, 'assets/icon.png') : null,
|
||||||
|
icon: (manifest.icon as string) ?? null,
|
||||||
|
permissions: manifest.permissions ?? [],
|
||||||
|
osDependencies: manifest.osDependencies ?? null,
|
||||||
|
has,
|
||||||
|
readme,
|
||||||
|
sourceCommit: commit,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { appName, status: 'synced', tag, version };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One plugin by app name. Unknown to the registry is a failure, not a silent no-op. */
|
||||||
|
export async function syncPlugin(appName: string, force = false): Promise<SyncOutcome> {
|
||||||
|
const row = (await listRegistry()).find((r) => r.appName === appName);
|
||||||
|
if (!row) return { appName, status: 'failed', error: 'not in the registry' };
|
||||||
|
return syncRegistered(row, force).catch((err) => ({
|
||||||
|
appName,
|
||||||
|
status: 'failed' as const,
|
||||||
|
error: err instanceof Error ? err.message : String(err),
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every registered plugin, one at a time.
|
||||||
|
*
|
||||||
|
* Sequential on purpose. Twenty-four concurrent fetches against one Gitea is a thundering herd for a job
|
||||||
|
* nothing is waiting on, and the whole set finishes in about a second anyway.
|
||||||
|
*
|
||||||
|
* Unlisted rows are included: `listed` hides a plugin from the public API, usually after a bad release,
|
||||||
|
* and it has to be able to come back with current metadata rather than with whatever was true when it
|
||||||
|
* was pulled.
|
||||||
|
*/
|
||||||
|
export async function syncAll(force = false): Promise<SyncOutcome[]> {
|
||||||
|
const rows = await listRegistry();
|
||||||
|
const out: SyncOutcome[] = [];
|
||||||
|
for (const row of rows) {
|
||||||
|
out.push(
|
||||||
|
await syncRegistered(row, force).catch((err) => ({
|
||||||
|
appName: row.appName,
|
||||||
|
status: 'failed' as const,
|
||||||
|
error: err instanceof Error ? err.message : String(err),
|
||||||
|
})),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
@@ -39,6 +39,15 @@ export type CatalogueEntry = {
|
|||||||
appName: string;
|
appName: string;
|
||||||
publisher: string;
|
publisher: string;
|
||||||
version: string;
|
version: string;
|
||||||
|
/**
|
||||||
|
* The git tag this entry was published from — the newest RELEASE, as opposed to whatever a manifest
|
||||||
|
* claims about itself.
|
||||||
|
*
|
||||||
|
* Optional, and absent means "cannot tell" rather than "no release": an older catalogue does not send
|
||||||
|
* the field at all, and a plugin that has never been tagged has nothing to put in it. Both send the
|
||||||
|
* badge back to the git rule.
|
||||||
|
*/
|
||||||
|
releaseTag?: string;
|
||||||
platform: string;
|
platform: string;
|
||||||
label: string;
|
label: string;
|
||||||
summary: string;
|
summary: string;
|
||||||
@@ -556,7 +565,15 @@ const MARKETPLACE_URL = process.env.MARKETPLACE_URL ?? 'https://marketplace.offi
|
|||||||
* plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a
|
* plugin published this morning shows up this morning. The cache is per PROCESS and dies with it — a
|
||||||
* restart is always a fresh look.
|
* restart is always a fresh look.
|
||||||
*/
|
*/
|
||||||
const CACHE_TTL_MS = 5 * 60 * 1000;
|
/*
|
||||||
|
One minute, and the number changed when the badge started depending on it.
|
||||||
|
|
||||||
|
Five was fine while this only drew the store's tiles; it is not fine now that "is there an update" is
|
||||||
|
answered from the catalogue, because the wait for a release to appear became the sync's minute PLUS
|
||||||
|
this one. The cost is one small request per process per minute, and only while somebody is looking at
|
||||||
|
a page that lists plugins.
|
||||||
|
*/
|
||||||
|
const CACHE_TTL_MS = 60 * 1000;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long to wait for the store before giving up on it.
|
* How long to wait for the store before giving up on it.
|
||||||
@@ -639,6 +656,22 @@ export async function fetchCatalogue(): Promise<CatalogueEntry[]> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What the store says the newest release of each plugin is, keyed by app name.
|
||||||
|
*
|
||||||
|
* This is the fact the update badge is built on. It replaces every host in the fleet running `git fetch`
|
||||||
|
* against twenty-one repositories to re-derive something the catalogue already knows — see
|
||||||
|
* `plugins/release.ts` for the rule and for what happens to a plugin that has no entry here.
|
||||||
|
*
|
||||||
|
* A missing key and a null value mean the same thing to a caller and are kept apart for the reader: no
|
||||||
|
* entry at all (unregistered, or a store that could not be reached) versus an entry with no release yet.
|
||||||
|
* Both fall through to the git rule, which is the only honest answer when the store cannot speak.
|
||||||
|
*/
|
||||||
|
export async function publishedTags(): Promise<Map<string, string | null>> {
|
||||||
|
const entries = await fetchCatalogue().catch(() => []);
|
||||||
|
return new Map(entries.map((e) => [e.appName, e.releaseTag ?? null]));
|
||||||
|
}
|
||||||
|
|
||||||
/** One entry by app name, or null. Used by install to find where to clone from. */
|
/** One entry by app name, or null. Used by install to find where to clone from. */
|
||||||
export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
|
export async function catalogueEntry(appName: string): Promise<CatalogueEntry | null> {
|
||||||
const all = await fetchCatalogue().catch(() => BUNDLED);
|
const all = await fetchCatalogue().catch(() => BUNDLED);
|
||||||
|
|||||||
@@ -59,7 +59,13 @@ export type PluginsSnapshot = {
|
|||||||
* the tree while the row stayed, and there is nothing to mount, describe or offer. The row is left in the
|
* the tree while the row stayed, and there is nothing to mount, describe or offer. The row is left in the
|
||||||
* database on purpose — deleting it here would turn "somebody moved the checkout" into silent data loss.
|
* database on purpose — deleting it here would turn "somebody moved the checkout" into silent data loss.
|
||||||
*/
|
*/
|
||||||
export async function snapshotPlugins(): Promise<PluginsSnapshot> {
|
/**
|
||||||
|
* @param published Newest released tag per plugin, from the marketplace — `publishedTags()`. Passed by
|
||||||
|
* the one caller that renders update badges, and by nothing else on purpose: every other caller wants
|
||||||
|
* "what is on disk", and boot is among them. Making this function fetch the catalogue itself would put
|
||||||
|
* a network request with a timeout in front of mounting plugins on a machine with no internet.
|
||||||
|
*/
|
||||||
|
export async function snapshotPlugins(published?: Map<string, string | null>): Promise<PluginsSnapshot> {
|
||||||
const [{ plugins, broken }, installs] = await Promise.all([discoverPlugins(), listPluginInstalls()]);
|
const [{ plugins, broken }, installs] = await Promise.all([discoverPlugins(), listPluginInstalls()]);
|
||||||
const byName = new Map(installs.map((row) => [row.appName, row]));
|
const byName = new Map(installs.map((row) => [row.appName, row]));
|
||||||
|
|
||||||
@@ -72,7 +78,7 @@ export async function snapshotPlugins(): Promise<PluginsSnapshot> {
|
|||||||
plugin,
|
plugin,
|
||||||
install,
|
install,
|
||||||
outdated: !!install && install.version !== plugin.manifest.version,
|
outdated: !!install && install.version !== plugin.manifest.version,
|
||||||
release: await releaseState(plugin.dir),
|
release: await releaseState(plugin.dir, published?.get(plugin.appName) ?? null),
|
||||||
};
|
};
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -94,6 +94,61 @@ describe('releaseState', () => {
|
|||||||
expect(s.updateable).toBe(true);
|
expect(s.updateable).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/*
|
||||||
|
The catalogue path. These are the ones that matter now: a host no longer fetches to find out what the
|
||||||
|
newest release is, so every assertion below is about the store's tag winning over local refs — and
|
||||||
|
about the fallback still being there when it has nothing to say.
|
||||||
|
*/
|
||||||
|
test('the published tag decides, without the checkout having ever fetched it', async () => {
|
||||||
|
const { work, origin, clone } = pair('published-newer');
|
||||||
|
|
||||||
|
// The clone is at v1 and has NEVER seen v2 — no fetch, no remote ref for it. Exactly the state a
|
||||||
|
// host is in a minute after somebody else tagged a release.
|
||||||
|
publish(work, origin, 'two', 'v1');
|
||||||
|
git(clone, 'fetch', '-q', '--tags', 'origin');
|
||||||
|
git(clone, 'checkout', '-q', 'v1');
|
||||||
|
|
||||||
|
const s = await releaseState(clone, 'v2');
|
||||||
|
expect(s.tagAtHead).toBe('v1');
|
||||||
|
expect(s.newestTag).toBe('v2');
|
||||||
|
expect(s.updateable).toBe(true);
|
||||||
|
expect(s.reason).toBe('new-release');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the published tag matching HEAD is up to date', async () => {
|
||||||
|
const { work, origin, clone } = pair('published-same');
|
||||||
|
publish(work, origin, 'two', 'v1');
|
||||||
|
git(clone, 'fetch', '-q', '--tags', 'origin');
|
||||||
|
git(clone, 'checkout', '-q', 'v1');
|
||||||
|
|
||||||
|
const s = await releaseState(clone, 'v1');
|
||||||
|
expect(s.updateable).toBe(false);
|
||||||
|
expect(s.reason).toBe('at-latest');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('an untagged checkout is updateable AND says what it would land on', async () => {
|
||||||
|
const { clone } = pair('published-untagged');
|
||||||
|
const s = await releaseState(clone, 'v9');
|
||||||
|
expect(s.tagAtHead).toBeNull();
|
||||||
|
// The old code returned null here, so the button read "Update" with nothing after it.
|
||||||
|
expect(s.newestTag).toBe('v9');
|
||||||
|
expect(s.updateable).toBe(true);
|
||||||
|
expect(s.reason).toBe('untagged');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('no published tag falls back to the git rule rather than to a guess', async () => {
|
||||||
|
const { work, origin, clone } = pair('no-entry');
|
||||||
|
publish(work, origin, 'two', 'v1');
|
||||||
|
git(clone, 'fetch', '-q', '--tags', 'origin');
|
||||||
|
git(clone, 'checkout', '-q', 'v1');
|
||||||
|
|
||||||
|
// Null, as it is for a plugin nobody registered or a store that could not be reached.
|
||||||
|
const s = await releaseState(clone, null);
|
||||||
|
expect(s.tagAtHead).toBe('v1');
|
||||||
|
expect(s.newestTag).toBe('v1');
|
||||||
|
expect(s.updateable).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
test('a directory that is not a checkout is not updateable', async () => {
|
test('a directory that is not a checkout is not updateable', async () => {
|
||||||
const d = join(root, 'plain');
|
const d = join(root, 'plain');
|
||||||
mkdirSync(d, { recursive: true });
|
mkdirSync(d, { recursive: true });
|
||||||
|
|||||||
@@ -29,18 +29,32 @@ import { join } from 'node:path';
|
|||||||
* is that tag, this checkout is at the latest release. If it is a different one, the remote has moved on.
|
* is that tag, this checkout is at the latest release. If it is a different one, the remote has moved on.
|
||||||
* That holds for any tag names anyone ever chooses, including ones that do not sort.
|
* That holds for any tag names anyone ever chooses, including ones that do not sort.
|
||||||
*
|
*
|
||||||
* ── The answer is never awaited on the network ──
|
* ── Where "the newest tag" comes from: the catalogue, not this machine's refs ──
|
||||||
*
|
*
|
||||||
* Every answer comes from refs already on disk. This is called once per plugin on a request that renders
|
* Every answer here is local, and this is called once per plugin on a request that renders a list —
|
||||||
* a list, and twenty-one network round trips would make the plugins screen feel broken.
|
* twenty-one network round trips would make the plugins screen feel broken. That left one hard problem:
|
||||||
|
* refs on disk go stale. Tags arrived at clone and at the fetch `pullPlugin` does, so you learned an
|
||||||
|
* update existed BY UPDATING. A machine that had not touched a plugin since August compared this
|
||||||
|
* morning's release against a snapshot from before it existed and said, truthfully and uselessly, that
|
||||||
|
* it was at the latest.
|
||||||
*
|
*
|
||||||
* But refs on disk go stale, and for a long time nothing refreshed them: tags arrived at clone and at
|
* That was patched with a rate-limited background `git fetch` per checkout, which worked and was the
|
||||||
* the fetch `pullPlugin` does, so you only learned an update existed BY UPDATING. A machine that had not
|
* wrong shape: every host in the fleet polling twenty-one repositories to re-derive a fact one place
|
||||||
* updated a plugin since August compared this morning's release against a snapshot from before it
|
* already knows. The marketplace publishes releases now — see `servers/marketplace/sync.ts` — so the
|
||||||
* existed and said, truthfully and uselessly, that it was at the latest.
|
* newest release is simply asked of the catalogue, which every host already fetches for the store.
|
||||||
*
|
*
|
||||||
* So `refreshRemote` runs a fetch in the background and this request answers from what is there now. The
|
* published tag (catalogue) vs tag at HEAD (this checkout) → differ, so there is an update
|
||||||
* NEXT read is current. Rate-limited per checkout and run one at a time — see the note on it.
|
*
|
||||||
|
* DIFFERENCE, not ordering. `2026.08.27.01` parses fine until somebody tags `v2`, and then it fails
|
||||||
|
* silently. Two strings that are not equal is a comparison that cannot rot.
|
||||||
|
*
|
||||||
|
* ── The git rule stays, for a plugin the store cannot speak for ──
|
||||||
|
*
|
||||||
|
* A checkout of something unpublished — a plugin being written, one never registered — has no catalogue
|
||||||
|
* entry, and the fallback below is the old rule against local remote-tracking refs. Those refs are no
|
||||||
|
* longer refreshed here, so for that case this can miss a tag until something else fetches. That is the
|
||||||
|
* accepted cost: the alternative was every machine polling every repository forever to cover a case that
|
||||||
|
* only arises on a developer's own box, where `git fetch` is a thing they were about to run anyway.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
export type ReleaseState = {
|
export type ReleaseState = {
|
||||||
@@ -53,65 +67,18 @@ export type ReleaseState = {
|
|||||||
reason: 'untagged' | 'new-release' | 'at-latest' | 'not-a-checkout';
|
reason: 'untagged' | 'new-release' | 'at-latest' | 'not-a-checkout';
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
|
||||||
* A background `git fetch --tags`, at most one per checkout per REFRESH_MS.
|
|
||||||
*
|
|
||||||
* ── Not awaited, on purpose ──
|
|
||||||
*
|
|
||||||
* The caller is rendering a list. Waiting would put the network on the critical path of a page that has
|
|
||||||
* to feel instant, which is the reason this file never fetched in the first place. The request answers
|
|
||||||
* from the refs it has; the fetch it starts is for the next one.
|
|
||||||
*
|
|
||||||
* The cost of that is one stale view after a release, corrected by a refresh. The alternative — never
|
|
||||||
* fetching — is a badge that is wrong until somebody updates for an unrelated reason.
|
|
||||||
*
|
|
||||||
* ── One at a time ──
|
|
||||||
*
|
|
||||||
* Twenty-one plugins would otherwise spawn twenty-one gits at once, every ten minutes, on machines that
|
|
||||||
* are not all fast. Chained rather than pooled because there is no hurry: nothing is waiting on it.
|
|
||||||
*
|
|
||||||
* Failure is silence. A checkout with no remote, an unreachable host, a repository that has moved — none
|
|
||||||
* of those should surface as anything, because the list still has a true answer without them.
|
|
||||||
*/
|
|
||||||
/*
|
|
||||||
Two minutes, and the number is chosen for how releases are actually watched.
|
|
||||||
|
|
||||||
It was ten. The way this gets used is: ship a plugin, open the page, look for the badge — and ten
|
|
||||||
minutes of that is indistinguishable from the feature not working. A release tagged at 12:42 was
|
|
||||||
invisible at 12:45 because the last fetch happened at 12:37, which is correct behaviour and no comfort.
|
|
||||||
|
|
||||||
The cost is bounded by the page being open: nothing fetches when nobody is looking, and the fetches are
|
|
||||||
chained, so the worst case is one small round trip per plugin every two minutes while somebody watches
|
|
||||||
the plugins screen.
|
|
||||||
*/
|
|
||||||
const REFRESH_MS = 2 * 60 * 1000;
|
|
||||||
const fetchedAt = new Map<string, number>();
|
|
||||||
let queue: Promise<unknown> = Promise.resolve();
|
|
||||||
|
|
||||||
export function refreshRemote(dir: string): void {
|
|
||||||
const last = fetchedAt.get(dir) ?? 0;
|
|
||||||
if (Date.now() - last < REFRESH_MS) return;
|
|
||||||
// Stamped BEFORE the fetch runs, so a slow one cannot be started again by the next twenty requests.
|
|
||||||
fetchedAt.set(dir, Date.now());
|
|
||||||
|
|
||||||
queue = queue.then(async () => {
|
|
||||||
const proc = Bun.spawn(['git', '-C', dir, 'fetch', '--tags', '--quiet', 'origin'], {
|
|
||||||
stdout: 'ignore',
|
|
||||||
stderr: 'ignore',
|
|
||||||
// A host that hangs must not hold the queue: every plugin behind it would stop refreshing too.
|
|
||||||
signal: AbortSignal.timeout(20_000),
|
|
||||||
});
|
|
||||||
await proc.exited.catch(() => {});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
const run = async (dir: string, args: string[]): Promise<string | null> => {
|
const run = async (dir: string, args: string[]): Promise<string | null> => {
|
||||||
const proc = Bun.spawn(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
|
const proc = Bun.spawn(['git', '-C', dir, ...args], { stdout: 'pipe', stderr: 'ignore' });
|
||||||
const out = await new Response(proc.stdout).text();
|
const out = await new Response(proc.stdout).text();
|
||||||
return (await proc.exited) === 0 ? out.trim() || null : null;
|
return (await proc.exited) === 0 ? out.trim() || null : null;
|
||||||
};
|
};
|
||||||
|
|
||||||
export async function releaseState(dir: string): Promise<ReleaseState> {
|
/**
|
||||||
|
* @param publishedTag The tag the marketplace says is this plugin's newest release, or null when the
|
||||||
|
* catalogue has nothing to say — an unregistered plugin, one never tagged, or a store that could not
|
||||||
|
* be reached. Null falls through to the git rule rather than to a guess.
|
||||||
|
*/
|
||||||
|
export async function releaseState(dir: string, publishedTag: string | null = null): Promise<ReleaseState> {
|
||||||
const none = { tagAtHead: null, newestTag: null };
|
const none = { tagAtHead: null, newestTag: null };
|
||||||
if (!existsSync(join(dir, '.git'))) {
|
if (!existsSync(join(dir, '.git'))) {
|
||||||
// Copied in by hand or bundled. Nothing to compare, and nothing to pull either — `pullPlugin` says
|
// Copied in by hand or bundled. Nothing to compare, and nothing to pull either — `pullPlugin` says
|
||||||
@@ -119,12 +86,19 @@ export async function releaseState(dir: string): Promise<ReleaseState> {
|
|||||||
return { ...none, updateable: false, reason: 'not-a-checkout' };
|
return { ...none, updateable: false, reason: 'not-a-checkout' };
|
||||||
}
|
}
|
||||||
|
|
||||||
// Started, not awaited: this answer comes from the refs on disk and the fetch is for the next one.
|
|
||||||
refreshRemote(dir);
|
|
||||||
|
|
||||||
// `--exact-match`, so this is "the tag AT this commit", not "the last tag before it". Fails, and
|
// `--exact-match`, so this is "the tag AT this commit", not "the last tag before it". Fails, and
|
||||||
// therefore returns null, when HEAD is between releases — which is the first half of the rule.
|
// therefore returns null, when HEAD is between releases — which is the first half of the rule.
|
||||||
const tagAtHead = await run(dir, ['describe', '--exact-match', '--tags', 'HEAD']);
|
const tagAtHead = await run(dir, ['describe', '--exact-match', '--tags', 'HEAD']);
|
||||||
|
|
||||||
|
if (publishedTag) {
|
||||||
|
// Reported even when HEAD is untagged, so the UI can say what pressing Update lands on. The old
|
||||||
|
// code returned a bare null here and the button read "Update" with nothing after it.
|
||||||
|
if (!tagAtHead) return { tagAtHead: null, newestTag: publishedTag, updateable: true, reason: 'untagged' };
|
||||||
|
return publishedTag === tagAtHead
|
||||||
|
? { tagAtHead, newestTag: publishedTag, updateable: false, reason: 'at-latest' }
|
||||||
|
: { tagAtHead, newestTag: publishedTag, updateable: true, reason: 'new-release' };
|
||||||
|
}
|
||||||
|
|
||||||
if (!tagAtHead) return { ...none, updateable: true, reason: 'untagged' };
|
if (!tagAtHead) return { ...none, updateable: true, reason: 'untagged' };
|
||||||
|
|
||||||
// Asked of the REMOTE branch, not of HEAD: a tag pushed after this checkout was made is not reachable
|
// Asked of the REMOTE branch, not of HEAD: a tag pushed after this checkout was made is not reachable
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { existsSync, readdirSync } from 'node:fs';
|
import { existsSync, readdirSync } from 'node:fs';
|
||||||
|
import { publishedTags } from './marketplace';
|
||||||
import { releaseState } from './release';
|
import { releaseState } from './release';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { PLUGINS_DIR } from './discover';
|
import { PLUGINS_DIR } from './discover';
|
||||||
@@ -144,7 +145,10 @@ export async function updatePlugin(
|
|||||||
|
|
||||||
Checked BEFORE the pull, because a pull is what would make the answer stale.
|
Checked BEFORE the pull, because a pull is what would make the answer stale.
|
||||||
*/
|
*/
|
||||||
const state = await releaseState(join(PLUGINS_DIR, appName));
|
// Against the CATALOGUE, like the badge. Without it this asked local remote-tracking refs, which are
|
||||||
|
// no longer refreshed in the background — so a plugin at the latest tag it knows about would have
|
||||||
|
// short-circuited here and told the owner it was up to date without checking anything.
|
||||||
|
const state = await releaseState(join(PLUGINS_DIR, appName), (await publishedTags()).get(appName) ?? null);
|
||||||
if (!state.updateable && state.reason === 'at-latest') {
|
if (!state.updateable && state.reason === 'at-latest') {
|
||||||
steps.push(`already at ${state.tagAtHead}, the newest release`);
|
steps.push(`already at ${state.tagAtHead}, the newest release`);
|
||||||
onStep?.(steps[steps.length - 1]!);
|
onStep?.(steps[steps.length - 1]!);
|
||||||
|
|||||||
@@ -534,9 +534,11 @@ export const PluginDetail = () => {
|
|||||||
how you teach someone to stop pressing it. */}
|
how you teach someone to stop pressing it. */}
|
||||||
{plugin.updateable && (
|
{plugin.updateable && (
|
||||||
<Button disabled={busy} onClick={() => run('update', plugin.appName)}>
|
<Button disabled={busy} onClick={() => run('update', plugin.appName)}>
|
||||||
{plugin.releaseReason === 'new-release' && plugin.newestTag
|
{/* Named whenever a release is known, not only on `new-release`. An untagged checkout
|
||||||
? `Update to ${plugin.newestTag}`
|
reports `untagged`, and it used to lose the destination with it — the button said
|
||||||
: 'Update'}
|
"Update" and the owner had to guess where it landed. The marketplace knows the newest
|
||||||
|
release even for a checkout that has never fetched it. */}
|
||||||
|
{plugin.newestTag ? `Update to ${plugin.newestTag}` : 'Update'}
|
||||||
</Button>
|
</Button>
|
||||||
)}
|
)}
|
||||||
{/* Said out loud rather than shown as an absent button. "No Update button" and "up to date"
|
{/* Said out loud rather than shown as an absent button. "No Update button" and "up to date"
|
||||||
|
|||||||
Reference in New Issue
Block a user