officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

platform events, and thumbnails warmed the moment files arrive

Browse Source 
a closed, named list of events — `files:uploaded`, `archive:extracted`,
`files:placed` — each carrying absolute paths and an owner. NOT an event bus in the
open sense: `emit(anyString, anything)` becomes an API nobody can change, because
the moment a plugin listens for a string that string is a contract with no list
saying so. a union means adding one is a deliberate edit and removing one fails to
compile.

TWO GUARANTEES, AND THEY ARE THE WHOLE DESIGN. emit never blocks: it schedules and
returns, so an upload does not wait for a subscriber thumbnailing a HEIC. and a
subscriber cannot fail the operation that triggered it — the upload is written and
answered before anybody is told, so a plugin that is down, slow or throwing gets a
line in the log. tested both ways, sync and async, plus that one failing listener
does not stop the next.

PLUGINS SUBSCRIBE THROUGH THE MANIFEST, same shape as `fileTypes`: `hooks: {
'files:uploaded': '/v1/on-upload' }`. resolved at EMIT time rather than at boot,
because plugins are installed, updated and disabled while the server runs and a
registration made at startup describes a machine that has since changed. nothing
here names a plugin.

CORE IS THE FIRST SUBSCRIBER, and that is deliberate rather than a shortcut. the
thumbnail cache belongs to the platform — its key, its route — so a plugin doing
this work would produce files the browser never looks at. the warmer calls the same
`buildThumbnail` the request path calls, which means HEIC still warms through
av-club via the fileTypes claim, without this file knowing that plugin exists.

AND A SINGLE-FLIGHT MAP, which the warmer made necessary. two callers now want the
same thumbnail as a matter of course: the warmer starts on upload and the browser
asks a moment later. measured before the fix — four concurrent requests plus the
warmer each decoded the same 4032x3024 HEIC and took a slot from a limit of four,
which is a warmer competing with the person waiting rather than helping them. after:
all five share one build, 5.5s once, then 130ms.

verified end to end: upload returns in 56ms, four concurrent requests all 200 in a
single 5.5s build, cached 130ms. 1679 tests, docs:check green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Officer-Session: 656f5c85-e49a-4f1f-b9f3-042a04b4d043
This commit is contained in:
pastilhas
2026-08-29 08:20:35 +01:00
co-authored by Claude Opus 5
parent 39c1122651
commit 000df7fe45
8 changed files with 456 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/servers/api/file-browser/router.ts
+32
View File
@@ -11,6 +11,7 @@ import { readOcrConfig } from '@@/api/server-settings/ocr';
import { getUserSettings } from 'officerdb';
import { buildThumbnail, clampSize, thumbnailKind } from './thumbnails';
import { pluginFileTypes, sidecarFetch } from './plugin-file-types';
import { emit } from '../../events/bus';
async function getUserTtsVoice(userId: number): Promise<string | null> {
try {
@@ -333,6 +334,7 @@ router.post('/upload', async (ctx) => {
going to be, next to where it belongs. Nothing sweeps them up — a job deleting files it did not
create is a worse trade than a fragment somebody can see.
*/
const written: string[] = [];
for (const file of files) {
if (!(file instanceof File)) continue;
const filePath = resolve(targetDir, file.name);
@@ -340,8 +342,19 @@ router.post('/upload', async (ctx) => {
await mkdir(dirname(filePath), { recursive: true });
await writeAtomic(filePath, file);
written.push(filePath);
}
/*
Announced AFTER the writes and BEFORE the response, which is the only ordering that is correct.
After, because a subscriber reads these from disk and they have to be there. Before returning is
fine because `emit` schedules and returns — the upload does not wait for anybody. What it must never
become is an await: a subscriber thumbnailing a HEIC would add five seconds to an upload that has
already finished.
*/
emit('files:uploaded', { paths: written, email: user.email });
return ctx.json({ ok: true });
});
@@ -1235,6 +1248,25 @@ router.post('/extract', async (ctx) => {
throw errors.BAD_REQUEST(stderr.trim() || 'Archive extraction failed');
}
/*
What actually came out, walked once rather than guessed from the archive listing.
Bounded, because an archive can hold a hundred thousand files and this runs on the request's way
out. The warmer has its own cap too; this one keeps the WALK from being the expensive part.
*/
const unpacked: string[] = [];
const collect = async (dir: string, depth = 0): Promise<void> => {
if (depth > 8 || unpacked.length >= 500) return;
for (const entry of await readdir(dir, { withFileTypes: true }).catch(() => [])) {
if (unpacked.length >= 500) return;
const full = join(dir, entry.name);
if (entry.isDirectory()) await collect(full, depth + 1);
else unpacked.push(full);
}
};
await collect(finalDest);
emit('archive:extracted', { paths: unpacked, email: user.email });
const extractedPath = '/' + finalDest.slice(rootDir.length).replace(/^\/+/, '');
return ctx.json({ extractedPath });
});
src/servers/api/file-browser/thumbnails.ts
+53 -27
View File
@@ -138,6 +138,19 @@ const release = () => {
* file another chance after somebody has installed a codec or replaced the file. Bounded, because a
* long-lived process browsing a lot of broken media should not grow without limit.
*/
/**
* Builds in flight, by cache key — so the same thumbnail is never made twice at once.
*
* Two callers want the same file all the time now: the warmer starts on upload, and the browser asks
* for it a moment later. Without this they both decode the same 4032x3024 HEIC, take a slot each from a
* limit of four, and finish at the same time having done the work twice — which turns a warmer meant to
* help into one competing with the person waiting.
*
* Keyed on the same string the cache is, so it deduplicates exactly what would collide: same file, same
* size, same content. Removed in a `finally`, or one failure would wedge that key until a restart.
*/
const inFlight = new Map<string, Promise<ThumbnailResult>>();
const failed = new Set<string>();
const FAILED_LIMIT = 5_000;
@@ -245,34 +258,39 @@ export async function buildThumbnail(
const cached = await stat(cachePath).catch(() => null);
if (cached?.isFile() && cached.size > 0) return { ok: true, path: cachePath };
await mkdir(cacheDir, { recursive: true });
await acquire();
try {
/*
// Somebody else is already making this one. Wait for theirs rather than starting a second.
const running = inFlight.get(key);
if (running) return running;
const work = (async (): Promise<ThumbnailResult> => {
await mkdir(cacheDir, { recursive: true });
await acquire();
try {
/*
`force_original_aspect_ratio=decrease` fits inside the box without cropping, and never enlarges
past the source — a 40px icon blown up to 256 is worse than a small one. `-2` on the odd dimension
keeps it even, which the JPEG encoder requires for chroma subsampling.
*/
const scale = `scale=${size}:${size}:force_original_aspect_ratio=decrease`;
const common = ['-v', 'error', '-y', '-frames:v', '1', '-vf', scale, '-f', 'mjpeg', cachePath];
const scale = `scale=${size}:${size}:force_original_aspect_ratio=decrease`;
const common = ['-v', 'error', '-y', '-frames:v', '1', '-vf', scale, '-f', 'mjpeg', cachePath];
let ok: boolean;
if (produce) {
const bytes = await produce().catch(() => null);
if (bytes?.length) await Bun.write(cachePath, bytes);
ok = !!bytes?.length;
} else if (kind === 'image') {
ok = await resizeImage(absPath, cachePath, size);
} else {
// `-ss` BEFORE `-i` is the fast seek — it jumps rather than decoding a second of video first.
ok = await spawnFfmpeg(['-ss', SEEK_SECONDS, '-i', absPath, ...common], cachePath);
// A clip shorter than the seek yields no frame, and that is a fact about its length rather than a
// broken file. Take the first frame instead before giving up on it.
if (!ok) ok = await spawnFfmpeg(['-i', absPath, ...common], cachePath);
}
let ok: boolean;
if (produce) {
const bytes = await produce().catch(() => null);
if (bytes?.length) await Bun.write(cachePath, bytes);
ok = !!bytes?.length;
} else if (kind === 'image') {
ok = await resizeImage(absPath, cachePath, size);
} else {
// `-ss` BEFORE `-i` is the fast seek — it jumps rather than decoding a second of video first.
ok = await spawnFfmpeg(['-ss', SEEK_SECONDS, '-i', absPath, ...common], cachePath);
// A clip shorter than the seek yields no frame, and that is a fact about its length rather than a
// broken file. Take the first frame instead before giving up on it.
if (!ok) ok = await spawnFfmpeg(['-i', absPath, ...common], cachePath);
}
if (!ok) {
/*
if (!ok) {
/*
A PRODUCER'S FAILURE IS NOT REMEMBERED, and that distinction is the whole point of this branch.
When sharp or ffmpeg fails, the file is the problem — it will fail identically forever, and
@@ -284,12 +302,20 @@ export async function buildThumbnail(
the SERVER was restarted — no client retry could ever get past it, because the answer was being
served from memory without the plugin being asked again.
*/
if (!produce) rememberFailure(key);
return { ok: false, reason: 'could not read an image from this file' };
}
if (!produce) rememberFailure(key);
return { ok: false, reason: 'could not read an image from this file' };
}
return { ok: true, path: cachePath };
return { ok: true, path: cachePath };
} finally {
release();
}
})();
inFlight.set(key, work);
try {
return await work;
} finally {
release();
inFlight.delete(key);
}
}