derived/ is built from originals/, and you can watch it happen
The mapping: derived/ mirrors originals/ directory for directory, and where originals holds a FILE, derived holds a DIRECTORY of the same name — extension included — with the renditions inside it. Keeping the extension is what makes it total: a.jpg and a.png are different originals and must not share a folder. The inverse is pure string work, so the filesystem is the record and there is no second copy of the truth. Images get a WebP ladder at 320/640/960/1280/1920/2560 plus full.webp, and nothing is ever upscaled — a rung wider than the original would be a soft file that srcset then confidently prefers over the sharp one. full.webp always exists because for a HEIC it is the only browser-readable copy of the picture there is. Video gets av-club's web-1080 and web-720 presets by name, plus a poster. Images are named by width because srcset descriptors are widths; video by height because that is av-club's vocabulary and nobody misreads 1080.mp4. The ladder is fixed here and the SITE chooses from it. This never reads website/src/media/slots.ts — a plugin that knows a website's layout is a plugin for one website, and that boundary is the difference between a template and something fofos-shaped. Each derived folder carries a manifest naming the size and mtime it was built from, so the sync is idempotent and resumable: a run that died halfway is finished by running it again. That is what makes an in-memory job acceptable rather than merely convenient. It builds into <name>.partial and renames, so an interruption can never leave a folder that looks finished. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+354
@@ -0,0 +1,354 @@
|
|||||||
|
import { mkdir, readdir, readFile, rename, rm, stat, writeFile } from 'node:fs/promises';
|
||||||
|
import { join, relative } from 'node:path';
|
||||||
|
import { sidecarPort } from '@@/sidecar-registry';
|
||||||
|
import { mediaPath } from './media';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Building `derived/` from `originals/`.
|
||||||
|
*
|
||||||
|
* ── The mapping ──
|
||||||
|
*
|
||||||
|
* `derived/` mirrors `originals/` directory for directory. Where originals holds a FILE, derived holds
|
||||||
|
* a DIRECTORY of the same name — extension included — and the renditions live inside it:
|
||||||
|
*
|
||||||
|
* originals/homepage_galeria01.jpg → derived/homepage_galeria01.jpg/320.webp
|
||||||
|
* derived/homepage_galeria01.jpg/640.webp …
|
||||||
|
* originals/ABOUT US/portrait.HEIC → derived/ABOUT US/portrait.HEIC/…
|
||||||
|
* originals/homepage_video.mp4 → derived/homepage_video.mp4/1080.mp4
|
||||||
|
* derived/homepage_video.mp4/poster.webp
|
||||||
|
*
|
||||||
|
* Keeping the extension is what makes the mapping total: `a.jpg` and `a.png` are different originals
|
||||||
|
* and must not share a directory. It also makes the inverse pure string work — no index, no table, no
|
||||||
|
* second copy of the truth. **The filesystem is the record.**
|
||||||
|
*
|
||||||
|
* ── The ladder is fixed here, and the SITE chooses from it ──
|
||||||
|
*
|
||||||
|
* The widths below cover every slot fofos has at up to 3× pixel density — `website/src/media/slots.ts`
|
||||||
|
* puts the largest at 1232 CSS px, so `minUpload` is 2464 and 2560 is the top rung. But this file does
|
||||||
|
* not read that one, and must not: a plugin that knows a website's layout is a plugin for one website.
|
||||||
|
* It emits a ladder; `srcset` and the `sizes` already written in `website/src/media/render.ts` do the
|
||||||
|
* choosing, because the site is the only thing that knows what space an image occupies.
|
||||||
|
*
|
||||||
|
* That boundary is the difference between this being a template and being fofos-shaped.
|
||||||
|
*
|
||||||
|
* ── Nothing is ever upscaled ──
|
||||||
|
*
|
||||||
|
* A rung wider than the original is not written. Inventing a 2560 from a 900px photograph produces a
|
||||||
|
* soft file that `srcset` would then confidently prefer over the sharp one — worse than not having it.
|
||||||
|
*
|
||||||
|
* ── Video rungs are named by height, images by width ──
|
||||||
|
*
|
||||||
|
* Deliberate, not an oversight. `srcset` descriptors are widths, so an image file called `1280` has to
|
||||||
|
* be 1280 pixels wide or the browser is being lied to. Video has no such mechanism, and every person
|
||||||
|
* reading `1080.mp4` knows what it means. Each unit matches its consumer.
|
||||||
|
*
|
||||||
|
* ── Every byte is written by av-club ──
|
||||||
|
*
|
||||||
|
* This module decides WHAT should exist and av-club decides how to make it. The video presets are its
|
||||||
|
* `web-1080` / `web-720` / `poster`, by name — the encoder flags behind them are its business, and
|
||||||
|
* revising them for every site at once is the reason it is a separate service.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** Image rungs, in pixels of width. Rungs wider than the original are skipped — see above. */
|
||||||
|
export const LADDER = [320, 640, 960, 1280, 1920, 2560] as const;
|
||||||
|
|
||||||
|
/** WebP quality. 82 is the usual knee: visually clean on photographs, well off the top of the curve. */
|
||||||
|
const WEBP_QUALITY = 82;
|
||||||
|
|
||||||
|
/** Video renditions, as av-club preset names paired with the source height each one needs to be worth making. */
|
||||||
|
const VIDEO_RUNGS = [
|
||||||
|
{ file: '1080.mp4', preset: 'web-1080', minSourceHeight: 720 },
|
||||||
|
{ file: '720.mp4', preset: 'web-720', minSourceHeight: 0 },
|
||||||
|
] as const;
|
||||||
|
|
||||||
|
const IMAGE_EXT = new Set(['jpg', 'jpeg', 'png', 'gif', 'webp', 'bmp', 'avif', 'tif', 'tiff', 'heic', 'heif']);
|
||||||
|
const VIDEO_EXT = new Set(['mp4', 'mov', 'm4v', 'webm', 'mkv', 'avi', 'wmv', 'flv', 'mpg', 'mpeg', '3gp', 'm2ts', 'mts']);
|
||||||
|
|
||||||
|
export type Kind = 'image' | 'video';
|
||||||
|
|
||||||
|
export type Variant = {
|
||||||
|
file: string;
|
||||||
|
width: number | null;
|
||||||
|
height: number | null;
|
||||||
|
bytes: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What a derived directory says about itself.
|
||||||
|
*
|
||||||
|
* Written last, and it is also the idempotency record: a source whose size and mtime still match the
|
||||||
|
* one recorded here has nothing to redo. That is what makes the initial sync resumable after a
|
||||||
|
* restart, and what makes the watcher's per-file call the same code path as the full pass.
|
||||||
|
*
|
||||||
|
* It is a manifest rather than a database row for the same reason the mapping is string work: a
|
||||||
|
* `derived/` you can delete and rebuild must not leave a row behind claiming it is still there.
|
||||||
|
*/
|
||||||
|
export type Manifest = {
|
||||||
|
kind: Kind;
|
||||||
|
source: { name: string; bytes: number; mtimeMs: number; width: number | null; height: number | null };
|
||||||
|
variants: Variant[];
|
||||||
|
/** ISO. For a person reading the folder, not for the code. */
|
||||||
|
builtAt: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const MANIFEST = 'manifest.json';
|
||||||
|
|
||||||
|
export const extensionOf = (name: string): string => {
|
||||||
|
const dot = name.lastIndexOf('.');
|
||||||
|
return dot <= 0 ? '' : name.slice(dot + 1).toLowerCase();
|
||||||
|
};
|
||||||
|
|
||||||
|
export const kindOf = (name: string): Kind | null => {
|
||||||
|
const ext = extensionOf(name);
|
||||||
|
if (IMAGE_EXT.has(ext)) return 'image';
|
||||||
|
if (VIDEO_EXT.has(ext)) return 'video';
|
||||||
|
return null;
|
||||||
|
};
|
||||||
|
|
||||||
|
export const originalsDir = async (email: string): Promise<string> => join(await mediaPath(email), 'originals');
|
||||||
|
export const derivedDir = async (email: string): Promise<string> => join(await mediaPath(email), 'derived');
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Talk to av-club.
|
||||||
|
*
|
||||||
|
* Straight to the sidecar's loopback port rather than back through the platform's own HTTP: this code
|
||||||
|
* IS the platform, and making it authenticate to itself to reach a service already on the machine adds
|
||||||
|
* a credential and a hop to buy nothing. `x-officer-user` is the sidecar's whole authentication and is
|
||||||
|
* exactly what its proxy would have sent — see `src/servers/api/file-browser/plugin-file-types.ts`,
|
||||||
|
* which reaches it the same way and explains why that is sound.
|
||||||
|
*/
|
||||||
|
class AvClubUnavailable extends Error {}
|
||||||
|
|
||||||
|
async function avClub<T>(path: string, email: string, init?: RequestInit): Promise<T> {
|
||||||
|
const port = sidecarPort('av-club');
|
||||||
|
if (!port) throw new AvClubUnavailable('av-club is not running. Install and start the AV Club plugin.');
|
||||||
|
|
||||||
|
const response = await fetch(`http://127.0.0.1:${port}${path}`, {
|
||||||
|
...init,
|
||||||
|
headers: { 'x-officer-user': email, ...(init?.body ? { 'content-type': 'application/json' } : {}), ...init?.headers },
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
const body = (await response.json().catch(() => null)) as { error?: { message?: string } } | null;
|
||||||
|
throw new Error(body?.error?.message ?? `av-club answered ${response.status}`);
|
||||||
|
}
|
||||||
|
return (await response.json()) as T;
|
||||||
|
}
|
||||||
|
|
||||||
|
const post = <T>(path: string, email: string, body: unknown): Promise<T> =>
|
||||||
|
avClub<T>(path, email, { method: 'POST', body: JSON.stringify(body) });
|
||||||
|
|
||||||
|
/** Is this derived directory already correct for this original? */
|
||||||
|
async function upToDate(dir: string, source: { bytes: number; mtimeMs: number }): Promise<boolean> {
|
||||||
|
const raw = await readFile(join(dir, MANIFEST), 'utf8').catch(() => null);
|
||||||
|
if (!raw) return false;
|
||||||
|
try {
|
||||||
|
const manifest = JSON.parse(raw) as Manifest;
|
||||||
|
return manifest.source.bytes === source.bytes && manifest.source.mtimeMs === source.mtimeMs;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build one original's directory of renditions.
|
||||||
|
*
|
||||||
|
* ── It builds beside, then swaps ──
|
||||||
|
*
|
||||||
|
* Into `<name>.partial`, renamed over the real one only once everything succeeded. A sync interrupted
|
||||||
|
* halfway — a restart, a kill, ffmpeg dying on one file — must never leave a directory that LOOKS
|
||||||
|
* finished, because the manifest is what everything else trusts. Half a ladder with a manifest is
|
||||||
|
* worse than no ladder at all.
|
||||||
|
*/
|
||||||
|
export async function deriveOne(
|
||||||
|
sourcePath: string,
|
||||||
|
targetDir: string,
|
||||||
|
email: string,
|
||||||
|
options: { force?: boolean } = {},
|
||||||
|
): Promise<{ built: boolean; manifest: Manifest }> {
|
||||||
|
const name = sourcePath.slice(sourcePath.lastIndexOf('/') + 1);
|
||||||
|
const kind = kindOf(name);
|
||||||
|
if (!kind) throw new Error(`${name} is neither an image nor a video this understands.`);
|
||||||
|
|
||||||
|
const info = await stat(sourcePath);
|
||||||
|
const source = { bytes: info.size, mtimeMs: Math.round(info.mtimeMs) };
|
||||||
|
|
||||||
|
if (!options.force && (await upToDate(targetDir, source))) {
|
||||||
|
const manifest = JSON.parse(await readFile(join(targetDir, MANIFEST), 'utf8')) as Manifest;
|
||||||
|
return { built: false, manifest };
|
||||||
|
}
|
||||||
|
|
||||||
|
const staging = `${targetDir}.partial`;
|
||||||
|
await rm(staging, { recursive: true, force: true });
|
||||||
|
await mkdir(staging, { recursive: true });
|
||||||
|
|
||||||
|
try {
|
||||||
|
const manifest =
|
||||||
|
kind === 'image'
|
||||||
|
? await buildImage(sourcePath, staging, name, source, email)
|
||||||
|
: await buildVideo(sourcePath, staging, name, source, email);
|
||||||
|
|
||||||
|
await writeFile(join(staging, MANIFEST), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8');
|
||||||
|
|
||||||
|
// `rename` cannot replace a non-empty directory, so the old one goes first. The window between the
|
||||||
|
// two is the reason this is not the step that produces the files.
|
||||||
|
await rm(targetDir, { recursive: true, force: true });
|
||||||
|
await rename(staging, targetDir);
|
||||||
|
|
||||||
|
return { built: true, manifest };
|
||||||
|
} catch (error) {
|
||||||
|
await rm(staging, { recursive: true, force: true }).catch(() => {});
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function buildImage(
|
||||||
|
sourcePath: string,
|
||||||
|
staging: string,
|
||||||
|
name: string,
|
||||||
|
source: { bytes: number; mtimeMs: number },
|
||||||
|
email: string,
|
||||||
|
): Promise<Manifest> {
|
||||||
|
const meta = await avClub<{ width: number | null; height: number | null }>(
|
||||||
|
`/v1/image?path=${encodeURIComponent(sourcePath)}`,
|
||||||
|
email,
|
||||||
|
);
|
||||||
|
const width = meta.width ?? 0;
|
||||||
|
const height = meta.height ?? 0;
|
||||||
|
if (!width || !height) throw new Error(`Could not read the dimensions of ${name}.`);
|
||||||
|
|
||||||
|
const variants: Variant[] = [];
|
||||||
|
|
||||||
|
/*
|
||||||
|
`full.webp` first, and always.
|
||||||
|
|
||||||
|
It is the top of the ladder, and for a HEIC it is the only browser-readable copy of the picture that
|
||||||
|
exists at all — the original renders nowhere. A lightbox, a zoom, or handing the client their own
|
||||||
|
photograph back all need it.
|
||||||
|
*/
|
||||||
|
variants.push(await encodeWebp(sourcePath, join(staging, 'full.webp'), null, email, width, height));
|
||||||
|
|
||||||
|
// Strictly less than the original: a rung at or above it would duplicate `full.webp` byte for byte.
|
||||||
|
for (const rung of LADDER) {
|
||||||
|
if (rung >= width) continue;
|
||||||
|
variants.push(await encodeWebp(sourcePath, join(staging, `${rung}.webp`), rung, email, width, height));
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
kind: 'image',
|
||||||
|
source: { name, bytes: source.bytes, mtimeMs: source.mtimeMs, width, height },
|
||||||
|
variants,
|
||||||
|
builtAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function encodeWebp(
|
||||||
|
sourcePath: string,
|
||||||
|
outputPath: string,
|
||||||
|
rung: number | null,
|
||||||
|
email: string,
|
||||||
|
sourceWidth: number,
|
||||||
|
sourceHeight: number,
|
||||||
|
): Promise<Variant> {
|
||||||
|
const result = await post<{ path: string; bytes: number }>('/v1/sharp', email, {
|
||||||
|
inputPath: sourcePath,
|
||||||
|
outputPath,
|
||||||
|
// `rotate` with no angle applies the EXIF orientation, which phone photographs rely on entirely. A
|
||||||
|
// ladder built without it is a set of sideways pictures.
|
||||||
|
pipeline: [
|
||||||
|
{ rotate: {} },
|
||||||
|
...(rung === null ? [] : [{ resize: { width: rung, fit: 'inside', withoutEnlargement: true } }]),
|
||||||
|
],
|
||||||
|
output: { webp: { quality: WEBP_QUALITY } },
|
||||||
|
});
|
||||||
|
|
||||||
|
const width = rung ?? sourceWidth;
|
||||||
|
return {
|
||||||
|
file: outputPath.slice(outputPath.lastIndexOf('/') + 1),
|
||||||
|
width,
|
||||||
|
height: Math.round((width / sourceWidth) * sourceHeight),
|
||||||
|
bytes: result.bytes,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function buildVideo(
|
||||||
|
sourcePath: string,
|
||||||
|
staging: string,
|
||||||
|
name: string,
|
||||||
|
source: { bytes: number; mtimeMs: number },
|
||||||
|
email: string,
|
||||||
|
): Promise<Manifest> {
|
||||||
|
const probe = await avClub<{ streams?: Array<{ codec_type?: string; width?: number; height?: number }> }>(
|
||||||
|
`/v1/probe?path=${encodeURIComponent(sourcePath)}`,
|
||||||
|
email,
|
||||||
|
);
|
||||||
|
const video = probe.streams?.find((stream) => stream.codec_type === 'video');
|
||||||
|
const height = video?.height ?? 0;
|
||||||
|
const width = video?.width ?? 0;
|
||||||
|
|
||||||
|
const variants: Variant[] = [];
|
||||||
|
|
||||||
|
for (const rung of VIDEO_RUNGS) {
|
||||||
|
// A 720p source re-encoded as "1080" is bigger and no better. The lowest rung has no floor, so
|
||||||
|
// every video gets at least one rendition however small it started.
|
||||||
|
if (height && height < rung.minSourceHeight) continue;
|
||||||
|
const result = await post<{ path: string; bytes: number }>('/v1/ffmpeg', email, {
|
||||||
|
inputPath: sourcePath,
|
||||||
|
outputPath: join(staging, rung.file),
|
||||||
|
preset: rung.preset,
|
||||||
|
});
|
||||||
|
variants.push({ file: rung.file, width: null, height: null, bytes: result.bytes });
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
The poster, in two steps.
|
||||||
|
|
||||||
|
av-club's `poster` preset writes a JPEG, which is its call and the right one for a general service.
|
||||||
|
Here everything a browser is handed is WebP, so it goes through sharp on the way out and the JPEG
|
||||||
|
is dropped. Two calls per video, once, at build time.
|
||||||
|
*/
|
||||||
|
const stillJpg = join(staging, 'poster.jpg');
|
||||||
|
await post('/v1/ffmpeg', email, {
|
||||||
|
inputPath: sourcePath,
|
||||||
|
outputPath: stillJpg,
|
||||||
|
preset: 'poster',
|
||||||
|
// A frame from a second in, because frame zero of a great many videos is black.
|
||||||
|
inputOptions: { ss: 1 },
|
||||||
|
}).catch(() => post('/v1/ffmpeg', email, { inputPath: sourcePath, outputPath: stillJpg, preset: 'poster' }));
|
||||||
|
|
||||||
|
const poster = await post<{ path: string; bytes: number }>('/v1/sharp', email, {
|
||||||
|
inputPath: stillJpg,
|
||||||
|
outputPath: join(staging, 'poster.webp'),
|
||||||
|
output: { webp: { quality: WEBP_QUALITY } },
|
||||||
|
});
|
||||||
|
await rm(stillJpg, { force: true });
|
||||||
|
variants.push({ file: 'poster.webp', width: null, height: null, bytes: poster.bytes });
|
||||||
|
|
||||||
|
return {
|
||||||
|
kind: 'video',
|
||||||
|
source: { name, bytes: source.bytes, mtimeMs: source.mtimeMs, width: width || null, height: height || null },
|
||||||
|
variants,
|
||||||
|
builtAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every file under `originals/`, relative to it, depth first. Directories are structure, not work. */
|
||||||
|
export async function listOriginals(root: string, prefix = ''): Promise<string[]> {
|
||||||
|
const entries = await readdir(join(root, prefix), { withFileTypes: true }).catch(() => []);
|
||||||
|
const found: string[] = [];
|
||||||
|
|
||||||
|
for (const entry of entries) {
|
||||||
|
// Dotfiles are `.DS_Store` and friends. Nothing under here is a deliberate hidden file.
|
||||||
|
if (entry.name.startsWith('.')) continue;
|
||||||
|
const rel = prefix ? `${prefix}/${entry.name}` : entry.name;
|
||||||
|
if (entry.isDirectory()) found.push(...(await listOriginals(root, rel)));
|
||||||
|
else if (kindOf(entry.name)) found.push(rel);
|
||||||
|
}
|
||||||
|
|
||||||
|
return found;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Where a relative original ends up. Exported because the watcher needs the same answer. */
|
||||||
|
export const targetFor = (derivedRoot: string, relativePath: string): string => join(derivedRoot, relativePath);
|
||||||
|
|
||||||
|
export { relative, AvClubUnavailable };
|
||||||
@@ -5,6 +5,7 @@ import { deleteConnection, readConnection, readCredentials, recordProbe, saveCon
|
|||||||
import { readSettings, saveRootPath } from './settings';
|
import { readSettings, saveRootPath } from './settings';
|
||||||
import { readWebsiteStatus, cloneWebsite, updateWebsite } from './website';
|
import { readWebsiteStatus, cloneWebsite, updateWebsite } from './website';
|
||||||
import { readMediaStatus, createMediaDir } from './media';
|
import { readMediaStatus, createMediaDir } from './media';
|
||||||
|
import { readSync, startSync } from './sync';
|
||||||
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
|
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -228,6 +229,28 @@ router.post('/_config/media/create', async (ctx) => {
|
|||||||
return ctx.json({ ...result, status: await readMediaStatus(user.email) }, result.ok ? 200 : 400);
|
return ctx.json({ ...result, status: await readMediaStatus(user.email) }, result.ok ? 200 : 400);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Building `derived/` from `originals/`.
|
||||||
|
*
|
||||||
|
* Two routes and no more: start one, and ask how it is going. The work is asynchronous because a
|
||||||
|
* folder of photographs takes minutes, and the panel polls the second route rather than holding a
|
||||||
|
* request open — a connection that has to survive an eight-minute encode is a connection that will
|
||||||
|
* not.
|
||||||
|
*
|
||||||
|
* `POST` answers 200 whether it started a run or found one already going, and says which in `started`.
|
||||||
|
* "Already running" is not a failure to report to somebody who pressed Sync; the honest answer is the
|
||||||
|
* run that is in progress, which is what comes back.
|
||||||
|
*/
|
||||||
|
router.post('/_config/media/sync', async (ctx) => {
|
||||||
|
const user = ctx.get('user');
|
||||||
|
return ctx.json(startSync(user.email));
|
||||||
|
});
|
||||||
|
|
||||||
|
router.get('/_config/media/sync', async (ctx) => {
|
||||||
|
const user = ctx.get('user');
|
||||||
|
return ctx.json(readSync(user.email));
|
||||||
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Is this plugin usable? Asked once by the detail panel rather than by all nine sections.
|
* Is this plugin usable? Asked once by the detail panel rather than by all nine sections.
|
||||||
*
|
*
|
||||||
|
|||||||
+143
@@ -0,0 +1,143 @@
|
|||||||
|
import { mkdir } from 'node:fs/promises';
|
||||||
|
import { dirname } from 'node:path';
|
||||||
|
import { deriveOne, derivedDir, listOriginals, originalsDir, targetFor } from './derive';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The sync: bring `derived/` into line with `originals/`.
|
||||||
|
*
|
||||||
|
* ── One job, in memory, per user ──
|
||||||
|
*
|
||||||
|
* Not a `pipeline_jobs` row. That table belongs to the task runner, which runs a script in a temp
|
||||||
|
* directory with a cwd and a log — a good fit for what a user wrote and a poor one for a plugin
|
||||||
|
* calling a service in-process. So the state lives here, and the cost is stated plainly: **a restart
|
||||||
|
* loses the job.**
|
||||||
|
*
|
||||||
|
* That cost is small because the work is idempotent. Every derived directory carries a manifest naming
|
||||||
|
* the size and mtime it was built from, so a sync that died halfway is finished by running it again —
|
||||||
|
* it skips everything already correct and picks up where the interruption left it. There is nothing to
|
||||||
|
* reconcile and nothing to clean up, which is the property that made an in-memory job acceptable
|
||||||
|
* rather than merely convenient.
|
||||||
|
*
|
||||||
|
* ── The same code path as a single file ──
|
||||||
|
*
|
||||||
|
* `deriveOne` is what the watcher will call for one upload. This walks and calls it; it decides
|
||||||
|
* nothing about how a file is built. A full sync and a single arrival must never be two implementations
|
||||||
|
* that can disagree about what a derived directory should contain.
|
||||||
|
*
|
||||||
|
* ── Two at a time ──
|
||||||
|
*
|
||||||
|
* av-club runs two jobs concurrently and queues the rest, so a third caller would only wait in its
|
||||||
|
* queue while making this module's progress harder to report. Matching its number keeps both slots
|
||||||
|
* busy and keeps "what is happening now" answerable.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type SyncError = { file: string; message: string };
|
||||||
|
|
||||||
|
export type SyncState = {
|
||||||
|
running: boolean;
|
||||||
|
/** How many originals the walk found. Null until the walk finishes. */
|
||||||
|
total: number | null;
|
||||||
|
/** Attempted — built, skipped and failed together. What a progress bar divides by `total`. */
|
||||||
|
done: number;
|
||||||
|
built: number;
|
||||||
|
skipped: number;
|
||||||
|
errors: SyncError[];
|
||||||
|
/** The files being worked on right now, for somebody watching. Empty when idle. */
|
||||||
|
current: string[];
|
||||||
|
startedAt: string | null;
|
||||||
|
finishedAt: string | null;
|
||||||
|
/** Set when the run stopped for a reason that was not about any single file — av-club being down. */
|
||||||
|
fatal: string | null;
|
||||||
|
};
|
||||||
|
|
||||||
|
const CONCURRENCY = 2;
|
||||||
|
|
||||||
|
const idle = (): SyncState => ({
|
||||||
|
running: false,
|
||||||
|
total: null,
|
||||||
|
done: 0,
|
||||||
|
built: 0,
|
||||||
|
skipped: 0,
|
||||||
|
errors: [],
|
||||||
|
current: [],
|
||||||
|
startedAt: null,
|
||||||
|
finishedAt: null,
|
||||||
|
fatal: null,
|
||||||
|
});
|
||||||
|
|
||||||
|
const states = new Map<string, SyncState>();
|
||||||
|
|
||||||
|
export const readSync = (email: string): SyncState => states.get(email) ?? idle();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Start one, unless one is already running.
|
||||||
|
*
|
||||||
|
* Returns the state either way, so the caller does not have to decide what "already running" means to
|
||||||
|
* a person: the panel shows the run that is in progress, which is the honest answer to "sync now".
|
||||||
|
*/
|
||||||
|
export function startSync(email: string): { started: boolean; state: SyncState } {
|
||||||
|
const existing = states.get(email);
|
||||||
|
if (existing?.running) return { started: false, state: existing };
|
||||||
|
|
||||||
|
const state: SyncState = { ...idle(), running: true, startedAt: new Date().toISOString() };
|
||||||
|
states.set(email, state);
|
||||||
|
|
||||||
|
// Deliberately not awaited. The route answers immediately with the job's opening state and the panel
|
||||||
|
// polls; an await here would be the request timing out on a folder of any size.
|
||||||
|
void run(email, state);
|
||||||
|
|
||||||
|
return { started: true, state };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function run(email: string, state: SyncState): Promise<void> {
|
||||||
|
try {
|
||||||
|
const from = await originalsDir(email);
|
||||||
|
const to = await derivedDir(email);
|
||||||
|
|
||||||
|
const files = await listOriginals(from);
|
||||||
|
state.total = files.length;
|
||||||
|
|
||||||
|
let next = 0;
|
||||||
|
const worker = async (): Promise<void> => {
|
||||||
|
for (;;) {
|
||||||
|
const index = next++;
|
||||||
|
const file = files[index];
|
||||||
|
if (file === undefined) return;
|
||||||
|
|
||||||
|
state.current.push(file);
|
||||||
|
try {
|
||||||
|
const target = targetFor(to, file);
|
||||||
|
await mkdir(dirname(target), { recursive: true });
|
||||||
|
const { built } = await deriveOne(`${from}/${file}`, target, email);
|
||||||
|
if (built) state.built++;
|
||||||
|
else state.skipped++;
|
||||||
|
} catch (error) {
|
||||||
|
const message = (error as Error).message;
|
||||||
|
/*
|
||||||
|
av-club being unreachable is not a fact about this file, and recording it once per file
|
||||||
|
would bury the one thing worth reading under four hundred copies of it. Stop the run and
|
||||||
|
say so once.
|
||||||
|
*/
|
||||||
|
if (message.includes('av-club is not running')) {
|
||||||
|
state.fatal = message;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
state.errors.push({ file, message });
|
||||||
|
} finally {
|
||||||
|
state.current = state.current.filter((name) => name !== file);
|
||||||
|
state.done++;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (state.fatal) return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
|
||||||
|
} catch (error) {
|
||||||
|
state.fatal = (error as Error).message;
|
||||||
|
} finally {
|
||||||
|
state.running = false;
|
||||||
|
state.current = [];
|
||||||
|
state.finishedAt = new Date().toISOString();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
import { RootLocationBlock } from './RootLocationBlock';
|
import { RootLocationBlock } from './RootLocationBlock';
|
||||||
|
import { MediaSyncBlock } from './MediaSyncBlock';
|
||||||
import { DatabaseBlock } from './DatabaseBlock';
|
import { DatabaseBlock } from './DatabaseBlock';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -7,8 +8,8 @@ import { DatabaseBlock } from './DatabaseBlock';
|
|||||||
* A list of blocks, and nothing else. Each one owns its own heading, its own state and its own status
|
* A list of blocks, and nothing else. Each one owns its own heading, its own state and its own status
|
||||||
* message; this file decides only which of them exist and in what order.
|
* message; this file decides only which of them exist and in what order.
|
||||||
*
|
*
|
||||||
* The order is the order things are needed in: where the site is on this machine, then how to read its
|
* The order is the order things are needed in: where the site is on this machine, then turning what
|
||||||
* data.
|
* was uploaded into what the site serves, then how to read its data.
|
||||||
*
|
*
|
||||||
* ── What used to be here ──
|
* ── What used to be here ──
|
||||||
*
|
*
|
||||||
@@ -21,6 +22,7 @@ import { DatabaseBlock } from './DatabaseBlock';
|
|||||||
export const AdministrationPanel = () => (
|
export const AdministrationPanel = () => (
|
||||||
<div className="flex size-full flex-col gap-4 overflow-auto p-5">
|
<div className="flex size-full flex-col gap-4 overflow-auto p-5">
|
||||||
<RootLocationBlock />
|
<RootLocationBlock />
|
||||||
|
<MediaSyncBlock />
|
||||||
<DatabaseBlock />
|
<DatabaseBlock />
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -0,0 +1,184 @@
|
|||||||
|
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||||
|
import { CircleAlert, CircleCheck, Loader2, RefreshCw, Wand2 } from 'lucide-react';
|
||||||
|
import { useClient } from 'hooks/useClient';
|
||||||
|
import { Button } from '@/components/ui/button';
|
||||||
|
import { readableError } from './errors';
|
||||||
|
import { Block } from './Block';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Building `derived/` from `originals/`.
|
||||||
|
*
|
||||||
|
* ── Why this is visible at all ──
|
||||||
|
*
|
||||||
|
* The work happens without anybody watching: a file arrives and something converts it minutes later.
|
||||||
|
* That is the right behaviour and the wrong experience — a person who has just uploaded forty
|
||||||
|
* photographs has no way to tell "being converted" from "nothing happened". So the state of the run is
|
||||||
|
* on the screen, whether the run was started here or by an upload.
|
||||||
|
*
|
||||||
|
* ── It polls, and only while something is running ──
|
||||||
|
*
|
||||||
|
* A full pass takes minutes, so the request that starts it returns immediately and this asks how it is
|
||||||
|
* going. The poll stops the moment the answer says it is not running — an idle panel makes no requests,
|
||||||
|
* which is what keeps a screen somebody left open from being a request every second all afternoon.
|
||||||
|
*
|
||||||
|
* ── The counts distinguish built from skipped, deliberately ──
|
||||||
|
*
|
||||||
|
* Re-running is normal and cheap: everything already correct is skipped, because each derived folder
|
||||||
|
* carries the size and mtime it was built from. A run reporting "412 skipped" is the system saying it
|
||||||
|
* had nothing to do, which is a different and more reassuring answer than "412 done".
|
||||||
|
*
|
||||||
|
* ── Failures stay ──
|
||||||
|
*
|
||||||
|
* A file that could not be converted is listed with the reason and does not clear itself. One bad file
|
||||||
|
* out of four hundred is exactly the thing a progress bar hides, and it is the only part of this
|
||||||
|
* anybody has to act on.
|
||||||
|
*/
|
||||||
|
|
||||||
|
type SyncError = { file: string; message: string };
|
||||||
|
|
||||||
|
type SyncState = {
|
||||||
|
running: boolean;
|
||||||
|
total: number | null;
|
||||||
|
done: number;
|
||||||
|
built: number;
|
||||||
|
skipped: number;
|
||||||
|
errors: SyncError[];
|
||||||
|
current: string[];
|
||||||
|
startedAt: string | null;
|
||||||
|
finishedAt: string | null;
|
||||||
|
fatal: string | null;
|
||||||
|
};
|
||||||
|
|
||||||
|
/** How often to ask, while something is happening. Fast enough to look live, slow enough to be free. */
|
||||||
|
const POLL_MS = 1000;
|
||||||
|
|
||||||
|
export const MediaSyncBlock = () => {
|
||||||
|
const client = useClient();
|
||||||
|
const clientRef = useRef(client);
|
||||||
|
clientRef.current = client;
|
||||||
|
|
||||||
|
const [state, setState] = useState<SyncState | null>(null);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const load = useCallback(async () => {
|
||||||
|
try {
|
||||||
|
setState(await clientRef.current.get<SyncState>('/fofos-admin/_config/media/sync'));
|
||||||
|
} catch {
|
||||||
|
// Owner-only, and Root Location above has already said so once if that is the reason.
|
||||||
|
setState(null);
|
||||||
|
}
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
void load();
|
||||||
|
}, [load]);
|
||||||
|
|
||||||
|
/*
|
||||||
|
The poll, keyed on whether a run is in progress.
|
||||||
|
|
||||||
|
The interval is torn down and not replaced when `running` goes false, so the last tick that
|
||||||
|
observes the finish is also the one that stops the asking. No separate "should I still be polling"
|
||||||
|
flag to get out of step with the answer.
|
||||||
|
*/
|
||||||
|
useEffect(() => {
|
||||||
|
if (!state?.running) return;
|
||||||
|
const timer = setInterval(() => void load(), POLL_MS);
|
||||||
|
return () => clearInterval(timer);
|
||||||
|
}, [state?.running, load]);
|
||||||
|
|
||||||
|
const start = async () => {
|
||||||
|
setBusy(true);
|
||||||
|
setError(null);
|
||||||
|
try {
|
||||||
|
const result = await clientRef.current.post<{ started: boolean; state: SyncState }>(
|
||||||
|
'/fofos-admin/_config/media/sync',
|
||||||
|
{},
|
||||||
|
);
|
||||||
|
setState(result.state);
|
||||||
|
} catch (caught) {
|
||||||
|
setError(readableError(caught));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const running = state?.running ?? false;
|
||||||
|
const total = state?.total ?? null;
|
||||||
|
const done = state?.done ?? 0;
|
||||||
|
// Null total means the walk has not finished, so there is nothing honest to divide by yet.
|
||||||
|
const percent = total && total > 0 ? Math.round((done / total) * 100) : null;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Block
|
||||||
|
icon={Wand2}
|
||||||
|
title="Derived media"
|
||||||
|
hint="Web-ready copies of everything in originals — resized, converted, and safe to rebuild."
|
||||||
|
status={
|
||||||
|
running ? (
|
||||||
|
<span className="flex items-center gap-1.5 text-xs text-muted-foreground">
|
||||||
|
<Loader2 className="size-3.5 animate-spin" />
|
||||||
|
{percent === null ? 'Looking' : `${percent}%`}
|
||||||
|
</span>
|
||||||
|
) : state?.finishedAt ? (
|
||||||
|
<span className="flex items-center gap-1.5 text-xs text-muted-foreground">
|
||||||
|
<CircleCheck className="size-3.5" />
|
||||||
|
Finished
|
||||||
|
</span>
|
||||||
|
) : undefined
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{running && (
|
||||||
|
<div className="space-y-2">
|
||||||
|
<div className="h-1.5 w-full overflow-hidden rounded-full bg-muted">
|
||||||
|
<div
|
||||||
|
className="h-full rounded-full bg-primary transition-[width] duration-500"
|
||||||
|
// Indeterminate until the walk has counted: a bar that starts full and shrinks is worse
|
||||||
|
// than one that has not started.
|
||||||
|
style={{ width: `${percent ?? 0}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<p className="truncate text-xs text-muted-foreground">
|
||||||
|
{total === null ? 'Reading originals…' : `${done} of ${total}`}
|
||||||
|
{state?.current.length ? ` · ${state.current.join(', ')}` : ''}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{!running && state && (state.built > 0 || state.skipped > 0 || state.errors.length > 0) && (
|
||||||
|
<p className="text-xs text-muted-foreground">
|
||||||
|
{state.built} built · {state.skipped} already current
|
||||||
|
{state.errors.length > 0 ? ` · ${state.errors.length} failed` : ''}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{state?.fatal && (
|
||||||
|
<p className="flex items-start gap-1.5 text-xs text-destructive">
|
||||||
|
<CircleAlert className="mt-px size-3.5 shrink-0" />
|
||||||
|
{state.fatal}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{state && state.errors.length > 0 && (
|
||||||
|
<ul className="max-h-40 space-y-1 overflow-auto rounded border border-border/60 p-2">
|
||||||
|
{state.errors.map((failure) => (
|
||||||
|
<li key={failure.file} className="text-xs">
|
||||||
|
<span className="font-medium">{failure.file}</span>
|
||||||
|
<span className="text-muted-foreground"> — {failure.message}</span>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{error && <p className="text-xs text-destructive">{error}</p>}
|
||||||
|
|
||||||
|
{/* Bottom row, like every other block here: the actions are not part of what you are reading. */}
|
||||||
|
<div className="flex justify-end gap-2">
|
||||||
|
<Button variant="outline" size="sm" onClick={() => void start()} disabled={busy || running}>
|
||||||
|
{running ? <Loader2 className="size-4 animate-spin" /> : <RefreshCw className="size-4" />}
|
||||||
|
{running ? 'Syncing' : state?.finishedAt ? 'Sync again' : 'Sync now'}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</Block>
|
||||||
|
);
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user