derived/ is built from originals/, and you can watch it happen
The mapping: derived/ mirrors originals/ directory for directory, and where originals holds a FILE, derived holds a DIRECTORY of the same name — extension included — with the renditions inside it. Keeping the extension is what makes it total: a.jpg and a.png are different originals and must not share a folder. The inverse is pure string work, so the filesystem is the record and there is no second copy of the truth. Images get a WebP ladder at 320/640/960/1280/1920/2560 plus full.webp, and nothing is ever upscaled — a rung wider than the original would be a soft file that srcset then confidently prefers over the sharp one. full.webp always exists because for a HEIC it is the only browser-readable copy of the picture there is. Video gets av-club's web-1080 and web-720 presets by name, plus a poster. Images are named by width because srcset descriptors are widths; video by height because that is av-club's vocabulary and nobody misreads 1080.mp4. The ladder is fixed here and the SITE chooses from it. This never reads website/src/media/slots.ts — a plugin that knows a website's layout is a plugin for one website, and that boundary is the difference between a template and something fofos-shaped. Each derived folder carries a manifest naming the size and mtime it was built from, so the sync is idempotent and resumable: a run that died halfway is finished by running it again. That is what makes an in-memory job acceptable rather than merely convenient. It builds into <name>.partial and renames, so an interruption can never leave a folder that looks finished. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+354
@@ -0,0 +1,354 @@
|
||||
import { mkdir, readdir, readFile, rename, rm, stat, writeFile } from 'node:fs/promises';
|
||||
import { join, relative } from 'node:path';
|
||||
import { sidecarPort } from '@@/sidecar-registry';
|
||||
import { mediaPath } from './media';
|
||||
|
||||
/**
|
||||
* Building `derived/` from `originals/`.
|
||||
*
|
||||
* ── The mapping ──
|
||||
*
|
||||
* `derived/` mirrors `originals/` directory for directory. Where originals holds a FILE, derived holds
|
||||
* a DIRECTORY of the same name — extension included — and the renditions live inside it:
|
||||
*
|
||||
* originals/homepage_galeria01.jpg → derived/homepage_galeria01.jpg/320.webp
|
||||
* derived/homepage_galeria01.jpg/640.webp …
|
||||
* originals/ABOUT US/portrait.HEIC → derived/ABOUT US/portrait.HEIC/…
|
||||
* originals/homepage_video.mp4 → derived/homepage_video.mp4/1080.mp4
|
||||
* derived/homepage_video.mp4/poster.webp
|
||||
*
|
||||
* Keeping the extension is what makes the mapping total: `a.jpg` and `a.png` are different originals
|
||||
* and must not share a directory. It also makes the inverse pure string work — no index, no table, no
|
||||
* second copy of the truth. **The filesystem is the record.**
|
||||
*
|
||||
* ── The ladder is fixed here, and the SITE chooses from it ──
|
||||
*
|
||||
* The widths below cover every slot fofos has at up to 3× pixel density — `website/src/media/slots.ts`
|
||||
* puts the largest at 1232 CSS px, so `minUpload` is 2464 and 2560 is the top rung. But this file does
|
||||
* not read that one, and must not: a plugin that knows a website's layout is a plugin for one website.
|
||||
* It emits a ladder; `srcset` and the `sizes` already written in `website/src/media/render.ts` do the
|
||||
* choosing, because the site is the only thing that knows what space an image occupies.
|
||||
*
|
||||
* That boundary is the difference between this being a template and being fofos-shaped.
|
||||
*
|
||||
* ── Nothing is ever upscaled ──
|
||||
*
|
||||
* A rung wider than the original is not written. Inventing a 2560 from a 900px photograph produces a
|
||||
* soft file that `srcset` would then confidently prefer over the sharp one — worse than not having it.
|
||||
*
|
||||
* ── Video rungs are named by height, images by width ──
|
||||
*
|
||||
* Deliberate, not an oversight. `srcset` descriptors are widths, so an image file called `1280` has to
|
||||
* be 1280 pixels wide or the browser is being lied to. Video has no such mechanism, and every person
|
||||
* reading `1080.mp4` knows what it means. Each unit matches its consumer.
|
||||
*
|
||||
* ── Every byte is written by av-club ──
|
||||
*
|
||||
* This module decides WHAT should exist and av-club decides how to make it. The video presets are its
|
||||
* `web-1080` / `web-720` / `poster`, by name — the encoder flags behind them are its business, and
|
||||
* revising them for every site at once is the reason it is a separate service.
|
||||
*/
|
||||
|
||||
/** Image rungs, in pixels of width. Rungs wider than the original are skipped — see above. */
|
||||
export const LADDER = [320, 640, 960, 1280, 1920, 2560] as const;
|
||||
|
||||
/** WebP quality. 82 is the usual knee: visually clean on photographs, well off the top of the curve. */
|
||||
const WEBP_QUALITY = 82;
|
||||
|
||||
/** Video renditions, as av-club preset names paired with the source height each one needs to be worth making. */
|
||||
const VIDEO_RUNGS = [
|
||||
{ file: '1080.mp4', preset: 'web-1080', minSourceHeight: 720 },
|
||||
{ file: '720.mp4', preset: 'web-720', minSourceHeight: 0 },
|
||||
] as const;
|
||||
|
||||
const IMAGE_EXT = new Set(['jpg', 'jpeg', 'png', 'gif', 'webp', 'bmp', 'avif', 'tif', 'tiff', 'heic', 'heif']);
|
||||
const VIDEO_EXT = new Set(['mp4', 'mov', 'm4v', 'webm', 'mkv', 'avi', 'wmv', 'flv', 'mpg', 'mpeg', '3gp', 'm2ts', 'mts']);
|
||||
|
||||
export type Kind = 'image' | 'video';
|
||||
|
||||
export type Variant = {
|
||||
file: string;
|
||||
width: number | null;
|
||||
height: number | null;
|
||||
bytes: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* What a derived directory says about itself.
|
||||
*
|
||||
* Written last, and it is also the idempotency record: a source whose size and mtime still match the
|
||||
* one recorded here has nothing to redo. That is what makes the initial sync resumable after a
|
||||
* restart, and what makes the watcher's per-file call the same code path as the full pass.
|
||||
*
|
||||
* It is a manifest rather than a database row for the same reason the mapping is string work: a
|
||||
* `derived/` you can delete and rebuild must not leave a row behind claiming it is still there.
|
||||
*/
|
||||
export type Manifest = {
|
||||
kind: Kind;
|
||||
source: { name: string; bytes: number; mtimeMs: number; width: number | null; height: number | null };
|
||||
variants: Variant[];
|
||||
/** ISO. For a person reading the folder, not for the code. */
|
||||
builtAt: string;
|
||||
};
|
||||
|
||||
const MANIFEST = 'manifest.json';
|
||||
|
||||
export const extensionOf = (name: string): string => {
|
||||
const dot = name.lastIndexOf('.');
|
||||
return dot <= 0 ? '' : name.slice(dot + 1).toLowerCase();
|
||||
};
|
||||
|
||||
export const kindOf = (name: string): Kind | null => {
|
||||
const ext = extensionOf(name);
|
||||
if (IMAGE_EXT.has(ext)) return 'image';
|
||||
if (VIDEO_EXT.has(ext)) return 'video';
|
||||
return null;
|
||||
};
|
||||
|
||||
export const originalsDir = async (email: string): Promise<string> => join(await mediaPath(email), 'originals');
|
||||
export const derivedDir = async (email: string): Promise<string> => join(await mediaPath(email), 'derived');
|
||||
|
||||
/**
|
||||
* Talk to av-club.
|
||||
*
|
||||
* Straight to the sidecar's loopback port rather than back through the platform's own HTTP: this code
|
||||
* IS the platform, and making it authenticate to itself to reach a service already on the machine adds
|
||||
* a credential and a hop to buy nothing. `x-officer-user` is the sidecar's whole authentication and is
|
||||
* exactly what its proxy would have sent — see `src/servers/api/file-browser/plugin-file-types.ts`,
|
||||
* which reaches it the same way and explains why that is sound.
|
||||
*/
|
||||
class AvClubUnavailable extends Error {}
|
||||
|
||||
async function avClub<T>(path: string, email: string, init?: RequestInit): Promise<T> {
|
||||
const port = sidecarPort('av-club');
|
||||
if (!port) throw new AvClubUnavailable('av-club is not running. Install and start the AV Club plugin.');
|
||||
|
||||
const response = await fetch(`http://127.0.0.1:${port}${path}`, {
|
||||
...init,
|
||||
headers: { 'x-officer-user': email, ...(init?.body ? { 'content-type': 'application/json' } : {}), ...init?.headers },
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const body = (await response.json().catch(() => null)) as { error?: { message?: string } } | null;
|
||||
throw new Error(body?.error?.message ?? `av-club answered ${response.status}`);
|
||||
}
|
||||
return (await response.json()) as T;
|
||||
}
|
||||
|
||||
const post = <T>(path: string, email: string, body: unknown): Promise<T> =>
|
||||
avClub<T>(path, email, { method: 'POST', body: JSON.stringify(body) });
|
||||
|
||||
/** Is this derived directory already correct for this original? */
|
||||
async function upToDate(dir: string, source: { bytes: number; mtimeMs: number }): Promise<boolean> {
|
||||
const raw = await readFile(join(dir, MANIFEST), 'utf8').catch(() => null);
|
||||
if (!raw) return false;
|
||||
try {
|
||||
const manifest = JSON.parse(raw) as Manifest;
|
||||
return manifest.source.bytes === source.bytes && manifest.source.mtimeMs === source.mtimeMs;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build one original's directory of renditions.
|
||||
*
|
||||
* ── It builds beside, then swaps ──
|
||||
*
|
||||
* Into `<name>.partial`, renamed over the real one only once everything succeeded. A sync interrupted
|
||||
* halfway — a restart, a kill, ffmpeg dying on one file — must never leave a directory that LOOKS
|
||||
* finished, because the manifest is what everything else trusts. Half a ladder with a manifest is
|
||||
* worse than no ladder at all.
|
||||
*/
|
||||
export async function deriveOne(
|
||||
sourcePath: string,
|
||||
targetDir: string,
|
||||
email: string,
|
||||
options: { force?: boolean } = {},
|
||||
): Promise<{ built: boolean; manifest: Manifest }> {
|
||||
const name = sourcePath.slice(sourcePath.lastIndexOf('/') + 1);
|
||||
const kind = kindOf(name);
|
||||
if (!kind) throw new Error(`${name} is neither an image nor a video this understands.`);
|
||||
|
||||
const info = await stat(sourcePath);
|
||||
const source = { bytes: info.size, mtimeMs: Math.round(info.mtimeMs) };
|
||||
|
||||
if (!options.force && (await upToDate(targetDir, source))) {
|
||||
const manifest = JSON.parse(await readFile(join(targetDir, MANIFEST), 'utf8')) as Manifest;
|
||||
return { built: false, manifest };
|
||||
}
|
||||
|
||||
const staging = `${targetDir}.partial`;
|
||||
await rm(staging, { recursive: true, force: true });
|
||||
await mkdir(staging, { recursive: true });
|
||||
|
||||
try {
|
||||
const manifest =
|
||||
kind === 'image'
|
||||
? await buildImage(sourcePath, staging, name, source, email)
|
||||
: await buildVideo(sourcePath, staging, name, source, email);
|
||||
|
||||
await writeFile(join(staging, MANIFEST), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8');
|
||||
|
||||
// `rename` cannot replace a non-empty directory, so the old one goes first. The window between the
|
||||
// two is the reason this is not the step that produces the files.
|
||||
await rm(targetDir, { recursive: true, force: true });
|
||||
await rename(staging, targetDir);
|
||||
|
||||
return { built: true, manifest };
|
||||
} catch (error) {
|
||||
await rm(staging, { recursive: true, force: true }).catch(() => {});
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function buildImage(
|
||||
sourcePath: string,
|
||||
staging: string,
|
||||
name: string,
|
||||
source: { bytes: number; mtimeMs: number },
|
||||
email: string,
|
||||
): Promise<Manifest> {
|
||||
const meta = await avClub<{ width: number | null; height: number | null }>(
|
||||
`/v1/image?path=${encodeURIComponent(sourcePath)}`,
|
||||
email,
|
||||
);
|
||||
const width = meta.width ?? 0;
|
||||
const height = meta.height ?? 0;
|
||||
if (!width || !height) throw new Error(`Could not read the dimensions of ${name}.`);
|
||||
|
||||
const variants: Variant[] = [];
|
||||
|
||||
/*
|
||||
`full.webp` first, and always.
|
||||
|
||||
It is the top of the ladder, and for a HEIC it is the only browser-readable copy of the picture that
|
||||
exists at all — the original renders nowhere. A lightbox, a zoom, or handing the client their own
|
||||
photograph back all need it.
|
||||
*/
|
||||
variants.push(await encodeWebp(sourcePath, join(staging, 'full.webp'), null, email, width, height));
|
||||
|
||||
// Strictly less than the original: a rung at or above it would duplicate `full.webp` byte for byte.
|
||||
for (const rung of LADDER) {
|
||||
if (rung >= width) continue;
|
||||
variants.push(await encodeWebp(sourcePath, join(staging, `${rung}.webp`), rung, email, width, height));
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'image',
|
||||
source: { name, bytes: source.bytes, mtimeMs: source.mtimeMs, width, height },
|
||||
variants,
|
||||
builtAt: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
async function encodeWebp(
|
||||
sourcePath: string,
|
||||
outputPath: string,
|
||||
rung: number | null,
|
||||
email: string,
|
||||
sourceWidth: number,
|
||||
sourceHeight: number,
|
||||
): Promise<Variant> {
|
||||
const result = await post<{ path: string; bytes: number }>('/v1/sharp', email, {
|
||||
inputPath: sourcePath,
|
||||
outputPath,
|
||||
// `rotate` with no angle applies the EXIF orientation, which phone photographs rely on entirely. A
|
||||
// ladder built without it is a set of sideways pictures.
|
||||
pipeline: [
|
||||
{ rotate: {} },
|
||||
...(rung === null ? [] : [{ resize: { width: rung, fit: 'inside', withoutEnlargement: true } }]),
|
||||
],
|
||||
output: { webp: { quality: WEBP_QUALITY } },
|
||||
});
|
||||
|
||||
const width = rung ?? sourceWidth;
|
||||
return {
|
||||
file: outputPath.slice(outputPath.lastIndexOf('/') + 1),
|
||||
width,
|
||||
height: Math.round((width / sourceWidth) * sourceHeight),
|
||||
bytes: result.bytes,
|
||||
};
|
||||
}
|
||||
|
||||
async function buildVideo(
|
||||
sourcePath: string,
|
||||
staging: string,
|
||||
name: string,
|
||||
source: { bytes: number; mtimeMs: number },
|
||||
email: string,
|
||||
): Promise<Manifest> {
|
||||
const probe = await avClub<{ streams?: Array<{ codec_type?: string; width?: number; height?: number }> }>(
|
||||
`/v1/probe?path=${encodeURIComponent(sourcePath)}`,
|
||||
email,
|
||||
);
|
||||
const video = probe.streams?.find((stream) => stream.codec_type === 'video');
|
||||
const height = video?.height ?? 0;
|
||||
const width = video?.width ?? 0;
|
||||
|
||||
const variants: Variant[] = [];
|
||||
|
||||
for (const rung of VIDEO_RUNGS) {
|
||||
// A 720p source re-encoded as "1080" is bigger and no better. The lowest rung has no floor, so
|
||||
// every video gets at least one rendition however small it started.
|
||||
if (height && height < rung.minSourceHeight) continue;
|
||||
const result = await post<{ path: string; bytes: number }>('/v1/ffmpeg', email, {
|
||||
inputPath: sourcePath,
|
||||
outputPath: join(staging, rung.file),
|
||||
preset: rung.preset,
|
||||
});
|
||||
variants.push({ file: rung.file, width: null, height: null, bytes: result.bytes });
|
||||
}
|
||||
|
||||
/*
|
||||
The poster, in two steps.
|
||||
|
||||
av-club's `poster` preset writes a JPEG, which is its call and the right one for a general service.
|
||||
Here everything a browser is handed is WebP, so it goes through sharp on the way out and the JPEG
|
||||
is dropped. Two calls per video, once, at build time.
|
||||
*/
|
||||
const stillJpg = join(staging, 'poster.jpg');
|
||||
await post('/v1/ffmpeg', email, {
|
||||
inputPath: sourcePath,
|
||||
outputPath: stillJpg,
|
||||
preset: 'poster',
|
||||
// A frame from a second in, because frame zero of a great many videos is black.
|
||||
inputOptions: { ss: 1 },
|
||||
}).catch(() => post('/v1/ffmpeg', email, { inputPath: sourcePath, outputPath: stillJpg, preset: 'poster' }));
|
||||
|
||||
const poster = await post<{ path: string; bytes: number }>('/v1/sharp', email, {
|
||||
inputPath: stillJpg,
|
||||
outputPath: join(staging, 'poster.webp'),
|
||||
output: { webp: { quality: WEBP_QUALITY } },
|
||||
});
|
||||
await rm(stillJpg, { force: true });
|
||||
variants.push({ file: 'poster.webp', width: null, height: null, bytes: poster.bytes });
|
||||
|
||||
return {
|
||||
kind: 'video',
|
||||
source: { name, bytes: source.bytes, mtimeMs: source.mtimeMs, width: width || null, height: height || null },
|
||||
variants,
|
||||
builtAt: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Every file under `originals/`, relative to it, depth first. Directories are structure, not work. */
|
||||
export async function listOriginals(root: string, prefix = ''): Promise<string[]> {
|
||||
const entries = await readdir(join(root, prefix), { withFileTypes: true }).catch(() => []);
|
||||
const found: string[] = [];
|
||||
|
||||
for (const entry of entries) {
|
||||
// Dotfiles are `.DS_Store` and friends. Nothing under here is a deliberate hidden file.
|
||||
if (entry.name.startsWith('.')) continue;
|
||||
const rel = prefix ? `${prefix}/${entry.name}` : entry.name;
|
||||
if (entry.isDirectory()) found.push(...(await listOriginals(root, rel)));
|
||||
else if (kindOf(entry.name)) found.push(rel);
|
||||
}
|
||||
|
||||
return found;
|
||||
}
|
||||
|
||||
/** Where a relative original ends up. Exported because the watcher needs the same answer. */
|
||||
export const targetFor = (derivedRoot: string, relativePath: string): string => join(derivedRoot, relativePath);
|
||||
|
||||
export { relative, AvClubUnavailable };
|
||||
@@ -5,6 +5,7 @@ import { deleteConnection, readConnection, readCredentials, recordProbe, saveCon
|
||||
import { readSettings, saveRootPath } from './settings';
|
||||
import { readWebsiteStatus, cloneWebsite, updateWebsite } from './website';
|
||||
import { readMediaStatus, createMediaDir } from './media';
|
||||
import { readSync, startSync } from './sync';
|
||||
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
|
||||
|
||||
/**
|
||||
@@ -228,6 +229,28 @@ router.post('/_config/media/create', async (ctx) => {
|
||||
return ctx.json({ ...result, status: await readMediaStatus(user.email) }, result.ok ? 200 : 400);
|
||||
});
|
||||
|
||||
/**
|
||||
* Building `derived/` from `originals/`.
|
||||
*
|
||||
* Two routes and no more: start one, and ask how it is going. The work is asynchronous because a
|
||||
* folder of photographs takes minutes, and the panel polls the second route rather than holding a
|
||||
* request open — a connection that has to survive an eight-minute encode is a connection that will
|
||||
* not.
|
||||
*
|
||||
* `POST` answers 200 whether it started a run or found one already going, and says which in `started`.
|
||||
* "Already running" is not a failure to report to somebody who pressed Sync; the honest answer is the
|
||||
* run that is in progress, which is what comes back.
|
||||
*/
|
||||
router.post('/_config/media/sync', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
return ctx.json(startSync(user.email));
|
||||
});
|
||||
|
||||
router.get('/_config/media/sync', async (ctx) => {
|
||||
const user = ctx.get('user');
|
||||
return ctx.json(readSync(user.email));
|
||||
});
|
||||
|
||||
/**
|
||||
* Is this plugin usable? Asked once by the detail panel rather than by all nine sections.
|
||||
*
|
||||
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
import { mkdir } from 'node:fs/promises';
|
||||
import { dirname } from 'node:path';
|
||||
import { deriveOne, derivedDir, listOriginals, originalsDir, targetFor } from './derive';
|
||||
|
||||
/**
|
||||
* The sync: bring `derived/` into line with `originals/`.
|
||||
*
|
||||
* ── One job, in memory, per user ──
|
||||
*
|
||||
* Not a `pipeline_jobs` row. That table belongs to the task runner, which runs a script in a temp
|
||||
* directory with a cwd and a log — a good fit for what a user wrote and a poor one for a plugin
|
||||
* calling a service in-process. So the state lives here, and the cost is stated plainly: **a restart
|
||||
* loses the job.**
|
||||
*
|
||||
* That cost is small because the work is idempotent. Every derived directory carries a manifest naming
|
||||
* the size and mtime it was built from, so a sync that died halfway is finished by running it again —
|
||||
* it skips everything already correct and picks up where the interruption left it. There is nothing to
|
||||
* reconcile and nothing to clean up, which is the property that made an in-memory job acceptable
|
||||
* rather than merely convenient.
|
||||
*
|
||||
* ── The same code path as a single file ──
|
||||
*
|
||||
* `deriveOne` is what the watcher will call for one upload. This walks and calls it; it decides
|
||||
* nothing about how a file is built. A full sync and a single arrival must never be two implementations
|
||||
* that can disagree about what a derived directory should contain.
|
||||
*
|
||||
* ── Two at a time ──
|
||||
*
|
||||
* av-club runs two jobs concurrently and queues the rest, so a third caller would only wait in its
|
||||
* queue while making this module's progress harder to report. Matching its number keeps both slots
|
||||
* busy and keeps "what is happening now" answerable.
|
||||
*/
|
||||
|
||||
export type SyncError = { file: string; message: string };
|
||||
|
||||
export type SyncState = {
|
||||
running: boolean;
|
||||
/** How many originals the walk found. Null until the walk finishes. */
|
||||
total: number | null;
|
||||
/** Attempted — built, skipped and failed together. What a progress bar divides by `total`. */
|
||||
done: number;
|
||||
built: number;
|
||||
skipped: number;
|
||||
errors: SyncError[];
|
||||
/** The files being worked on right now, for somebody watching. Empty when idle. */
|
||||
current: string[];
|
||||
startedAt: string | null;
|
||||
finishedAt: string | null;
|
||||
/** Set when the run stopped for a reason that was not about any single file — av-club being down. */
|
||||
fatal: string | null;
|
||||
};
|
||||
|
||||
const CONCURRENCY = 2;
|
||||
|
||||
const idle = (): SyncState => ({
|
||||
running: false,
|
||||
total: null,
|
||||
done: 0,
|
||||
built: 0,
|
||||
skipped: 0,
|
||||
errors: [],
|
||||
current: [],
|
||||
startedAt: null,
|
||||
finishedAt: null,
|
||||
fatal: null,
|
||||
});
|
||||
|
||||
const states = new Map<string, SyncState>();
|
||||
|
||||
export const readSync = (email: string): SyncState => states.get(email) ?? idle();
|
||||
|
||||
/**
|
||||
* Start one, unless one is already running.
|
||||
*
|
||||
* Returns the state either way, so the caller does not have to decide what "already running" means to
|
||||
* a person: the panel shows the run that is in progress, which is the honest answer to "sync now".
|
||||
*/
|
||||
export function startSync(email: string): { started: boolean; state: SyncState } {
|
||||
const existing = states.get(email);
|
||||
if (existing?.running) return { started: false, state: existing };
|
||||
|
||||
const state: SyncState = { ...idle(), running: true, startedAt: new Date().toISOString() };
|
||||
states.set(email, state);
|
||||
|
||||
// Deliberately not awaited. The route answers immediately with the job's opening state and the panel
|
||||
// polls; an await here would be the request timing out on a folder of any size.
|
||||
void run(email, state);
|
||||
|
||||
return { started: true, state };
|
||||
}
|
||||
|
||||
async function run(email: string, state: SyncState): Promise<void> {
|
||||
try {
|
||||
const from = await originalsDir(email);
|
||||
const to = await derivedDir(email);
|
||||
|
||||
const files = await listOriginals(from);
|
||||
state.total = files.length;
|
||||
|
||||
let next = 0;
|
||||
const worker = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
const index = next++;
|
||||
const file = files[index];
|
||||
if (file === undefined) return;
|
||||
|
||||
state.current.push(file);
|
||||
try {
|
||||
const target = targetFor(to, file);
|
||||
await mkdir(dirname(target), { recursive: true });
|
||||
const { built } = await deriveOne(`${from}/${file}`, target, email);
|
||||
if (built) state.built++;
|
||||
else state.skipped++;
|
||||
} catch (error) {
|
||||
const message = (error as Error).message;
|
||||
/*
|
||||
av-club being unreachable is not a fact about this file, and recording it once per file
|
||||
would bury the one thing worth reading under four hundred copies of it. Stop the run and
|
||||
say so once.
|
||||
*/
|
||||
if (message.includes('av-club is not running')) {
|
||||
state.fatal = message;
|
||||
return;
|
||||
}
|
||||
state.errors.push({ file, message });
|
||||
} finally {
|
||||
state.current = state.current.filter((name) => name !== file);
|
||||
state.done++;
|
||||
}
|
||||
|
||||
if (state.fatal) return;
|
||||
}
|
||||
};
|
||||
|
||||
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
|
||||
} catch (error) {
|
||||
state.fatal = (error as Error).message;
|
||||
} finally {
|
||||
state.running = false;
|
||||
state.current = [];
|
||||
state.finishedAt = new Date().toISOString();
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user