fofos/fofos-admin
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Sync says what it is doing, file by file

Browse Source 
The counters said how MUCH and never which file, so a run that was working normally and one that had
wedged produced the same screen: `done` stops moving either way. A photograph is around thirty encodes
and a video is three plus a poster ladder, and none of that was visible from outside.

So the run keeps a transcript and the button opens it.

Every rung reports itself by CONSTRUCTION. Ten `variants.push` call sites became one `add`, which is
the point — a rung added later is described without anybody remembering to describe it. The slow
stages that produce nothing until they finish say so BEFORE they start instead: the HEIC decode, each
video rung, the poster frame. Those are the pauses that read as a hang.

`?since=<n>` is what keeps it cheap. The transcript is thousands of lines and polls once a second, so
sending all of it every tick would be the same megabyte over and over; the client keeps what it has
and asks for the rest. No `since` means everything, which is what a modal opened mid-run wants.

The transcript lives in the shared slot, not the modal, so closing it does not throw away the run —
somebody who looks, closes, and looks again is watching one run rather than two. `startedAt` is what
identifies a run, because `n` restarts at 1 and appending across a boundary would interleave two.

Polling, not a socket. Each line transfers once and the events are seconds apart, so a second
transport to open, authenticate and reconnect would buy nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-31 12:16:52 +01:00
co-authored by Claude Opus 5
parent 78db209ea1
commit ddcba1c95d
6 changed files with 442 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/derive.ts
+54 -13
View File
@@ -538,12 +538,25 @@ async function topUp(dir: string, manifest: Manifest, email: string): Promise<Ma
* finished, because the manifest is what everything else trusts. Half a ladder with a manifest is
* worse than no ladder at all.
*/
/** Told what just happened, rung by rung. See `onStep` — a no-op unless somebody is watching. */
export type Step = (message: string) => void;
export async function deriveOne(
sourcePath: string,
targetDir: string,
email: string,
options: { force?: boolean } = {},
options: { force?: boolean; onStep?: Step } = {},
): Promise<{ built: boolean; manifest: Manifest }> {
/*
Say what is happening, if anybody asked.
A no-op by default, so nothing here depends on being watched — the sync passes one through to fill
its transcript; the watcher and the tests pass nothing. One photograph is around thirty encodes and
a video is three plus a poster ladder, so without this a run that is working normally and a run
that is wedged look identical from outside.
*/
const step = options.onStep ?? (() => {});
const name = sourcePath.slice(sourcePath.lastIndexOf('/') + 1);
const kind = kindOf(name);
if (!kind) throw new Error(`${name} is neither an image nor a video this understands.`);
@@ -563,8 +576,8 @@ export async function deriveOne(
try {
const manifest =
kind === 'image'
? await buildImage(sourcePath, staging, name, source, email)
: await buildVideo(sourcePath, staging, name, source, email);
? await buildImage(sourcePath, staging, name, source, email, step)
: await buildVideo(sourcePath, staging, name, source, email, step);
await writeFile(join(staging, MANIFEST), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8');
@@ -596,6 +609,7 @@ async function buildImage(
name: string,
source: { bytes: number; mtimeMs: number },
email: string,
step: Step,
): Promise<Manifest> {
const meta = await avClub<{ width: number | null; height: number | null; needsConversion: boolean }>(
`/v1/image?path=${encodeURIComponent(sourcePath)}`,
@@ -624,6 +638,9 @@ async function buildImage(
*/
const master = meta.needsConversion ? join(staging, MASTER) : sourcePath;
if (meta.needsConversion) {
// Announced BEFORE it happens, unlike a rung. This is the 8.5-second one, it produces nothing
// anybody sees, and a transcript that went quiet here is the exact case this exists to explain.
step(`decoding ${name} once (${width}x${height})`);
await post('/v1/sharp', email, {
inputPath: sourcePath,
outputPath: master,
@@ -639,7 +656,17 @@ async function buildImage(
*/
const lossless = await prefersLossless(master, staging, email, width);
/*
Collect and announce, so a rung cannot be added without being described.
Ten `variants.push` call sites became one `add`, which is the point: a rung added later reports
itself by construction rather than because somebody remembered to.
*/
const variants: Variant[] = [];
const add = (variant: Variant): void => {
variants.push(variant);
step(`${variant.file} · ${(variant.bytes / 1024).toFixed(0)} KB`);
};
/*
`full.webp` first, and always.
@@ -648,7 +675,7 @@ async function buildImage(
exists at all — the original renders nowhere. A lightbox, a zoom, or handing the client their own
photograph back all need it.
*/
variants.push(await encodeWebp(master, join(staging, 'full.webp'), null, email, width, height, lossless));
add(await encodeWebp(master, join(staging, 'full.webp'), null, email, width, height, lossless));
/*
AVIF beside every rung — except for a picture that wanted the lossless path.
@@ -660,13 +687,13 @@ async function buildImage(
*/
const buildAvif = !lossless;
if (buildAvif) variants.push(await encodeAvif(master, join(staging, 'full.avif'), null, email, width, height));
if (buildAvif) add(await encodeAvif(master, join(staging, 'full.avif'), null, email, width, height));
// Strictly less than the original: a rung at or above it would duplicate `full.webp` byte for byte.
for (const rung of LADDER) {
if (rung >= width) continue;
variants.push(await encodeWebp(master, join(staging, `${rung}.webp`), rung, email, width, height, lossless));
if (buildAvif) variants.push(await encodeAvif(master, join(staging, `${rung}.avif`), rung, email, width, height));
add(await encodeWebp(master, join(staging, `${rung}.webp`), rung, email, width, height, lossless));
if (buildAvif) add(await encodeAvif(master, join(staging, `${rung}.avif`), rung, email, width, height));
}
/*
@@ -686,8 +713,8 @@ async function buildImage(
if (!squareEnough) {
for (const rung of SQUARE_LADDER) {
if (rung > Math.min(width, height)) continue;
variants.push(await encodeSquare(master, join(staging, `${rung}sq.webp`), rung, email, lossless));
if (buildAvif) variants.push(await encodeSquareAvif(master, join(staging, `${rung}sq.avif`), rung, email));
add(await encodeSquare(master, join(staging, `${rung}sq.webp`), rung, email, lossless));
if (buildAvif) add(await encodeSquareAvif(master, join(staging, `${rung}sq.avif`), rung, email));
}
}
@@ -870,6 +897,7 @@ async function buildVideo(
name: string,
source: { bytes: number; mtimeMs: number },
email: string,
step: Step,
): Promise<Manifest> {
const probe = await avClub<{ streams?: Array<{ codec_type?: string; width?: number; height?: number }> }>(
`/v1/probe?path=${encodeURIComponent(sourcePath)}`,
@@ -879,7 +907,17 @@ async function buildVideo(
const height = video?.height ?? 0;
const width = video?.width ?? 0;
/*
Collect and announce, so a rung cannot be added without being described.
Ten `variants.push` call sites became one `add`, which is the point: a rung added later reports
itself by construction rather than because somebody remembered to.
*/
const variants: Variant[] = [];
const add = (variant: Variant): void => {
variants.push(variant);
step(`${variant.file} · ${(variant.bytes / 1024).toFixed(0)} KB`);
};
/*
Every rung the source is actually tall enough for, and the floor when it is tall enough for none.
@@ -892,6 +930,8 @@ async function buildVideo(
const rungs = wanted.length ? wanted : [VIDEO_FLOOR];
for (const rung of rungs) {
// Before, because this is minutes and the line after it is the only other one this loop prints.
step(`encoding ${rung.file}`);
const result = await post<{ path: string; bytes: number }>('/v1/ffmpeg', email, {
inputPath: sourcePath,
outputPath: join(staging, rung.file),
@@ -900,7 +940,7 @@ async function buildVideo(
// see sidecar/ffmpeg/run.ts. Used only to lighten the audio on the 480 rung.
...(rung.options ? { options: rung.options } : {}),
});
variants.push({ file: rung.file, width: null, height: null, bytes: result.bytes });
add({ file: rung.file, width: null, height: null, bytes: result.bytes });
}
/*
@@ -911,6 +951,7 @@ async function buildVideo(
is dropped. Two calls per video, once, at build time.
*/
const stillJpg = join(staging, 'poster.jpg');
step('extracting the poster frame');
await post('/v1/ffmpeg', email, {
inputPath: sourcePath,
outputPath: stillJpg,
@@ -933,7 +974,7 @@ async function buildVideo(
order rather than the other one.
*/
const placeholder = await encodePlaceholder(stillJpg, join(staging, 'placeholder.webp'), email);
variants.push({ file: 'poster.webp', width: width || null, height: height || null, bytes: poster.bytes });
add({ file: 'poster.webp', width: width || null, height: height || null, bytes: poster.bytes });
/*
The poster's own ladder, cut from the same still.
@@ -954,7 +995,7 @@ async function buildVideo(
pipeline: [{ resize: { width: rung, fit: 'inside', withoutEnlargement: true } }],
output: { format: 'webp', options: { quality: WEBP_QUALITY_POSTER } },
});
variants.push({
add({
file: `poster-${rung}.webp`,
width: rung,
height: height ? Math.round((rung / width) * height) : null,
@@ -969,7 +1010,7 @@ async function buildVideo(
pipeline: [{ resize: { width: rung, fit: 'inside', withoutEnlargement: true } }],
output: { format: 'avif', options: { quality: avifQualityFor(rung) } },
});
variants.push({
add({
file: `poster-${rung}.avif`,
width: rung,
height: height ? Math.round((rung / width) * height) : null,