The counters said how MUCH and never which file, so a run that was working normally and one that had
wedged produced the same screen: `done` stops moving either way. A photograph is around thirty encodes
and a video is three plus a poster ladder, and none of that was visible from outside.
So the run keeps a transcript and the button opens it.
Every rung reports itself by CONSTRUCTION. Ten `variants.push` call sites became one `add`, which is
the point — a rung added later is described without anybody remembering to describe it. The slow
stages that produce nothing until they finish say so BEFORE they start instead: the HEIC decode, each
video rung, the poster frame. Those are the pauses that read as a hang.
`?since=<n>` is what keeps it cheap. The transcript is thousands of lines and polls once a second, so
sending all of it every tick would be the same megabyte over and over; the client keeps what it has
and asks for the rest. No `since` means everything, which is what a modal opened mid-run wants.
The transcript lives in the shared slot, not the modal, so closing it does not throw away the run —
somebody who looks, closes, and looks again is watching one run rather than two. `startedAt` is what
identifies a run, because `n` restarts at 1 and appending across a boundary would interleave two.
Polling, not a socket. Each line transfers once and the events are seconds apart, so a second
transport to open, authenticate and reconnect would buy nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moving a file into a folder made the sync look at a new path, find nothing, and
convert it again — minutes of work to produce bytes that already existed a
directory away — while the old folder stayed behind forever, because nothing ever
deleted one.
A move preserves size and mtime, and both are already in the manifest and in the
stat. So the sync indexes what is already derived on that pair, and a file that
has moved has its ladder RENAMED across.
An ambiguous match is refused rather than guessed. Two originals can share a size
and an mtime — a camera dump can write a whole card in one second — and
rebuilding is merely slow where attaching the wrong photographs to a name is
wrong.
Orphans are removed afterwards, and only on a WHOLE-TREE sync. A scoped run knows
about one folder, so everything outside it looks abandoned; deleting on that
basis would take out the library the first time somebody re-synced a subfolder.
Safe otherwise because derived/ is reproducible by definition, which is what its
name has meant since these two folders stopped being called `user` and `treated`.
Both are counted, recorded in the ledger and shown only when they happened, so an
ordinary run stays one short line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things the panel could not answer. A run's state lives in a Map, so a
restart erased it — and with it any record that a file FAILED. That is the gap
the manifests structurally cannot cover: a derived folder records the size,
mtime and variants it was built from, so provenance is already precise, but a
file that did not build has no folder and leaves no trace. Once the watcher is
running, a failure at three in the morning would never be discovered.
So one JSON line per run, appended to media/.sync-log.jsonl. A file rather than
a table because db/schema.ts is emphatic that the connection is the only thing
this plugin stores, and a ledger is losable — re-run and it rebuilds. Beside
derived/ rather than inside it, so wiping derived does not erase the history of
what happened before, which is the comparison it exists for. Written in a
finally, so a run that died still records that it did.
And the line above the button now says what is TRUE rather than what a run once
did: reconcile() walks originals, reads each manifest, and reports current,
missing and changed. It uses upToDate — the same predicate the sync uses to skip
a file — because two definitions of "current" is how a panel comes to say
everything is in step while the next run rebuilds half of it. Missing and
changed are separated because only the second means somebody edited something.
The ledger is not polled. It reads the whole tree, so it loads on arrival and
again when a run stops — the only two moments its answer can change.
Sync also takes an optional folder now, so a subtree can be re-synced on its own
rather than the whole library. The scope is recorded on each run, because "3
built" means different things for one folder and for everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A HEIC was being decoded once per rung. av-club is stateless per request, so
seven rungs asked for from the original are seven ImageMagick conversions of it
— measured on fofos at 8.5 s each, against 1.1 s for the resize and encode that
follow. Six of the seven produced a pixel-identical intermediate and threw it
away: about 67 seconds per photograph where 16 would do, paid again on every
future upload.
So when av-club says a format needs converting, convert it ONCE into the staging
directory and cut every rung from that. `needsConversion` is asked rather than
guessed, because which formats are expensive is av-club's fact and a list of
them kept here would go on paying after its build stopped needing it. JPEG and
PNG are untouched — writing a full-resolution intermediate to save six cheap
reads would cost more than it saved.
Concurrency drops to one. Two was chosen to match av-club and is the right
number on a machine with room; fofos has 1.87 GiB of RAM, and two concurrent
12-megapixel decodes left 520 MiB free with 648 MB already paged out. The work
is CPU-bound on a box already at 99%, so the second worker was not buying
throughput — it was buying a second resident bitmap.
And the sharp calls now name their output format properly. `output` takes
`{ format, options }`; passing `{ webp: { … } }` was silently ignored, so
everything built before this encoded at sharp's default quality rather than the
82 asked for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mapping: derived/ mirrors originals/ directory for directory, and where
originals holds a FILE, derived holds a DIRECTORY of the same name — extension
included — with the renditions inside it. Keeping the extension is what makes it
total: a.jpg and a.png are different originals and must not share a folder. The
inverse is pure string work, so the filesystem is the record and there is no
second copy of the truth.
Images get a WebP ladder at 320/640/960/1280/1920/2560 plus full.webp, and
nothing is ever upscaled — a rung wider than the original would be a soft file
that srcset then confidently prefers over the sharp one. full.webp always
exists because for a HEIC it is the only browser-readable copy of the picture
there is.
Video gets av-club's web-1080 and web-720 presets by name, plus a poster. Images
are named by width because srcset descriptors are widths; video by height
because that is av-club's vocabulary and nobody misreads 1080.mp4.
The ladder is fixed here and the SITE chooses from it. This never reads
website/src/media/slots.ts — a plugin that knows a website's layout is a plugin
for one website, and that boundary is the difference between a template and
something fofos-shaped.
Each derived folder carries a manifest naming the size and mtime it was built
from, so the sync is idempotent and resumable: a run that died halfway is
finished by running it again. That is what makes an in-memory job acceptable
rather than merely convenient. It builds into <name>.partial and renames, so an
interruption can never leave a folder that looks finished.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pastilhasandClaude Opus 5