decode once, encode seven times, and one file at a time
A HEIC was being decoded once per rung. av-club is stateless per request, so
seven rungs asked for from the original are seven ImageMagick conversions of it
— measured on fofos at 8.5 s each, against 1.1 s for the resize and encode that
follow. Six of the seven produced a pixel-identical intermediate and threw it
away: about 67 seconds per photograph where 16 would do, paid again on every
future upload.
So when av-club says a format needs converting, convert it ONCE into the staging
directory and cut every rung from that. `needsConversion` is asked rather than
guessed, because which formats are expensive is av-club's fact and a list of
them kept here would go on paying after its build stopped needing it. JPEG and
PNG are untouched — writing a full-resolution intermediate to save six cheap
reads would cost more than it saved.
Concurrency drops to one. Two was chosen to match av-club and is the right
number on a machine with room; fofos has 1.87 GiB of RAM, and two concurrent
12-megapixel decodes left 520 MiB free with 648 MB already paged out. The work
is CPU-bound on a box already at 99%, so the second worker was not buying
throughput — it was buying a second resident bitmap.
And the sharp calls now name their output format properly. `output` takes
`{ format, options }`; passing `{ webp: { … } }` was silently ignored, so
everything built before this encoded at sharp's default quality rather than the
82 asked for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+11
-5
@@ -24,11 +24,16 @@ import { deriveOne, derivedDir, listOriginals, originalsDir, targetFor } from '.
|
||||
* nothing about how a file is built. A full sync and a single arrival must never be two implementations
|
||||
* that can disagree about what a derived directory should contain.
|
||||
*
|
||||
* ── Two at a time ──
|
||||
* ── One at a time ──
|
||||
*
|
||||
* av-club runs two jobs concurrently and queues the rest, so a third caller would only wait in its
|
||||
* queue while making this module's progress harder to report. Matching its number keeps both slots
|
||||
* busy and keeps "what is happening now" answerable.
|
||||
* It was two, to match av-club's own concurrency. That is the right number on a machine with room and
|
||||
* the wrong one here: converting a 12-megapixel HEIC holds a full-resolution bitmap, and fofos has
|
||||
* 1.87 GiB of RAM. Two at once left about 520 MiB free with 648 MB already paged out — CPU pinned at
|
||||
* 99%, and an out-of-memory kill one large photograph away.
|
||||
*
|
||||
* Nothing is lost by going single-file. The work is CPU-bound on a box that was already saturated, so
|
||||
* the second worker was not buying throughput; it was buying a second resident bitmap. It also makes
|
||||
* "what is happening now" one filename instead of two, which is the more useful thing to read.
|
||||
*/
|
||||
|
||||
export type SyncError = { file: string; message: string };
|
||||
@@ -50,7 +55,8 @@ export type SyncState = {
|
||||
fatal: string | null;
|
||||
};
|
||||
|
||||
const CONCURRENCY = 2;
|
||||
/** One. See the note above — this is a memory limit, not a politeness. */
|
||||
const CONCURRENCY = 1;
|
||||
|
||||
const idle = (): SyncState => ({
|
||||
running: false,
|
||||
|
||||
Reference in New Issue
Block a user