a moved file relocates its ladder instead of rebuilding it, and orphans go

Moving a file into a folder made the sync look at a new path, find nothing, and
convert it again — minutes of work to produce bytes that already existed a
directory away — while the old folder stayed behind forever, because nothing ever
deleted one.

A move preserves size and mtime, and both are already in the manifest and in the
stat. So the sync indexes what is already derived on that pair, and a file that
has moved has its ladder RENAMED across.

An ambiguous match is refused rather than guessed. Two originals can share a size
and an mtime — a camera dump can write a whole card in one second — and
rebuilding is merely slow where attaching the wrong photographs to a name is
wrong.

Orphans are removed afterwards, and only on a WHOLE-TREE sync. A scoped run knows
about one folder, so everything outside it looks abandoned; deleting on that
basis would take out the library the first time somebody re-synced a subfolder.
Safe otherwise because derived/ is reproducible by definition, which is what its
name has meant since these two folders stopped being called `user` and `treated`.

Both are counted, recorded in the ledger and shown only when they happened, so an
ordinary run stays one short line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-29 20:42:20 +01:00
co-authored by Claude Opus 5
parent b634de5604
commit 01652f3480
4 changed files with 193 additions and 4 deletions
+59 -3
View File
@@ -1,6 +1,16 @@
import { mkdir } from 'node:fs/promises';
import { mkdir, stat } from 'node:fs/promises';
import { dirname } from 'node:path';
import { deriveOne, derivedDir, listOriginals, originalsDir, safeScope, targetFor } from './derive';
import {
deriveOne,
derivedDir,
indexDerived,
listOriginals,
originalsDir,
pruneDerived,
relocateDerived,
safeScope,
targetFor,
} from './derive';
import { appendRun, type RunTrigger } from './ledger';
/**
@@ -47,6 +57,10 @@ export type SyncState = {
done: number;
built: number;
skipped: number;
/** Ladders that were RENAMED into place because their original moved, rather than rebuilt. */
moved: number;
/** Derived folders removed because their original is gone. Only on a whole-tree sync. */
pruned: number;
errors: SyncError[];
/** The files being worked on right now, for somebody watching. Empty when idle. */
current: string[];
@@ -73,6 +87,8 @@ const idle = (): SyncState => ({
done: 0,
built: 0,
skipped: 0,
moved: 0,
pruned: 0,
errors: [],
current: [],
startedAt: null,
@@ -123,6 +139,16 @@ async function run(email: string, state: SyncState): Promise<void> {
const files = await listOriginals(from, state.scope);
state.total = files.length;
/*
What is already built, indexed by what it was built FROM.
Built once, before the walk, so a file that has simply MOVED can have its ladder renamed into
place instead of converted again — minutes of work to produce bytes that already exist a
directory away. Size and mtime survive a move and are already in both the manifest and the stat,
so the index costs one pass over derived/ and nothing else.
*/
const derivedIndex = await indexDerived(to);
let next = 0;
const worker = async (): Promise<void> => {
for (;;) {
@@ -134,8 +160,24 @@ async function run(email: string, state: SyncState): Promise<void> {
try {
const target = targetFor(to, file);
await mkdir(dirname(target), { recursive: true });
/*
A move, before a build.
`deriveOne` decides by looking for a manifest at the target, so a file that changed folder
has none there and would convert from scratch. This asks the index whether the same bytes
were already built somewhere else and, if exactly one folder matches, renames it across.
An ambiguous match is refused inside `relocateDerived` — rebuilding is slow, attaching the
wrong photographs to a name is wrong.
*/
const info = await stat(`${from}/${file}`).catch(() => null);
const relocated = info
? await relocateDerived(derivedIndex, to, file, { bytes: info.size, mtimeMs: Math.round(info.mtimeMs) })
: false;
const { built } = await deriveOne(`${from}/${file}`, target, email);
if (built) state.built++;
if (relocated && !built) state.moved++;
else if (built) state.built++;
else state.skipped++;
} catch (error) {
const message = (error as Error).message;
@@ -159,6 +201,18 @@ async function run(email: string, state: SyncState): Promise<void> {
};
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
/*
Then the orphans — derived folders whose original no longer exists.
Only on a WHOLE-TREE sync. A scoped one knows about a single folder, so everything outside it
looks abandoned, and deleting on that basis would take out the library the first time somebody
re-synced one subfolder. Safe otherwise because derived/ is reproducible by definition, which is
what the name has meant since these two folders stopped being called `user` and `treated`.
*/
if (!state.scope && !state.fatal) {
state.pruned = (await pruneDerived(from, to)).length;
}
} catch (error) {
state.fatal = (error as Error).message;
} finally {
@@ -180,6 +234,8 @@ async function run(email: string, state: SyncState): Promise<void> {
total: state.total ?? state.done,
built: state.built,
skipped: state.skipped,
moved: state.moved,
pruned: state.pruned,
errors: state.errors,
fatal: state.fatal,
});