a moved file relocates its ladder instead of rebuilding it, and orphans go
Moving a file into a folder made the sync look at a new path, find nothing, and convert it again — minutes of work to produce bytes that already existed a directory away — while the old folder stayed behind forever, because nothing ever deleted one. A move preserves size and mtime, and both are already in the manifest and in the stat. So the sync indexes what is already derived on that pair, and a file that has moved has its ladder RENAMED across. An ambiguous match is refused rather than guessed. Two originals can share a size and an mtime — a camera dump can write a whole card in one second — and rebuilding is merely slow where attaching the wrong photographs to a name is wrong. Orphans are removed afterwards, and only on a WHOLE-TREE sync. A scoped run knows about one folder, so everything outside it looks abandoned; deleting on that basis would take out the library the first time somebody re-synced a subfolder. Safe otherwise because derived/ is reproducible by definition, which is what its name has meant since these two folders stopped being called `user` and `treated`. Both are counted, recorded in the ledger and shown only when they happened, so an ordinary run stays one short line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+59
-3
@@ -1,6 +1,16 @@
|
||||
import { mkdir } from 'node:fs/promises';
|
||||
import { mkdir, stat } from 'node:fs/promises';
|
||||
import { dirname } from 'node:path';
|
||||
import { deriveOne, derivedDir, listOriginals, originalsDir, safeScope, targetFor } from './derive';
|
||||
import {
|
||||
deriveOne,
|
||||
derivedDir,
|
||||
indexDerived,
|
||||
listOriginals,
|
||||
originalsDir,
|
||||
pruneDerived,
|
||||
relocateDerived,
|
||||
safeScope,
|
||||
targetFor,
|
||||
} from './derive';
|
||||
import { appendRun, type RunTrigger } from './ledger';
|
||||
|
||||
/**
|
||||
@@ -47,6 +57,10 @@ export type SyncState = {
|
||||
done: number;
|
||||
built: number;
|
||||
skipped: number;
|
||||
/** Ladders that were RENAMED into place because their original moved, rather than rebuilt. */
|
||||
moved: number;
|
||||
/** Derived folders removed because their original is gone. Only on a whole-tree sync. */
|
||||
pruned: number;
|
||||
errors: SyncError[];
|
||||
/** The files being worked on right now, for somebody watching. Empty when idle. */
|
||||
current: string[];
|
||||
@@ -73,6 +87,8 @@ const idle = (): SyncState => ({
|
||||
done: 0,
|
||||
built: 0,
|
||||
skipped: 0,
|
||||
moved: 0,
|
||||
pruned: 0,
|
||||
errors: [],
|
||||
current: [],
|
||||
startedAt: null,
|
||||
@@ -123,6 +139,16 @@ async function run(email: string, state: SyncState): Promise<void> {
|
||||
const files = await listOriginals(from, state.scope);
|
||||
state.total = files.length;
|
||||
|
||||
/*
|
||||
What is already built, indexed by what it was built FROM.
|
||||
|
||||
Built once, before the walk, so a file that has simply MOVED can have its ladder renamed into
|
||||
place instead of converted again — minutes of work to produce bytes that already exist a
|
||||
directory away. Size and mtime survive a move and are already in both the manifest and the stat,
|
||||
so the index costs one pass over derived/ and nothing else.
|
||||
*/
|
||||
const derivedIndex = await indexDerived(to);
|
||||
|
||||
let next = 0;
|
||||
const worker = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
@@ -134,8 +160,24 @@ async function run(email: string, state: SyncState): Promise<void> {
|
||||
try {
|
||||
const target = targetFor(to, file);
|
||||
await mkdir(dirname(target), { recursive: true });
|
||||
|
||||
/*
|
||||
A move, before a build.
|
||||
|
||||
`deriveOne` decides by looking for a manifest at the target, so a file that changed folder
|
||||
has none there and would convert from scratch. This asks the index whether the same bytes
|
||||
were already built somewhere else and, if exactly one folder matches, renames it across.
|
||||
An ambiguous match is refused inside `relocateDerived` — rebuilding is slow, attaching the
|
||||
wrong photographs to a name is wrong.
|
||||
*/
|
||||
const info = await stat(`${from}/${file}`).catch(() => null);
|
||||
const relocated = info
|
||||
? await relocateDerived(derivedIndex, to, file, { bytes: info.size, mtimeMs: Math.round(info.mtimeMs) })
|
||||
: false;
|
||||
|
||||
const { built } = await deriveOne(`${from}/${file}`, target, email);
|
||||
if (built) state.built++;
|
||||
if (relocated && !built) state.moved++;
|
||||
else if (built) state.built++;
|
||||
else state.skipped++;
|
||||
} catch (error) {
|
||||
const message = (error as Error).message;
|
||||
@@ -159,6 +201,18 @@ async function run(email: string, state: SyncState): Promise<void> {
|
||||
};
|
||||
|
||||
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
|
||||
|
||||
/*
|
||||
Then the orphans — derived folders whose original no longer exists.
|
||||
|
||||
Only on a WHOLE-TREE sync. A scoped one knows about a single folder, so everything outside it
|
||||
looks abandoned, and deleting on that basis would take out the library the first time somebody
|
||||
re-synced one subfolder. Safe otherwise because derived/ is reproducible by definition, which is
|
||||
what the name has meant since these two folders stopped being called `user` and `treated`.
|
||||
*/
|
||||
if (!state.scope && !state.fatal) {
|
||||
state.pruned = (await pruneDerived(from, to)).length;
|
||||
}
|
||||
} catch (error) {
|
||||
state.fatal = (error as Error).message;
|
||||
} finally {
|
||||
@@ -180,6 +234,8 @@ async function run(email: string, state: SyncState): Promise<void> {
|
||||
total: state.total ?? state.done,
|
||||
built: state.built,
|
||||
skipped: state.skipped,
|
||||
moved: state.moved,
|
||||
pruned: state.pruned,
|
||||
errors: state.errors,
|
||||
fatal: state.fatal,
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user