fofos/fofos-admin
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backups go to another machine, and old ones are pruned

Browse Source 
Three things.

The local directory defaults to <officerdev root>/dockers/postgres/dumps,
derived from OFFICER_ROOT rather than written down — the same reasoning as
rootPath: a path fixed at install freezes the layout of the machine it was
installed on, and a restore elsewhere then points somewhere that is not there
while looking deliberately chosen.

The destination is SSH, and it is now what "set up" MEANS: a dump sitting on the
machine it came from is a second copy on one disk, lost to the same failure that
would make anybody want it. So local is staging and the copy off the box is the
backup, and a schedule with nowhere to send it does not count as configured.

Testable, in two steps reported separately, because they fail for different
reasons and the fixes differ: the connection is keys and networking, the
directory is permissions on the far side. It tests what is in the FORM rather
than what is stored — testing a saved destination while somebody looks at a
different one is worse than not testing.

Everything goes out with BatchMode=yes. Nobody is at a terminal when this runs,
so a key that has lost access must FAIL rather than block forever on a prompt
nothing will answer. It is also why no credential is stored: there is none to
store and none to leak.

The remote path is the one value here a shell interprets — `ssh host mkdir -p
<path>` runs through it — so shell characters are refused rather than escaped.
Refusing is checkable by reading it; escaping is checkable by being right about
every shell.

And rotation keeps the newest N per database at both ends. Per database, because
two sharing one budget means a busy one evicts the other's history. Sorted by
NAME rather than mtime: the timestamp is in the filename and is sortable, while
mtime stops meaning "when this was taken" the moment a file is copied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-29 13:03:30 +01:00
co-authored by Claude Opus 5
parent ac3d8b76d9
commit 5fc710f072
4 changed files with 403 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/backups.ts
+228 -70
View File
@@ -1,13 +1,20 @@
import { mkdir, readdir, stat } from 'node:fs/promises';
import { mkdir, readdir, stat, unlink } from 'node:fs/promises';
import { isAbsolute, join } from 'node:path';
import { CronJob, CronTime, validateCronExpression } from 'cron';
import { db } from 'officerdb/db';
import { eq } from 'drizzle-orm';
import { OFFICER_ROOT } from '@@/data-path';
import { fofosAdminSettings } from '../db/schema';
import { readDatabases } from './databases';
/**
* Backups of the site's databases: when, where, and the thing that actually runs them.
* Backups of the site's databases: when, where, where else, and how many to keep.
*
* ── Local is staging; the SSH destination is the backup ──
*
* A dump sitting on the machine it came from is a second copy on one disk, lost to the same failure
* that would make anybody want it. So a run writes locally and then copies off, and both halves are
* pruned to the same count.
*
* ── Cron for the schedule, a due-check for the durability ──
*
@@ -16,36 +23,54 @@ import { readDatabases } from './databases';
*
* What cron does NOT do is run a job it was down for. A daily backup at three on a machine that
* restarts overnight is skipped silently, and nothing looks wrong: the schedule is still there, the
* next fire is still in the future, and a month later there is one backup. So the schedule is only half
* of this. On every start, and on every fire, the last run is compared against the schedule's own
* period, and a backup that should already have happened happens now.
* next fire is still in the future, and a month later there is one file. So the schedule is half of
* this. On every start, and every fifteen minutes, the last run is compared against the schedule's own
* period and a backup that should already have happened happens now. That is why `lastBackupAt` is in
* the database — it is the only thing that knows a run was missed.
*
* That is why `lastBackupAt` is in the database rather than in memory. It is the only thing that knows
* a run was missed.
* ── SSH authenticates with the owner's key, never a password ──
*
* ── One job, replaced rather than accumulated ──
*
* The handle is module-level and stopped before a new one is made. Without that, saving the schedule
* twice leaves two jobs running and the third save leaves three — a bug that shows up as backups
* arriving in pairs, long after anybody would connect it to a settings form.
* Nobody is at a terminal when this runs, so every ssh and scp goes out with `BatchMode=yes`: a key
* that has lost access then FAILS rather than blocking forever on a prompt nothing will answer. It is
* also why no credential is stored here — there is none to store, and none to leak.
*/
/** How often to look for a missed run when nothing is scheduled to fire. */
/** How often to look for a run that was missed while this machine was down. */
const CATCHUP_TICK_MS = 15 * 60 * 1000;
/** Long enough for a slow link, short enough that a wedged transfer is not forever. */
const SSH_TIMEOUT_MS = 5 * 60 * 1000;
/** Never wait on a prompt: there is nobody to answer it. */
const SSH_OPTIONS = ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', '-o', 'StrictHostKeyChecking=accept-new'];
/**
* Where dumps go when nobody has said otherwise.
*
* Beside the Postgres container's own directory, under the officerdev root — derived from
* `OFFICER_ROOT` rather than written down, so it follows the machine instead of the install.
*/
export const defaultBackupDir = (): string => join(OFFICER_ROOT, 'dockers', 'postgres', 'dumps');
export type BackupFile = { file: string; bytes: number; at: string };
export type BackupSettings = {
cron: string | null;
dir: string | null;
dir: string;
/** False when the directory is only the computed default — nobody has chosen it. */
dirIsSet: boolean;
sshHost: string | null;
sshPath: string | null;
keep: number | null;
lastBackupAt: string | null;
/** From the schedule, so the panel does not have to know how to read cron. */
/** From the schedule, so the panel never has to read cron. */
nextRunAt: string | null;
/** Newest first. What is actually on disk, rather than what should be. */
recent: { file: string; bytes: number; at: string }[];
recent: BackupFile[];
};
const readRow = async () => (await db.select().from(fofosAdminSettings).limit(1))[0] ?? null;
/** Two successive fires, which is the schedule's period — what "a run was missed" is measured against. */
/** Two successive fires — the schedule's period, which is what "a run was missed" is measured against. */
function periodMs(expression: string): number | null {
try {
const time = new CronTime(expression);
@@ -60,8 +85,7 @@ function periodMs(expression: string): number | null {
export function checkCron(expression: string): string {
const trimmed = expression.trim();
if (!trimmed) throw new Error('A schedule is required.');
const result = validateCronExpression(trimmed);
if (!result.valid) throw new Error(`"${trimmed}" is not a schedule cron understands.`);
if (!validateCronExpression(trimmed).valid) throw new Error(`"${trimmed}" is not a schedule cron understands.`);
return trimmed;
}
@@ -69,13 +93,61 @@ export function checkDir(path: string): string {
const trimmed = path.trim();
if (!trimmed) throw new Error('A location is required.');
if (!isAbsolute(trimmed)) throw new Error('The location must be an absolute path.');
return trimmed.replace(/\/+$/, '');
return trimmed.replace(/\/+$/, '') || '/';
}
/**
* `user@host`, or a Host from ssh_config.
*
* Narrow on purpose. This value is handed to `ssh` as an argument, and while an argv array means no
* shell parses it here, a string beginning with `-` would be read by ssh as an OPTION — which is a
* different and more interesting problem than a bad hostname.
*/
export function checkHost(host: string): string {
const trimmed = host.trim();
if (!trimmed) throw new Error('An SSH destination is required.');
if (!/^[A-Za-z0-9][A-Za-z0-9._@-]*$/.test(trimmed)) {
throw new Error(`"${trimmed}" is not a host this will connect to. Use user@host, or a Host from your ssh config.`);
}
return trimmed;
}
/**
* An absolute path on the far side.
*
* This one IS interpreted by the remote shell — `ssh host mkdir -p /path` runs through it — so the
* characters a shell would act on are refused rather than escaped. Refusing is checkable by reading it;
* escaping is checkable by being right about every shell.
*/
export function checkRemotePath(path: string): string {
const trimmed = path.trim();
if (!trimmed) throw new Error('A remote directory is required.');
if (!trimmed.startsWith('/')) throw new Error('The remote directory must be an absolute path.');
if (/["'`$;&|<>()\\\s]/.test(trimmed)) {
throw new Error('The remote directory may not contain spaces, quotes or shell characters.');
}
return trimmed.replace(/\/+$/, '') || '/';
}
const run = async (argv: string[], timeoutMs = SSH_TIMEOUT_MS): Promise<{ ok: boolean; output: string }> => {
const proc = Bun.spawn(argv, { stdout: 'pipe', stderr: 'pipe', stdin: 'ignore' });
const timer = setTimeout(() => proc.kill(), timeoutMs);
try {
const [stdout, stderr, code] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
proc.exited,
]);
return { ok: code === 0, output: `${stdout}${stderr}`.trim() };
} finally {
clearTimeout(timer);
}
};
export async function readBackups(): Promise<BackupSettings> {
const row = await readRow();
const cron = row?.backupCron ?? null;
const dir = row?.backupDir ?? null;
const dir = row?.backupDir ?? defaultBackupDir();
let nextRunAt: string | null = null;
if (cron) {
@@ -83,46 +155,82 @@ export async function readBackups(): Promise<BackupSettings> {
nextRunAt = new CronTime(cron).getNextDateFrom(new Date()).toISO();
} catch {
// A stored expression that no longer parses. Reported as no next run rather than thrown: the
// panel has to be able to render the settings in order to fix them.
// panel has to render in order to be the place somebody fixes it.
}
}
return {
cron,
dir,
dirIsSet: Boolean(row?.backupDir),
sshHost: row?.backupSshHost ?? null,
sshPath: row?.backupSshPath ?? null,
keep: row?.backupKeep ?? null,
lastBackupAt: row?.lastBackupAt?.toISOString() ?? null,
nextRunAt,
recent: dir ? await listBackups(dir) : [],
recent: await listDumps(dir),
};
}
async function listBackups(dir: string): Promise<{ file: string; bytes: number; at: string }[]> {
async function listDumps(dir: string): Promise<BackupFile[]> {
const names = await readdir(dir).catch(() => []);
const found = [];
const found: BackupFile[] = [];
for (const name of names) {
if (!name.endsWith('.sql')) continue;
const info = await stat(join(dir, name)).catch(() => null);
if (info) found.push({ file: name, bytes: info.size, at: info.mtime.toISOString() });
}
return found.sort((a, b) => b.at.localeCompare(a.at)).slice(0, 10);
return found.sort((a, b) => b.file.localeCompare(a.file));
}
export async function saveBackups(expression: string, directory: string): Promise<BackupSettings> {
const cron = checkCron(expression);
const dir = checkDir(directory);
/**
* Prove the destination before trusting it with anything.
*
* Two steps, reported separately, because they fail for different reasons and the fix is different:
* the connection is keys and networking, the directory is permissions on the far side.
*/
export async function testDestination(host: string, path: string): Promise<{ ok: boolean; output: string }> {
const target = checkHost(host);
const remote = checkRemotePath(path);
const connect = await run(['ssh', ...SSH_OPTIONS, target, 'true'], 30_000);
if (!connect.ok) {
return { ok: false, output: `$ ssh ${target}\n${connect.output || 'Could not connect.'}` };
}
const make = await run(['ssh', ...SSH_OPTIONS, target, `mkdir -p ${remote}`], 30_000);
if (!make.ok) {
return { ok: false, output: `$ ssh ${target} mkdir -p ${remote}\n${make.output || 'Could not create it.'}` };
}
return { ok: true, output: `Connected to ${target}\n${remote} is ready.` };
}
export async function saveBackups(input: {
cron: string;
dir: string;
sshHost: string;
sshPath: string;
keep: number | null;
}): Promise<BackupSettings> {
const cron = checkCron(input.cron);
const dir = checkDir(input.dir);
const sshHost = checkHost(input.sshHost);
const sshPath = checkRemotePath(input.sshPath);
if (input.keep !== null && (!Number.isInteger(input.keep) || input.keep < 1)) {
throw new Error('Keep must be a whole number of dumps, or blank for all of them.');
}
// Made now rather than at the first run, so a location that cannot be created is a message beside the
// form instead of a failure at three in the morning.
await mkdir(dir, { recursive: true });
const row = await readRow();
const values = { backupCron: cron, backupDir: dir, backupSshHost: sshHost, backupSshPath: sshPath, backupKeep: input.keep };
if (row) {
await db
.update(fofosAdminSettings)
.set({ backupCron: cron, backupDir: dir, updatedAt: new Date() })
.where(eq(fofosAdminSettings.id, row.id));
await db.update(fofosAdminSettings).set({ ...values, updatedAt: new Date() }).where(eq(fofosAdminSettings.id, row.id));
} else {
await db.insert(fofosAdminSettings).values({ backupCron: cron, backupDir: dir });
await db.insert(fofosAdminSettings).values(values);
}
await schedule();
@@ -130,15 +238,14 @@ export async function saveBackups(expression: string, directory: string): Promis
}
/**
* Dump every named database, once.
* Dump every named database, copy each off, then prune both ends.
*
* `pg_dump` per database rather than `pg_dumpall`: the two databases here are the site's, and a cluster
* dump would also carry Officer's own, which is not this plugin's to copy around.
* `pg_dump` per database rather than `pg_dumpall`: the two here are the site's, and a cluster dump
* would also carry Officer's own, which is not this plugin's to copy to somebody else's machine.
*/
export async function runBackup(): Promise<{ ok: boolean; files: string[]; output: string }> {
const row = await readRow();
const dir = row?.backupDir;
if (!dir) throw new Error('No backup location is set.');
const dir = row?.backupDir ?? defaultBackupDir();
await mkdir(dir, { recursive: true });
const raw = process.env.OFFICERDEV_POSTGRES_URL ?? process.env.POSTGRES_URL;
@@ -148,41 +255,46 @@ export async function runBackup(): Promise<{ ok: boolean; files: string[]; outpu
const targets = [status.production, status.development].filter((entry) => entry.name && entry.exists);
if (!targets.length) throw new Error('Neither database exists yet.');
// Sortable, and the same shape as the file this plugin's first manual dump used.
const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\..+$/, '').replace('T', '-');
const files: string[] = [];
const lines: string[] = [];
const files: string[] = [];
let ok = true;
for (const target of targets) {
const url = new URL(raw);
url.pathname = `/${target.name}`;
const out = join(dir, `${target.name}-${stamp}.sql`);
const name = `${target.name}-${stamp}.sql`;
const out = join(dir, name);
const proc = Bun.spawn(['pg_dump', url.toString(), '-f', out], {
stdout: 'pipe',
stderr: 'pipe',
stdin: 'ignore',
});
const [stderr, code] = await Promise.all([new Response(proc.stderr).text(), proc.exited]);
if (code === 0) {
files.push(`${target.name}-${stamp}.sql`);
const info = await stat(out).catch(() => null);
lines.push(`${target.name} -> ${files.at(-1)} (${info?.size ?? 0} bytes)`);
} else {
const dumped = await run(['pg_dump', url.toString(), '-f', out]);
if (!dumped.ok) {
ok = false;
lines.push(`${target.name} FAILED: ${stderr.trim()}`);
lines.push(`${target.name}: pg_dump failed — ${dumped.output}`);
continue;
}
const info = await stat(out).catch(() => null);
files.push(name);
lines.push(`${name} ${info?.size ?? 0} bytes`);
if (row?.backupSshHost && row.backupSshPath) {
const sent = await run(['scp', ...SSH_OPTIONS, out, `${row.backupSshHost}:${row.backupSshPath}/`]);
if (sent.ok) lines.push(` -> ${row.backupSshHost}:${row.backupSshPath}/${name}`);
else {
ok = false;
lines.push(` -> COPY FAILED: ${sent.output}`);
}
}
}
/*
Recorded even when one of them failed.
if (row?.backupKeep) lines.push(...(await prune(row.backupKeep, dir, row.backupSshHost, row.backupSshPath)));
Otherwise a database that cannot be dumped makes every subsequent tick think a run is overdue, and
the panel spends the rest of the week trying and failing at fifteen-minute intervals. The failure is
in the output where somebody will see it; the schedule is not the place to express it.
/*
Recorded even when something failed.
Otherwise one undumpable database makes every catch-up tick think a run is overdue, and the panel
spends the week retrying at fifteen-minute intervals. The failure is in the output, where somebody
will read it; the schedule is not the place to express it.
*/
if (row) {
await db
@@ -194,13 +306,60 @@ export async function runBackup(): Promise<{ ok: boolean; files: string[]; outpu
return { ok, files, output: lines.join('\n') };
}
/**
* Keep the newest `keep` dumps PER DATABASE, in both places.
*
* Per database, because two databases sharing one budget means a busy one evicts the other's history —
* and the filenames already carry the database they came from, so the grouping is free.
*
* Sorted by NAME, not mtime. The timestamp is in the filename and is sortable; mtime is not the same
* thing after a file has been copied, touched, or restored from somewhere else.
*/
async function prune(keep: number, dir: string, host: string | null, path: string | null): Promise<string[]> {
const lines: string[] = [];
const group = (names: string[]) => {
const byDatabase = new Map<string, string[]>();
for (const name of names) {
const database = name.replace(/-\d{8}-\d{6}\.sql$/, '');
if (database === name) continue;
byDatabase.set(database, [...(byDatabase.get(database) ?? []), name]);
}
return byDatabase;
};
const local = (await listDumps(dir)).map((entry) => entry.file);
for (const [, names] of group(local)) {
for (const name of names.sort().reverse().slice(keep)) {
await unlink(join(dir, name)).catch(() => {});
lines.push(`pruned ${name}`);
}
}
if (host && path) {
// `ls` rather than `find`: it is on every machine, and the answer is one flat directory.
const listed = await run(['ssh', ...SSH_OPTIONS, host, `ls -1 ${path}`], 30_000);
if (listed.ok) {
const remote = listed.output.split('\n').map((line) => line.trim()).filter((line) => line.endsWith('.sql'));
const doomed: string[] = [];
for (const [, names] of group(remote)) doomed.push(...names.sort().reverse().slice(keep));
if (doomed.length) {
// One call, and each name is one this module wrote — the pattern above only matches its own.
const removed = await run(['ssh', ...SSH_OPTIONS, host, `cd ${path} && rm -f ${doomed.join(' ')}`], 60_000);
lines.push(removed.ok ? `pruned ${doomed.length} on ${host}` : `remote prune failed: ${removed.output}`);
}
}
}
return lines;
}
let job: CronJob | null = null;
let catchup: ReturnType<typeof setInterval> | null = null;
/** Is a scheduled run already overdue — because the machine was off when it should have happened? */
/** Is a scheduled run already overdue, because the machine was off when it should have happened? */
async function runIfMissed(): Promise<void> {
const row = await readRow();
if (!row?.backupCron || !row.backupDir) return;
if (!row?.backupCron) return;
const period = periodMs(row.backupCron);
if (!period) return;
@@ -214,11 +373,10 @@ async function runIfMissed(): Promise<void> {
}
/**
* Start, restart, or stop the timer to match what is stored.
* Start, restart or stop the timer to match what is stored.
*
* Called on save and once at startup. Idempotent, and it always stops what was running first — two
* saves leaving two jobs is a bug that surfaces as backups arriving in pairs, weeks later, with nothing
* connecting it to a settings form.
* Always stops what was running first. Two saves leaving two jobs is a bug that surfaces as backups
* arriving in pairs, weeks later, with nothing connecting it to a settings form.
*/
export async function schedule(): Promise<void> {
job?.stop();
@@ -227,7 +385,7 @@ export async function schedule(): Promise<void> {
catchup = null;
const row = await readRow();
if (!row?.backupCron || !row.backupDir) return;
if (!row?.backupCron) return;
try {
job = new CronJob(row.backupCron, () => void runBackup().catch(() => {}), null, true);
@@ -238,6 +396,6 @@ export async function schedule(): Promise<void> {
}
catchup = setInterval(() => void runIfMissed(), CATCHUP_TICK_MS);
// And once now, because the most likely moment to discover a missed run is the restart after it.
// And once now, because the restart after a missed run is the most likely moment to discover it.
void runIfMissed();
}
api/router.ts
+33 -3
View File
@@ -9,7 +9,7 @@ import { readSync, startSync } from './sync';
import { readRuns } from './ledger';
import { reconcile } from './derive';
import { createDatabases, pushSchema, readDatabases } from './databases';
import { readBackups, runBackup, saveBackups, schedule } from './backups';
import { readBackups, runBackup, saveBackups, schedule, testDestination } from './backups';
import { join } from 'node:path';
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
@@ -330,9 +330,39 @@ router.post('/_config/databases/create', async (ctx) => {
router.get('/_config/backups', async (ctx) => ctx.json(await readBackups()));
router.put('/_config/backups', async (ctx) => {
const { cron, dir } = (ctx.get('body') ?? {}) as { cron?: string; dir?: string };
const body = (ctx.get('body') ?? {}) as {
cron?: string;
dir?: string;
sshHost?: string;
sshPath?: string;
keep?: number | null;
};
try {
return ctx.json(await saveBackups(cron ?? '', dir ?? ''));
return ctx.json(
await saveBackups({
cron: body.cron ?? '',
dir: body.dir ?? '',
sshHost: body.sshHost ?? '',
sshPath: body.sshPath ?? '',
keep: body.keep ?? null,
}),
);
} catch (error) {
return ctx.json({ error: (error as Error).message }, 400);
}
});
/**
* Prove the destination before anything is trusted to it.
*
* Takes the values from the FORM rather than from storage, so it tests what is on screen — testing a
* saved destination while somebody looks at a different one is worse than not testing at all.
*/
router.post('/_config/backups/test', async (ctx) => {
const { sshHost, sshPath } = (ctx.get('body') ?? {}) as { sshHost?: string; sshPath?: string };
try {
const result = await testDestination(sshHost ?? '', sshPath ?? '');
return ctx.json(result, result.ok ? 200 : 400);
} catch (error) {
return ctx.json({ error: (error as Error).message }, 400);
}