backups go to another machine, and old ones are pruned
Three things. The local directory defaults to <officerdev root>/dockers/postgres/dumps, derived from OFFICER_ROOT rather than written down — the same reasoning as rootPath: a path fixed at install freezes the layout of the machine it was installed on, and a restore elsewhere then points somewhere that is not there while looking deliberately chosen. The destination is SSH, and it is now what "set up" MEANS: a dump sitting on the machine it came from is a second copy on one disk, lost to the same failure that would make anybody want it. So local is staging and the copy off the box is the backup, and a schedule with nowhere to send it does not count as configured. Testable, in two steps reported separately, because they fail for different reasons and the fixes differ: the connection is keys and networking, the directory is permissions on the far side. It tests what is in the FORM rather than what is stored — testing a saved destination while somebody looks at a different one is worse than not testing. Everything goes out with BatchMode=yes. Nobody is at a terminal when this runs, so a key that has lost access must FAIL rather than block forever on a prompt nothing will answer. It is also why no credential is stored: there is none to store and none to leak. The remote path is the one value here a shell interprets — `ssh host mkdir -p <path>` runs through it — so shell characters are refused rather than escaped. Refusing is checkable by reading it; escaping is checkable by being right about every shell. And rotation keeps the newest N per database at both ends. Per database, because two sharing one budget means a busy one evicts the other's history. Sorted by NAME rather than mtime: the timestamp is in the filename and is sortable, while mtime stops meaning "when this was taken" the moment a file is copied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+227
-69
@@ -1,13 +1,20 @@
|
|||||||
import { mkdir, readdir, stat } from 'node:fs/promises';
|
import { mkdir, readdir, stat, unlink } from 'node:fs/promises';
|
||||||
import { isAbsolute, join } from 'node:path';
|
import { isAbsolute, join } from 'node:path';
|
||||||
import { CronJob, CronTime, validateCronExpression } from 'cron';
|
import { CronJob, CronTime, validateCronExpression } from 'cron';
|
||||||
import { db } from 'officerdb/db';
|
import { db } from 'officerdb/db';
|
||||||
import { eq } from 'drizzle-orm';
|
import { eq } from 'drizzle-orm';
|
||||||
|
import { OFFICER_ROOT } from '@@/data-path';
|
||||||
import { fofosAdminSettings } from '../db/schema';
|
import { fofosAdminSettings } from '../db/schema';
|
||||||
import { readDatabases } from './databases';
|
import { readDatabases } from './databases';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Backups of the site's databases: when, where, and the thing that actually runs them.
|
* Backups of the site's databases: when, where, where else, and how many to keep.
|
||||||
|
*
|
||||||
|
* ── Local is staging; the SSH destination is the backup ──
|
||||||
|
*
|
||||||
|
* A dump sitting on the machine it came from is a second copy on one disk, lost to the same failure
|
||||||
|
* that would make anybody want it. So a run writes locally and then copies off, and both halves are
|
||||||
|
* pruned to the same count.
|
||||||
*
|
*
|
||||||
* ── Cron for the schedule, a due-check for the durability ──
|
* ── Cron for the schedule, a due-check for the durability ──
|
||||||
*
|
*
|
||||||
@@ -16,36 +23,54 @@ import { readDatabases } from './databases';
|
|||||||
*
|
*
|
||||||
* What cron does NOT do is run a job it was down for. A daily backup at three on a machine that
|
* What cron does NOT do is run a job it was down for. A daily backup at three on a machine that
|
||||||
* restarts overnight is skipped silently, and nothing looks wrong: the schedule is still there, the
|
* restarts overnight is skipped silently, and nothing looks wrong: the schedule is still there, the
|
||||||
* next fire is still in the future, and a month later there is one backup. So the schedule is only half
|
* next fire is still in the future, and a month later there is one file. So the schedule is half of
|
||||||
* of this. On every start, and on every fire, the last run is compared against the schedule's own
|
* this. On every start, and every fifteen minutes, the last run is compared against the schedule's own
|
||||||
* period, and a backup that should already have happened happens now.
|
* period and a backup that should already have happened happens now. That is why `lastBackupAt` is in
|
||||||
|
* the database — it is the only thing that knows a run was missed.
|
||||||
*
|
*
|
||||||
* That is why `lastBackupAt` is in the database rather than in memory. It is the only thing that knows
|
* ── SSH authenticates with the owner's key, never a password ──
|
||||||
* a run was missed.
|
|
||||||
*
|
*
|
||||||
* ── One job, replaced rather than accumulated ──
|
* Nobody is at a terminal when this runs, so every ssh and scp goes out with `BatchMode=yes`: a key
|
||||||
*
|
* that has lost access then FAILS rather than blocking forever on a prompt nothing will answer. It is
|
||||||
* The handle is module-level and stopped before a new one is made. Without that, saving the schedule
|
* also why no credential is stored here — there is none to store, and none to leak.
|
||||||
* twice leaves two jobs running and the third save leaves three — a bug that shows up as backups
|
|
||||||
* arriving in pairs, long after anybody would connect it to a settings form.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
/** How often to look for a missed run when nothing is scheduled to fire. */
|
/** How often to look for a run that was missed while this machine was down. */
|
||||||
const CATCHUP_TICK_MS = 15 * 60 * 1000;
|
const CATCHUP_TICK_MS = 15 * 60 * 1000;
|
||||||
|
|
||||||
|
/** Long enough for a slow link, short enough that a wedged transfer is not forever. */
|
||||||
|
const SSH_TIMEOUT_MS = 5 * 60 * 1000;
|
||||||
|
|
||||||
|
/** Never wait on a prompt: there is nobody to answer it. */
|
||||||
|
const SSH_OPTIONS = ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', '-o', 'StrictHostKeyChecking=accept-new'];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Where dumps go when nobody has said otherwise.
|
||||||
|
*
|
||||||
|
* Beside the Postgres container's own directory, under the officerdev root — derived from
|
||||||
|
* `OFFICER_ROOT` rather than written down, so it follows the machine instead of the install.
|
||||||
|
*/
|
||||||
|
export const defaultBackupDir = (): string => join(OFFICER_ROOT, 'dockers', 'postgres', 'dumps');
|
||||||
|
|
||||||
|
export type BackupFile = { file: string; bytes: number; at: string };
|
||||||
|
|
||||||
export type BackupSettings = {
|
export type BackupSettings = {
|
||||||
cron: string | null;
|
cron: string | null;
|
||||||
dir: string | null;
|
dir: string;
|
||||||
|
/** False when the directory is only the computed default — nobody has chosen it. */
|
||||||
|
dirIsSet: boolean;
|
||||||
|
sshHost: string | null;
|
||||||
|
sshPath: string | null;
|
||||||
|
keep: number | null;
|
||||||
lastBackupAt: string | null;
|
lastBackupAt: string | null;
|
||||||
/** From the schedule, so the panel does not have to know how to read cron. */
|
/** From the schedule, so the panel never has to read cron. */
|
||||||
nextRunAt: string | null;
|
nextRunAt: string | null;
|
||||||
/** Newest first. What is actually on disk, rather than what should be. */
|
recent: BackupFile[];
|
||||||
recent: { file: string; bytes: number; at: string }[];
|
|
||||||
};
|
};
|
||||||
|
|
||||||
const readRow = async () => (await db.select().from(fofosAdminSettings).limit(1))[0] ?? null;
|
const readRow = async () => (await db.select().from(fofosAdminSettings).limit(1))[0] ?? null;
|
||||||
|
|
||||||
/** Two successive fires, which is the schedule's period — what "a run was missed" is measured against. */
|
/** Two successive fires — the schedule's period, which is what "a run was missed" is measured against. */
|
||||||
function periodMs(expression: string): number | null {
|
function periodMs(expression: string): number | null {
|
||||||
try {
|
try {
|
||||||
const time = new CronTime(expression);
|
const time = new CronTime(expression);
|
||||||
@@ -60,8 +85,7 @@ function periodMs(expression: string): number | null {
|
|||||||
export function checkCron(expression: string): string {
|
export function checkCron(expression: string): string {
|
||||||
const trimmed = expression.trim();
|
const trimmed = expression.trim();
|
||||||
if (!trimmed) throw new Error('A schedule is required.');
|
if (!trimmed) throw new Error('A schedule is required.');
|
||||||
const result = validateCronExpression(trimmed);
|
if (!validateCronExpression(trimmed).valid) throw new Error(`"${trimmed}" is not a schedule cron understands.`);
|
||||||
if (!result.valid) throw new Error(`"${trimmed}" is not a schedule cron understands.`);
|
|
||||||
return trimmed;
|
return trimmed;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,13 +93,61 @@ export function checkDir(path: string): string {
|
|||||||
const trimmed = path.trim();
|
const trimmed = path.trim();
|
||||||
if (!trimmed) throw new Error('A location is required.');
|
if (!trimmed) throw new Error('A location is required.');
|
||||||
if (!isAbsolute(trimmed)) throw new Error('The location must be an absolute path.');
|
if (!isAbsolute(trimmed)) throw new Error('The location must be an absolute path.');
|
||||||
return trimmed.replace(/\/+$/, '');
|
return trimmed.replace(/\/+$/, '') || '/';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `user@host`, or a Host from ssh_config.
|
||||||
|
*
|
||||||
|
* Narrow on purpose. This value is handed to `ssh` as an argument, and while an argv array means no
|
||||||
|
* shell parses it here, a string beginning with `-` would be read by ssh as an OPTION — which is a
|
||||||
|
* different and more interesting problem than a bad hostname.
|
||||||
|
*/
|
||||||
|
export function checkHost(host: string): string {
|
||||||
|
const trimmed = host.trim();
|
||||||
|
if (!trimmed) throw new Error('An SSH destination is required.');
|
||||||
|
if (!/^[A-Za-z0-9][A-Za-z0-9._@-]*$/.test(trimmed)) {
|
||||||
|
throw new Error(`"${trimmed}" is not a host this will connect to. Use user@host, or a Host from your ssh config.`);
|
||||||
|
}
|
||||||
|
return trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An absolute path on the far side.
|
||||||
|
*
|
||||||
|
* This one IS interpreted by the remote shell — `ssh host mkdir -p /path` runs through it — so the
|
||||||
|
* characters a shell would act on are refused rather than escaped. Refusing is checkable by reading it;
|
||||||
|
* escaping is checkable by being right about every shell.
|
||||||
|
*/
|
||||||
|
export function checkRemotePath(path: string): string {
|
||||||
|
const trimmed = path.trim();
|
||||||
|
if (!trimmed) throw new Error('A remote directory is required.');
|
||||||
|
if (!trimmed.startsWith('/')) throw new Error('The remote directory must be an absolute path.');
|
||||||
|
if (/["'`$;&|<>()\\\s]/.test(trimmed)) {
|
||||||
|
throw new Error('The remote directory may not contain spaces, quotes or shell characters.');
|
||||||
|
}
|
||||||
|
return trimmed.replace(/\/+$/, '') || '/';
|
||||||
|
}
|
||||||
|
|
||||||
|
const run = async (argv: string[], timeoutMs = SSH_TIMEOUT_MS): Promise<{ ok: boolean; output: string }> => {
|
||||||
|
const proc = Bun.spawn(argv, { stdout: 'pipe', stderr: 'pipe', stdin: 'ignore' });
|
||||||
|
const timer = setTimeout(() => proc.kill(), timeoutMs);
|
||||||
|
try {
|
||||||
|
const [stdout, stderr, code] = await Promise.all([
|
||||||
|
new Response(proc.stdout).text(),
|
||||||
|
new Response(proc.stderr).text(),
|
||||||
|
proc.exited,
|
||||||
|
]);
|
||||||
|
return { ok: code === 0, output: `${stdout}${stderr}`.trim() };
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
export async function readBackups(): Promise<BackupSettings> {
|
export async function readBackups(): Promise<BackupSettings> {
|
||||||
const row = await readRow();
|
const row = await readRow();
|
||||||
const cron = row?.backupCron ?? null;
|
const cron = row?.backupCron ?? null;
|
||||||
const dir = row?.backupDir ?? null;
|
const dir = row?.backupDir ?? defaultBackupDir();
|
||||||
|
|
||||||
let nextRunAt: string | null = null;
|
let nextRunAt: string | null = null;
|
||||||
if (cron) {
|
if (cron) {
|
||||||
@@ -83,46 +155,82 @@ export async function readBackups(): Promise<BackupSettings> {
|
|||||||
nextRunAt = new CronTime(cron).getNextDateFrom(new Date()).toISO();
|
nextRunAt = new CronTime(cron).getNextDateFrom(new Date()).toISO();
|
||||||
} catch {
|
} catch {
|
||||||
// A stored expression that no longer parses. Reported as no next run rather than thrown: the
|
// A stored expression that no longer parses. Reported as no next run rather than thrown: the
|
||||||
// panel has to be able to render the settings in order to fix them.
|
// panel has to render in order to be the place somebody fixes it.
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
cron,
|
cron,
|
||||||
dir,
|
dir,
|
||||||
|
dirIsSet: Boolean(row?.backupDir),
|
||||||
|
sshHost: row?.backupSshHost ?? null,
|
||||||
|
sshPath: row?.backupSshPath ?? null,
|
||||||
|
keep: row?.backupKeep ?? null,
|
||||||
lastBackupAt: row?.lastBackupAt?.toISOString() ?? null,
|
lastBackupAt: row?.lastBackupAt?.toISOString() ?? null,
|
||||||
nextRunAt,
|
nextRunAt,
|
||||||
recent: dir ? await listBackups(dir) : [],
|
recent: await listDumps(dir),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
async function listBackups(dir: string): Promise<{ file: string; bytes: number; at: string }[]> {
|
async function listDumps(dir: string): Promise<BackupFile[]> {
|
||||||
const names = await readdir(dir).catch(() => []);
|
const names = await readdir(dir).catch(() => []);
|
||||||
const found = [];
|
const found: BackupFile[] = [];
|
||||||
for (const name of names) {
|
for (const name of names) {
|
||||||
if (!name.endsWith('.sql')) continue;
|
if (!name.endsWith('.sql')) continue;
|
||||||
const info = await stat(join(dir, name)).catch(() => null);
|
const info = await stat(join(dir, name)).catch(() => null);
|
||||||
if (info) found.push({ file: name, bytes: info.size, at: info.mtime.toISOString() });
|
if (info) found.push({ file: name, bytes: info.size, at: info.mtime.toISOString() });
|
||||||
}
|
}
|
||||||
return found.sort((a, b) => b.at.localeCompare(a.at)).slice(0, 10);
|
return found.sort((a, b) => b.file.localeCompare(a.file));
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function saveBackups(expression: string, directory: string): Promise<BackupSettings> {
|
/**
|
||||||
const cron = checkCron(expression);
|
* Prove the destination before trusting it with anything.
|
||||||
const dir = checkDir(directory);
|
*
|
||||||
|
* Two steps, reported separately, because they fail for different reasons and the fix is different:
|
||||||
|
* the connection is keys and networking, the directory is permissions on the far side.
|
||||||
|
*/
|
||||||
|
export async function testDestination(host: string, path: string): Promise<{ ok: boolean; output: string }> {
|
||||||
|
const target = checkHost(host);
|
||||||
|
const remote = checkRemotePath(path);
|
||||||
|
|
||||||
|
const connect = await run(['ssh', ...SSH_OPTIONS, target, 'true'], 30_000);
|
||||||
|
if (!connect.ok) {
|
||||||
|
return { ok: false, output: `$ ssh ${target}\n${connect.output || 'Could not connect.'}` };
|
||||||
|
}
|
||||||
|
|
||||||
|
const make = await run(['ssh', ...SSH_OPTIONS, target, `mkdir -p ${remote}`], 30_000);
|
||||||
|
if (!make.ok) {
|
||||||
|
return { ok: false, output: `$ ssh ${target} mkdir -p ${remote}\n${make.output || 'Could not create it.'}` };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { ok: true, output: `Connected to ${target}\n${remote} is ready.` };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function saveBackups(input: {
|
||||||
|
cron: string;
|
||||||
|
dir: string;
|
||||||
|
sshHost: string;
|
||||||
|
sshPath: string;
|
||||||
|
keep: number | null;
|
||||||
|
}): Promise<BackupSettings> {
|
||||||
|
const cron = checkCron(input.cron);
|
||||||
|
const dir = checkDir(input.dir);
|
||||||
|
const sshHost = checkHost(input.sshHost);
|
||||||
|
const sshPath = checkRemotePath(input.sshPath);
|
||||||
|
if (input.keep !== null && (!Number.isInteger(input.keep) || input.keep < 1)) {
|
||||||
|
throw new Error('Keep must be a whole number of dumps, or blank for all of them.');
|
||||||
|
}
|
||||||
|
|
||||||
// Made now rather than at the first run, so a location that cannot be created is a message beside the
|
// Made now rather than at the first run, so a location that cannot be created is a message beside the
|
||||||
// form instead of a failure at three in the morning.
|
// form instead of a failure at three in the morning.
|
||||||
await mkdir(dir, { recursive: true });
|
await mkdir(dir, { recursive: true });
|
||||||
|
|
||||||
const row = await readRow();
|
const row = await readRow();
|
||||||
|
const values = { backupCron: cron, backupDir: dir, backupSshHost: sshHost, backupSshPath: sshPath, backupKeep: input.keep };
|
||||||
if (row) {
|
if (row) {
|
||||||
await db
|
await db.update(fofosAdminSettings).set({ ...values, updatedAt: new Date() }).where(eq(fofosAdminSettings.id, row.id));
|
||||||
.update(fofosAdminSettings)
|
|
||||||
.set({ backupCron: cron, backupDir: dir, updatedAt: new Date() })
|
|
||||||
.where(eq(fofosAdminSettings.id, row.id));
|
|
||||||
} else {
|
} else {
|
||||||
await db.insert(fofosAdminSettings).values({ backupCron: cron, backupDir: dir });
|
await db.insert(fofosAdminSettings).values(values);
|
||||||
}
|
}
|
||||||
|
|
||||||
await schedule();
|
await schedule();
|
||||||
@@ -130,15 +238,14 @@ export async function saveBackups(expression: string, directory: string): Promis
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Dump every named database, once.
|
* Dump every named database, copy each off, then prune both ends.
|
||||||
*
|
*
|
||||||
* `pg_dump` per database rather than `pg_dumpall`: the two databases here are the site's, and a cluster
|
* `pg_dump` per database rather than `pg_dumpall`: the two here are the site's, and a cluster dump
|
||||||
* dump would also carry Officer's own, which is not this plugin's to copy around.
|
* would also carry Officer's own, which is not this plugin's to copy to somebody else's machine.
|
||||||
*/
|
*/
|
||||||
export async function runBackup(): Promise<{ ok: boolean; files: string[]; output: string }> {
|
export async function runBackup(): Promise<{ ok: boolean; files: string[]; output: string }> {
|
||||||
const row = await readRow();
|
const row = await readRow();
|
||||||
const dir = row?.backupDir;
|
const dir = row?.backupDir ?? defaultBackupDir();
|
||||||
if (!dir) throw new Error('No backup location is set.');
|
|
||||||
await mkdir(dir, { recursive: true });
|
await mkdir(dir, { recursive: true });
|
||||||
|
|
||||||
const raw = process.env.OFFICERDEV_POSTGRES_URL ?? process.env.POSTGRES_URL;
|
const raw = process.env.OFFICERDEV_POSTGRES_URL ?? process.env.POSTGRES_URL;
|
||||||
@@ -148,41 +255,46 @@ export async function runBackup(): Promise<{ ok: boolean; files: string[]; outpu
|
|||||||
const targets = [status.production, status.development].filter((entry) => entry.name && entry.exists);
|
const targets = [status.production, status.development].filter((entry) => entry.name && entry.exists);
|
||||||
if (!targets.length) throw new Error('Neither database exists yet.');
|
if (!targets.length) throw new Error('Neither database exists yet.');
|
||||||
|
|
||||||
// Sortable, and the same shape as the file this plugin's first manual dump used.
|
|
||||||
const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\..+$/, '').replace('T', '-');
|
const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\..+$/, '').replace('T', '-');
|
||||||
|
|
||||||
const files: string[] = [];
|
|
||||||
const lines: string[] = [];
|
const lines: string[] = [];
|
||||||
|
const files: string[] = [];
|
||||||
let ok = true;
|
let ok = true;
|
||||||
|
|
||||||
for (const target of targets) {
|
for (const target of targets) {
|
||||||
const url = new URL(raw);
|
const url = new URL(raw);
|
||||||
url.pathname = `/${target.name}`;
|
url.pathname = `/${target.name}`;
|
||||||
const out = join(dir, `${target.name}-${stamp}.sql`);
|
const name = `${target.name}-${stamp}.sql`;
|
||||||
|
const out = join(dir, name);
|
||||||
|
|
||||||
const proc = Bun.spawn(['pg_dump', url.toString(), '-f', out], {
|
const dumped = await run(['pg_dump', url.toString(), '-f', out]);
|
||||||
stdout: 'pipe',
|
if (!dumped.ok) {
|
||||||
stderr: 'pipe',
|
|
||||||
stdin: 'ignore',
|
|
||||||
});
|
|
||||||
const [stderr, code] = await Promise.all([new Response(proc.stderr).text(), proc.exited]);
|
|
||||||
|
|
||||||
if (code === 0) {
|
|
||||||
files.push(`${target.name}-${stamp}.sql`);
|
|
||||||
const info = await stat(out).catch(() => null);
|
|
||||||
lines.push(`${target.name} -> ${files.at(-1)} (${info?.size ?? 0} bytes)`);
|
|
||||||
} else {
|
|
||||||
ok = false;
|
ok = false;
|
||||||
lines.push(`${target.name} FAILED: ${stderr.trim()}`);
|
lines.push(`${target.name}: pg_dump failed — ${dumped.output}`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const info = await stat(out).catch(() => null);
|
||||||
|
files.push(name);
|
||||||
|
lines.push(`${name} ${info?.size ?? 0} bytes`);
|
||||||
|
|
||||||
|
if (row?.backupSshHost && row.backupSshPath) {
|
||||||
|
const sent = await run(['scp', ...SSH_OPTIONS, out, `${row.backupSshHost}:${row.backupSshPath}/`]);
|
||||||
|
if (sent.ok) lines.push(` -> ${row.backupSshHost}:${row.backupSshPath}/${name}`);
|
||||||
|
else {
|
||||||
|
ok = false;
|
||||||
|
lines.push(` -> COPY FAILED: ${sent.output}`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (row?.backupKeep) lines.push(...(await prune(row.backupKeep, dir, row.backupSshHost, row.backupSshPath)));
|
||||||
|
|
||||||
/*
|
/*
|
||||||
Recorded even when one of them failed.
|
Recorded even when something failed.
|
||||||
|
|
||||||
Otherwise a database that cannot be dumped makes every subsequent tick think a run is overdue, and
|
Otherwise one undumpable database makes every catch-up tick think a run is overdue, and the panel
|
||||||
the panel spends the rest of the week trying and failing at fifteen-minute intervals. The failure is
|
spends the week retrying at fifteen-minute intervals. The failure is in the output, where somebody
|
||||||
in the output where somebody will see it; the schedule is not the place to express it.
|
will read it; the schedule is not the place to express it.
|
||||||
*/
|
*/
|
||||||
if (row) {
|
if (row) {
|
||||||
await db
|
await db
|
||||||
@@ -194,13 +306,60 @@ export async function runBackup(): Promise<{ ok: boolean; files: string[]; outpu
|
|||||||
return { ok, files, output: lines.join('\n') };
|
return { ok, files, output: lines.join('\n') };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Keep the newest `keep` dumps PER DATABASE, in both places.
|
||||||
|
*
|
||||||
|
* Per database, because two databases sharing one budget means a busy one evicts the other's history —
|
||||||
|
* and the filenames already carry the database they came from, so the grouping is free.
|
||||||
|
*
|
||||||
|
* Sorted by NAME, not mtime. The timestamp is in the filename and is sortable; mtime is not the same
|
||||||
|
* thing after a file has been copied, touched, or restored from somewhere else.
|
||||||
|
*/
|
||||||
|
async function prune(keep: number, dir: string, host: string | null, path: string | null): Promise<string[]> {
|
||||||
|
const lines: string[] = [];
|
||||||
|
const group = (names: string[]) => {
|
||||||
|
const byDatabase = new Map<string, string[]>();
|
||||||
|
for (const name of names) {
|
||||||
|
const database = name.replace(/-\d{8}-\d{6}\.sql$/, '');
|
||||||
|
if (database === name) continue;
|
||||||
|
byDatabase.set(database, [...(byDatabase.get(database) ?? []), name]);
|
||||||
|
}
|
||||||
|
return byDatabase;
|
||||||
|
};
|
||||||
|
|
||||||
|
const local = (await listDumps(dir)).map((entry) => entry.file);
|
||||||
|
for (const [, names] of group(local)) {
|
||||||
|
for (const name of names.sort().reverse().slice(keep)) {
|
||||||
|
await unlink(join(dir, name)).catch(() => {});
|
||||||
|
lines.push(`pruned ${name}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (host && path) {
|
||||||
|
// `ls` rather than `find`: it is on every machine, and the answer is one flat directory.
|
||||||
|
const listed = await run(['ssh', ...SSH_OPTIONS, host, `ls -1 ${path}`], 30_000);
|
||||||
|
if (listed.ok) {
|
||||||
|
const remote = listed.output.split('\n').map((line) => line.trim()).filter((line) => line.endsWith('.sql'));
|
||||||
|
const doomed: string[] = [];
|
||||||
|
for (const [, names] of group(remote)) doomed.push(...names.sort().reverse().slice(keep));
|
||||||
|
if (doomed.length) {
|
||||||
|
// One call, and each name is one this module wrote — the pattern above only matches its own.
|
||||||
|
const removed = await run(['ssh', ...SSH_OPTIONS, host, `cd ${path} && rm -f ${doomed.join(' ')}`], 60_000);
|
||||||
|
lines.push(removed.ok ? `pruned ${doomed.length} on ${host}` : `remote prune failed: ${removed.output}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return lines;
|
||||||
|
}
|
||||||
|
|
||||||
let job: CronJob | null = null;
|
let job: CronJob | null = null;
|
||||||
let catchup: ReturnType<typeof setInterval> | null = null;
|
let catchup: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
||||||
/** Is a scheduled run already overdue — because the machine was off when it should have happened? */
|
/** Is a scheduled run already overdue, because the machine was off when it should have happened? */
|
||||||
async function runIfMissed(): Promise<void> {
|
async function runIfMissed(): Promise<void> {
|
||||||
const row = await readRow();
|
const row = await readRow();
|
||||||
if (!row?.backupCron || !row.backupDir) return;
|
if (!row?.backupCron) return;
|
||||||
|
|
||||||
const period = periodMs(row.backupCron);
|
const period = periodMs(row.backupCron);
|
||||||
if (!period) return;
|
if (!period) return;
|
||||||
@@ -214,11 +373,10 @@ async function runIfMissed(): Promise<void> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Start, restart, or stop the timer to match what is stored.
|
* Start, restart or stop the timer to match what is stored.
|
||||||
*
|
*
|
||||||
* Called on save and once at startup. Idempotent, and it always stops what was running first — two
|
* Always stops what was running first. Two saves leaving two jobs is a bug that surfaces as backups
|
||||||
* saves leaving two jobs is a bug that surfaces as backups arriving in pairs, weeks later, with nothing
|
* arriving in pairs, weeks later, with nothing connecting it to a settings form.
|
||||||
* connecting it to a settings form.
|
|
||||||
*/
|
*/
|
||||||
export async function schedule(): Promise<void> {
|
export async function schedule(): Promise<void> {
|
||||||
job?.stop();
|
job?.stop();
|
||||||
@@ -227,7 +385,7 @@ export async function schedule(): Promise<void> {
|
|||||||
catchup = null;
|
catchup = null;
|
||||||
|
|
||||||
const row = await readRow();
|
const row = await readRow();
|
||||||
if (!row?.backupCron || !row.backupDir) return;
|
if (!row?.backupCron) return;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
job = new CronJob(row.backupCron, () => void runBackup().catch(() => {}), null, true);
|
job = new CronJob(row.backupCron, () => void runBackup().catch(() => {}), null, true);
|
||||||
@@ -238,6 +396,6 @@ export async function schedule(): Promise<void> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
catchup = setInterval(() => void runIfMissed(), CATCHUP_TICK_MS);
|
catchup = setInterval(() => void runIfMissed(), CATCHUP_TICK_MS);
|
||||||
// And once now, because the most likely moment to discover a missed run is the restart after it.
|
// And once now, because the restart after a missed run is the most likely moment to discover it.
|
||||||
void runIfMissed();
|
void runIfMissed();
|
||||||
}
|
}
|
||||||
|
|||||||
+33
-3
@@ -9,7 +9,7 @@ import { readSync, startSync } from './sync';
|
|||||||
import { readRuns } from './ledger';
|
import { readRuns } from './ledger';
|
||||||
import { reconcile } from './derive';
|
import { reconcile } from './derive';
|
||||||
import { createDatabases, pushSchema, readDatabases } from './databases';
|
import { createDatabases, pushSchema, readDatabases } from './databases';
|
||||||
import { readBackups, runBackup, saveBackups, schedule } from './backups';
|
import { readBackups, runBackup, saveBackups, schedule, testDestination } from './backups';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
|
import { fofosRequest, FofosError, probe, type FofosResponse } from './fofos-client';
|
||||||
|
|
||||||
@@ -330,9 +330,39 @@ router.post('/_config/databases/create', async (ctx) => {
|
|||||||
router.get('/_config/backups', async (ctx) => ctx.json(await readBackups()));
|
router.get('/_config/backups', async (ctx) => ctx.json(await readBackups()));
|
||||||
|
|
||||||
router.put('/_config/backups', async (ctx) => {
|
router.put('/_config/backups', async (ctx) => {
|
||||||
const { cron, dir } = (ctx.get('body') ?? {}) as { cron?: string; dir?: string };
|
const body = (ctx.get('body') ?? {}) as {
|
||||||
|
cron?: string;
|
||||||
|
dir?: string;
|
||||||
|
sshHost?: string;
|
||||||
|
sshPath?: string;
|
||||||
|
keep?: number | null;
|
||||||
|
};
|
||||||
try {
|
try {
|
||||||
return ctx.json(await saveBackups(cron ?? '', dir ?? ''));
|
return ctx.json(
|
||||||
|
await saveBackups({
|
||||||
|
cron: body.cron ?? '',
|
||||||
|
dir: body.dir ?? '',
|
||||||
|
sshHost: body.sshHost ?? '',
|
||||||
|
sshPath: body.sshPath ?? '',
|
||||||
|
keep: body.keep ?? null,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
return ctx.json({ error: (error as Error).message }, 400);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Prove the destination before anything is trusted to it.
|
||||||
|
*
|
||||||
|
* Takes the values from the FORM rather than from storage, so it tests what is on screen — testing a
|
||||||
|
* saved destination while somebody looks at a different one is worse than not testing at all.
|
||||||
|
*/
|
||||||
|
router.post('/_config/backups/test', async (ctx) => {
|
||||||
|
const { sshHost, sshPath } = (ctx.get('body') ?? {}) as { sshHost?: string; sshPath?: string };
|
||||||
|
try {
|
||||||
|
const result = await testDestination(sshHost ?? '', sshPath ?? '');
|
||||||
|
return ctx.json(result, result.ok ? 200 : 400);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
return ctx.json({ error: (error as Error).message }, 400);
|
return ctx.json({ error: (error as Error).message }, 400);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -115,8 +115,36 @@ export const fofosAdminSettings = pgTable('fofos_admin_settings', {
|
|||||||
* half-configured state to interpret — it is off.
|
* half-configured state to interpret — it is off.
|
||||||
*/
|
*/
|
||||||
backupCron: text('backup_cron'),
|
backupCron: text('backup_cron'),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Where dumps are written on THIS machine, before they are sent anywhere.
|
||||||
|
*
|
||||||
|
* Null means the default, computed at read time from `OFFICER_ROOT` — the same reasoning as
|
||||||
|
* `rootPath` above: a path written in at install freezes the layout of the machine it was installed
|
||||||
|
* on, and a restore elsewhere then points at a directory that is not there while looking chosen.
|
||||||
|
*/
|
||||||
backupDir: text('backup_dir'),
|
backupDir: text('backup_dir'),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Where they are COPIED to. `user@host`, and an absolute path on it.
|
||||||
|
*
|
||||||
|
* A backup that only exists on the machine it came from is not a backup — it is a second copy on the
|
||||||
|
* same disk, lost to the same failure. So the local directory is staging and this is the destination.
|
||||||
|
*
|
||||||
|
* Authenticated by the owner's existing SSH key, never a password: there is nobody at a terminal when
|
||||||
|
* this runs, and a credential stored here would be one this plugin had to hold and could leak.
|
||||||
|
*/
|
||||||
|
backupSshHost: text('backup_ssh_host'),
|
||||||
|
backupSshPath: text('backup_ssh_path'),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How many dumps to keep, per database, in each place.
|
||||||
|
*
|
||||||
|
* Null means keep everything, which is a real choice and a bad default — a nightly dump of a small
|
||||||
|
* site fills a disk slowly enough that nobody notices until something else fails.
|
||||||
|
*/
|
||||||
|
backupKeep: integer('backup_keep'),
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* When the last one finished, which is what makes the schedule survive a restart.
|
* When the last one finished, which is what makes the schedule survive a restart.
|
||||||
*
|
*
|
||||||
|
|||||||
+114
-10
@@ -1,5 +1,5 @@
|
|||||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||||
import { Check, ChevronDown, DatabaseBackup, Loader2, PlayCircle } from 'lucide-react';
|
import { Check, ChevronDown, DatabaseBackup, Loader2, PlayCircle, PlugZap } from 'lucide-react';
|
||||||
import { useClient } from 'hooks/useClient';
|
import { useClient } from 'hooks/useClient';
|
||||||
import { Button } from '@/components/ui/button';
|
import { Button } from '@/components/ui/button';
|
||||||
import { Input } from '@/components/ui/input';
|
import { Input } from '@/components/ui/input';
|
||||||
@@ -32,7 +32,11 @@ type Backup = { file: string; bytes: number; at: string };
|
|||||||
|
|
||||||
type Settings = {
|
type Settings = {
|
||||||
cron: string | null;
|
cron: string | null;
|
||||||
dir: string | null;
|
dir: string;
|
||||||
|
dirIsSet: boolean;
|
||||||
|
sshHost: string | null;
|
||||||
|
sshPath: string | null;
|
||||||
|
keep: number | null;
|
||||||
lastBackupAt: string | null;
|
lastBackupAt: string | null;
|
||||||
nextRunAt: string | null;
|
nextRunAt: string | null;
|
||||||
recent: Backup[];
|
recent: Backup[];
|
||||||
@@ -62,8 +66,11 @@ export const BackupsRow = () => {
|
|||||||
const [open, setOpen] = useState(false);
|
const [open, setOpen] = useState(false);
|
||||||
const [cron, setCron] = useState('0 3 * * *');
|
const [cron, setCron] = useState('0 3 * * *');
|
||||||
const [dir, setDir] = useState('');
|
const [dir, setDir] = useState('');
|
||||||
|
const [sshHost, setSshHost] = useState('');
|
||||||
|
const [sshPath, setSshPath] = useState('');
|
||||||
|
const [keep, setKeep] = useState('7');
|
||||||
const [picking, setPicking] = useState(false);
|
const [picking, setPicking] = useState(false);
|
||||||
const [busy, setBusy] = useState<'save' | 'run' | null>(null);
|
const [busy, setBusy] = useState<'save' | 'run' | 'test' | null>(null);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
const [output, setOutput] = useState<string | null>(null);
|
const [output, setOutput] = useState<string | null>(null);
|
||||||
const [failed, setFailed] = useState(false);
|
const [failed, setFailed] = useState(false);
|
||||||
@@ -72,8 +79,13 @@ export const BackupsRow = () => {
|
|||||||
try {
|
try {
|
||||||
const current = await clientRef.current.get<Settings>('/fofos-admin/_config/backups');
|
const current = await clientRef.current.get<Settings>('/fofos-admin/_config/backups');
|
||||||
setSettings(current);
|
setSettings(current);
|
||||||
setCron((typed) => (current.cron ?? typed));
|
setCron((typed) => current.cron ?? typed);
|
||||||
setDir((typed) => typed || (current.dir ?? ''));
|
// The server computes the default when nobody has chosen one, so the field is never empty and
|
||||||
|
// never shows a path that is only a guess without it also being the one that would be used.
|
||||||
|
setDir((typed) => typed || current.dir);
|
||||||
|
setSshHost((typed) => typed || (current.sshHost ?? ''));
|
||||||
|
setSshPath((typed) => typed || (current.sshPath ?? ''));
|
||||||
|
setKeep((typed) => (current.keep === null ? typed : String(current.keep)));
|
||||||
} catch {
|
} catch {
|
||||||
setSettings(null);
|
setSettings(null);
|
||||||
}
|
}
|
||||||
@@ -87,7 +99,15 @@ export const BackupsRow = () => {
|
|||||||
setBusy('save');
|
setBusy('save');
|
||||||
setError(null);
|
setError(null);
|
||||||
try {
|
try {
|
||||||
setSettings(await clientRef.current.put<Settings>('/fofos-admin/_config/backups', { cron, dir }));
|
setSettings(
|
||||||
|
await clientRef.current.put<Settings>('/fofos-admin/_config/backups', {
|
||||||
|
cron,
|
||||||
|
dir,
|
||||||
|
sshHost,
|
||||||
|
sshPath,
|
||||||
|
keep: keep.trim() ? Number(keep) : null,
|
||||||
|
}),
|
||||||
|
);
|
||||||
setOpen(false);
|
setOpen(false);
|
||||||
} catch (failure) {
|
} catch (failure) {
|
||||||
setError(readableError(failure));
|
setError(readableError(failure));
|
||||||
@@ -96,6 +116,27 @@ export const BackupsRow = () => {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/** Connection first, then the directory. Different failures, different fixes — see the API. */
|
||||||
|
const test = async () => {
|
||||||
|
setBusy('test');
|
||||||
|
setError(null);
|
||||||
|
setFailed(false);
|
||||||
|
setOutput(null);
|
||||||
|
try {
|
||||||
|
const result = await clientRef.current.post<{ ok: boolean; output: string }>(
|
||||||
|
'/fofos-admin/_config/backups/test',
|
||||||
|
{ sshHost, sshPath },
|
||||||
|
);
|
||||||
|
setOutput(result.output);
|
||||||
|
setFailed(!result.ok);
|
||||||
|
} catch (failure) {
|
||||||
|
setOutput(readableError(failure));
|
||||||
|
setFailed(true);
|
||||||
|
} finally {
|
||||||
|
setBusy(null);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const run = async () => {
|
const run = async () => {
|
||||||
setBusy('run');
|
setBusy('run');
|
||||||
setError(null);
|
setError(null);
|
||||||
@@ -120,7 +161,8 @@ export const BackupsRow = () => {
|
|||||||
|
|
||||||
if (!settings) return null;
|
if (!settings) return null;
|
||||||
|
|
||||||
const configured = Boolean(settings.cron && settings.dir);
|
// A schedule with nowhere to send it is not set up — the point of this is the copy off the box.
|
||||||
|
const configured = Boolean(settings.cron && settings.sshHost && settings.sshPath);
|
||||||
const chosen = CHOICES.find((choice) => choice.cron === settings.cron);
|
const chosen = CHOICES.find((choice) => choice.cron === settings.cron);
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -130,7 +172,12 @@ export const BackupsRow = () => {
|
|||||||
<p className="text-xs text-muted-foreground">Backups</p>
|
<p className="text-xs text-muted-foreground">Backups</p>
|
||||||
<p className="truncate font-mono text-xs">
|
<p className="truncate font-mono text-xs">
|
||||||
{configured ? (chosen?.label ?? settings.cron) : 'Not set up'}
|
{configured ? (chosen?.label ?? settings.cron) : 'Not set up'}
|
||||||
{configured && settings.dir ? <span className="text-muted-foreground"> · {settings.dir}</span> : null}
|
{configured && settings.sshHost ? (
|
||||||
|
<span className="text-muted-foreground">
|
||||||
|
{' '}
|
||||||
|
→ {settings.sshHost}:{settings.sshPath}
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -188,7 +235,9 @@ export const BackupsRow = () => {
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="space-y-1">
|
<div className="space-y-1">
|
||||||
<p className="text-xs font-medium">Where</p>
|
<p className="text-xs font-medium">
|
||||||
|
Staged locally <span className="font-normal text-muted-foreground/70">before being sent</span>
|
||||||
|
</p>
|
||||||
<div className="flex gap-2">
|
<div className="flex gap-2">
|
||||||
<Input
|
<Input
|
||||||
value={dir}
|
value={dir}
|
||||||
@@ -213,13 +262,68 @@ export const BackupsRow = () => {
|
|||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div className="space-y-1">
|
||||||
|
<p className="text-xs font-medium">
|
||||||
|
Sent to <span className="font-normal text-muted-foreground/70">another machine, over SSH</span>
|
||||||
|
</p>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<Input
|
||||||
|
value={sshHost}
|
||||||
|
onChange={(event) => setSshHost(event.target.value)}
|
||||||
|
placeholder="user@host"
|
||||||
|
className="w-48 shrink-0 font-mono text-xs"
|
||||||
|
autoComplete="off"
|
||||||
|
spellCheck={false}
|
||||||
|
/>
|
||||||
|
<Input
|
||||||
|
value={sshPath}
|
||||||
|
onChange={(event) => setSshPath(event.target.value)}
|
||||||
|
placeholder="/srv/backups/fofos"
|
||||||
|
className="w-full font-mono text-xs"
|
||||||
|
autoComplete="off"
|
||||||
|
spellCheck={false}
|
||||||
|
/>
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
size="sm"
|
||||||
|
className="shrink-0"
|
||||||
|
onClick={() => void test()}
|
||||||
|
disabled={busy !== null || !sshHost.trim() || !sshPath.trim()}
|
||||||
|
>
|
||||||
|
{busy === 'test' ? <Loader2 className="size-3.5 animate-spin" /> : <PlugZap className="size-3.5" />}
|
||||||
|
Test
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
{/* No password field, and there will not be one: this runs with nobody at a terminal, so it
|
||||||
|
authenticates with the owner's SSH key and stores no credential of its own. */}
|
||||||
|
<p className="text-xs text-muted-foreground/70">Authenticated with this machine’s SSH key.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="space-y-1">
|
||||||
|
<p className="text-xs font-medium">
|
||||||
|
Keep <span className="font-normal text-muted-foreground/70">newest per database, both ends</span>
|
||||||
|
</p>
|
||||||
|
<Input
|
||||||
|
value={keep}
|
||||||
|
onChange={(event) => setKeep(event.target.value.replace(/[^0-9]/g, ''))}
|
||||||
|
placeholder="all"
|
||||||
|
className="w-24 font-mono text-xs"
|
||||||
|
autoComplete="off"
|
||||||
|
inputMode="numeric"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
{error && <p className="text-xs text-destructive">{error}</p>}
|
{error && <p className="text-xs text-destructive">{error}</p>}
|
||||||
|
|
||||||
<div className="flex justify-end gap-2">
|
<div className="flex justify-end gap-2">
|
||||||
<Button variant="ghost" size="sm" onClick={() => setOpen(false)} disabled={busy !== null}>
|
<Button variant="ghost" size="sm" onClick={() => setOpen(false)} disabled={busy !== null}>
|
||||||
Cancel
|
Cancel
|
||||||
</Button>
|
</Button>
|
||||||
<Button size="sm" onClick={() => void save()} disabled={busy !== null || !cron.trim() || !dir.trim()}>
|
<Button
|
||||||
|
size="sm"
|
||||||
|
onClick={() => void save()}
|
||||||
|
disabled={busy !== null || !cron.trim() || !dir.trim() || !sshHost.trim() || !sshPath.trim()}
|
||||||
|
>
|
||||||
{busy === 'save' ? <Loader2 className="size-3.5 animate-spin" /> : <Check className="size-3.5" />}
|
{busy === 'save' ? <Loader2 className="size-3.5 animate-spin" /> : <Check className="size-3.5" />}
|
||||||
Save
|
Save
|
||||||
</Button>
|
</Button>
|
||||||
|
|||||||
Reference in New Issue
Block a user