officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs:check — the documentation's checkable claims, against the code

Browse Source 
"Update the docs before a deploy" is a step that gets skipped at two in the morning, and its absence is
invisible. Every piece of drift found today was mechanical — a count, a route name, a process list — so it is
checkable, and a check that exits non-zero cannot be quietly not done.

It found two things on its first run, both wrong for over a week:

  platform/CLAUDE.md    "27 entries as of 2026-08-13"    the registry holds 18
  officer-dev/CLAUDE.md "Officer is twenty PM2 entries"  there are 13

The second was worse than a wrong number. It listed eleven processes that do not run here — vault, slskd,
headscale, transmission, invoiceshelf, wallet, photos, jellyfin, gitea, music, vnc — while omitting six that
do, and named two mybiblepal entries as sharing the daemon when nothing does. Accurate on 2026-08-07, and
nobody re-derives a list of processes while editing the sentence around it. Both are corrected, with what
they used to claim recorded, because the interesting part is how a maintained list decays rather than that it
did.

Three checks: the permission registry count, the PM2 entry count, and every settings route Spotlight offers
resolving to a section some screen declares. That last one is the class the counts cannot catch — a name, not
a number — and it is the drift that shipped a dead link to /settings/users.

THE RULE FOR ADDING ONE, written into the file: a check compares a documented claim against a value DERIVED
AT RUN TIME, never against a second hard-coded copy. A check with the answer written into it is one more
place to drift, and it drifts in the same edit that misses the doc.

What it deliberately cannot do is read prose and judge it. It looks only at claims with a machine-checkable
counterpart — a small fraction of the documentation, and the fraction that rots fastest.

The settings check parses source rather than importing it, and says why: those sections are object literals
inside components, so importing them drags React and a DOM into a CLI.

`skipped:` lines are printed rather than swallowed — a machine without pm2 cannot check that count, and
saying so beats reporting a pass it did not perform.

Recorded as required before tagging in CLAUDE.md, and in prompts/deploy-commander.md as explicitly NOT the
rollout agent's job: it executes a plan, and a prose edit between two hosts is not that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-22 16:42:15 +00:00
co-authored by Claude Opus 5
parent 29d998b6c3
commit 4a206086af
5 changed files with 220 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/docs-check.ts
+148
View File
@@ -0,0 +1,148 @@
/*
Does the documentation still match the code?
── Why a script and not a habit ──
"Update the docs before a deploy" is a step that gets skipped at two in the morning, and its absence is
invisible. Every piece of drift found on 2026-08-22 was MECHANICAL — a count, a route name, a process
list — so it is checkable, and a check that exits non-zero cannot be quietly not done.
What it will not do is read prose and judge it. It looks only at claims with a machine-checkable
counterpart, which is a small fraction of the documentation and the fraction that rots fastest, because a
number is exactly the thing nobody re-derives while editing a sentence around it.
Real examples this was written from, both live at the time:
platform/CLAUDE.md "27 entries as of 2026-08-13" → the registry held 18
officer-dev/CLAUDE.md "Officer is twenty PM2 entries" → there were 13
Spotlight's settings list 9 rows against 21 sections, and one route that does not exist
── The rule for adding a check ──
A check must compare a documented claim against a value DERIVED AT RUN TIME, never against a second
hard-coded copy. A check with the answer written into it is one more place to drift, and it will drift in
the same edit that misses the doc.
*/
import { readFileSync, existsSync } from 'node:fs';
import { join, resolve } from 'node:path';
type Finding = { file: string; claim: string; documented: string; actual: string };
type CheckResult = { findings: Finding[]; skipped?: string };
const ROOT = resolve(import.meta.dir, '..');
const WORKSPACE = resolve(ROOT, '..');
const read = (path: string): string | null => (existsSync(path) ? readFileSync(path, 'utf-8') : null);
/** Spelled-out numbers appear in prose more often than digits, and both need catching. */
const WORDS: Record<string, number> = {
ten: 10,
eleven: 11,
twelve: 12,
thirteen: 13,
fourteen: 14,
fifteen: 15,
sixteen: 16,
seventeen: 17,
eighteen: 18,
nineteen: 19,
twenty: 20,
'twenty-one': 21,
'twenty-two': 22,
'twenty-three': 23,
'twenty-four': 24,
'twenty-five': 25,
};
const asNumber = (raw: string): number | null => {
const digits = Number(raw);
if (Number.isFinite(digits)) return digits;
return WORDS[raw.toLowerCase()] ?? null;
};
/**
* A documented count, found by a pattern with one capture group, compared to the live value.
*
* Returns a finding per MISMATCH, not per occurrence: a doc stating the right number twice is fine.
*/
function checkCount(file: string, pattern: RegExp, actual: number, claim: string): CheckResult {
const body = read(file);
if (!body) return { findings: [], skipped: `${file} not found` };
const findings: Finding[] = [];
for (const match of body.matchAll(pattern)) {
const stated = asNumber(match[1] ?? '');
if (stated === null || stated === actual) continue;
findings.push({ file, claim, documented: String(stated), actual: String(actual) });
}
return { findings };
}
async function main(): Promise<void> {
const all: Finding[] = [];
const skipped: string[] = [];
const record = (r: CheckResult) => {
all.push(...r.findings);
if (r.skipped) skipped.push(r.skipped);
};
// 1. How many permissions the registry holds. The number appears in prose and is never recomputed.
const { PERMISSIONS } = await import('../src/servers/permissions/registry');
record(
checkCount(
join(ROOT, 'CLAUDE.md'),
/([0-9]+|[a-z-]+) entries as of/gi,
PERMISSIONS.length,
'permission registry entries',
),
);
// 2. How many PM2 entries Officer is. Documented as going stale "every time a sidecar lands", which is an
// admission that it needs a check rather than a caveat.
const pm2 = Bun.spawnSync([process.env.SHELL || '/bin/sh', '-ic', 'pm2 jlist'], { stdout: 'pipe', stderr: 'pipe' });
if (pm2.exitCode === 0) {
try {
const list = JSON.parse(pm2.stdout.toString()) as Array<{ name: string }>;
const officer = list.filter((p) => p.name === 'officer' || p.name.startsWith('officer-')).length;
record(
checkCount(
join(WORKSPACE, 'CLAUDE.md'),
/Officer is ([a-z-]+|[0-9]+) PM2 entries/gi,
officer,
'officer PM2 entries',
),
);
} catch {
skipped.push('pm2 jlist did not parse');
}
} else {
// Honest rather than silent: a machine with no pm2 cannot check this, and saying so is the point.
skipped.push('pm2 unavailable — PM2 entry count unchecked');
}
// 3. Every settings path Spotlight offers must exist as a declared section. This is the drift that shipped
// a dead link to /settings/users, and it is the class the other two cannot catch: a name, not a count.
const { SPOTLIGHT_SETTINGS_PATHS } = await import('./docs-check-settings');
const declared = await SPOTLIGHT_SETTINGS_PATHS();
for (const bad of declared.missing) {
all.push({
file: 'Spotlight/useSpotlightItems.ts',
claim: 'settings route offered by Spotlight',
documented: bad,
actual: 'no screen declares that section',
});
}
for (const note of skipped) console.log(` skipped: ${note}`);
if (all.length === 0) {
console.log('docs:check — every checkable claim matches the code');
return;
}
console.log(`\ndocs:check — ${all.length} claim(s) no longer match the code:\n`);
for (const f of all) {
console.log(` ${f.file}`);
console.log(` ${f.claim}: says ${f.documented}, actually ${f.actual}\n`);
}
process.exit(1);
}
await main();