From 4a206086affac11938eaa4f58bd241ffa61530ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Sat, 22 Aug 2026 16:42:15 +0000 Subject: [PATCH] =?UTF-8?q?docs:check=20=E2=80=94=20the=20documentation's?= =?UTF-8?q?=20checkable=20claims,=20against=20the=20code?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Update the docs before a deploy" is a step that gets skipped at two in the morning, and its absence is invisible. Every piece of drift found today was mechanical — a count, a route name, a process list — so it is checkable, and a check that exits non-zero cannot be quietly not done. It found two things on its first run, both wrong for over a week: platform/CLAUDE.md "27 entries as of 2026-08-13" the registry holds 18 officer-dev/CLAUDE.md "Officer is twenty PM2 entries" there are 13 The second was worse than a wrong number. It listed eleven processes that do not run here — vault, slskd, headscale, transmission, invoiceshelf, wallet, photos, jellyfin, gitea, music, vnc — while omitting six that do, and named two mybiblepal entries as sharing the daemon when nothing does. Accurate on 2026-08-07, and nobody re-derives a list of processes while editing the sentence around it. Both are corrected, with what they used to claim recorded, because the interesting part is how a maintained list decays rather than that it did. Three checks: the permission registry count, the PM2 entry count, and every settings route Spotlight offers resolving to a section some screen declares. That last one is the class the counts cannot catch — a name, not a number — and it is the drift that shipped a dead link to /settings/users. THE RULE FOR ADDING ONE, written into the file: a check compares a documented claim against a value DERIVED AT RUN TIME, never against a second hard-coded copy. A check with the answer written into it is one more place to drift, and it drifts in the same edit that misses the doc. What it deliberately cannot do is read prose and judge it. It looks only at claims with a machine-checkable counterpart — a small fraction of the documentation, and the fraction that rots fastest. The settings check parses source rather than importing it, and says why: those sections are object literals inside components, so importing them drags React and a DOM into a CLI. `skipped:` lines are printed rather than swallowed — a machine without pm2 cannot check that count, and saying so beats reporting a pass it did not perform. Recorded as required before tagging in CLAUDE.md, and in prompts/deploy-commander.md as explicitly NOT the rollout agent's job: it executes a plan, and a prose edit between two hosts is not that. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 13 ++- package.json | 3 +- prompts/deploy-commander.md | 15 ++++ scripts/docs-check-settings.ts | 43 ++++++++++ scripts/docs-check.ts | 148 +++++++++++++++++++++++++++++++++ 5 files changed, 220 insertions(+), 2 deletions(-) create mode 100644 scripts/docs-check-settings.ts create mode 100644 scripts/docs-check.ts diff --git a/CLAUDE.md b/CLAUDE.md index 3e6b74ba..ae6fa81a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -194,7 +194,7 @@ ahead of everything, and it re-verifies the token itself so it covers routes tha - `permissions/registry.ts` — the single enumeration of what the platform can do, in five kinds: `core` (every account, not deniable), `app` (**the grantable surface**), `confined` (grantable, but only to an account that has a Linux user), `execution` and `admin` (owner only, and `execution` is - never grantable at any level). 27 entries as of 2026-08-13. + never grantable at any level). **18 entries as of 2026-08-22** — verified by `bun run docs:check`, which fails if this number drifts again: 5 core, 3 app, 3 confined, 3 execution, 4 admin. It said 27 for nine days. - `permissions/authorize.ts` — resolves "may this account do this". Owner short-circuits first; every other answer is role grants plus core, with `execution`/`admin` stripped even if a row grants them, and `confined` stripped for an account with no `osUser`. @@ -224,9 +224,20 @@ bunx tsgo # typecheck (not tsc) bun test # tests bun format # prettier over every dirty file — see the note below before running it bun db:push # apply the schema to Postgres +bun docs:check # do the docs still match the code? REQUIRED before tagging a release bun setup # runs scripts/install.sh — blank machine to running platform ``` +**`bun docs:check` before every release, and leave it green.** It compares documented claims against values +derived at run time — the permission count, the PM2 entry count, every settings route Spotlight offers — and +exits non-zero on drift. A check must never compare a doc against a second hard-coded copy; that is one more +place to drift, and it drifts in the same edit that misses the doc. + +It is narrow on purpose. It cannot read prose and judge it, only claims with a machine-checkable +counterpart — which is a small part of the documentation and the part that rots fastest, because a number is +exactly what nobody re-derives while editing the sentence around it. Both claims it found on the day it was +written had been wrong for over a week. + `scripts/install.sh` is only an orchestrator — it runs the two halves in order and does nothing itself: `setup/machine-setup/machine-setup.sh` (28 sections: packages, tailnet, runtimes, docker, shell) then `setup/officer-setup.sh` (11: pre-flight, layout, repository, dependencies, database, environment, secrets, diff --git a/package.json b/package.json index 8f9ffd55..f0f6f8be 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,8 @@ "format:all": "prettier --write \"src/**/*.{ts,tsx}\"", "format:check": "prettier --check \"src/**/*.{ts,tsx}\"", "setup": "bash scripts/install.sh", - "check:theme": "bun scripts/check-theme-tokens.ts" + "check:theme": "bun scripts/check-theme-tokens.ts", + "docs:check": "bun run scripts/docs-check.ts" }, "dependencies": { "@anthropic-ai/claude-agent-sdk": "^0.2.41", diff --git a/prompts/deploy-commander.md b/prompts/deploy-commander.md index dbb43ef0..8fd6bbb5 100644 --- a/prompts/deploy-commander.md +++ b/prompts/deploy-commander.md @@ -96,6 +96,21 @@ load-bearing. A missing report is a missing record; a missing lock is a second d It sits under `data/` rather than in the repository so a `git pull` can never clobber it, and so a fresh clone does not arrive believing it has deployed nothing. +## Before a release is tagged — not your job, but know it happened + +Whoever cuts a release runs `bun run docs:check` first and leaves it green. It compares documented claims +against values derived at run time — how many permissions the registry holds, how many PM2 entries Officer +is, whether every settings route Spotlight offers actually exists — and exits non-zero on drift. + +It exists because "update the docs before a deploy" is a step that gets skipped at two in the morning and +whose absence is invisible. On 2026-08-22 it found, immediately: `platform/CLAUDE.md` claiming 27 permission +entries against a registry holding 18, and the workspace `CLAUDE.md` claiming twenty PM2 entries against +thirteen, listing eleven processes that do not run here while omitting six that do. + +**You do not run it and you do not fix documentation.** You execute a plan and report. It is here so that a +report saying "the docs are stale" has somewhere to point, and so nobody adds doc-fixing to a rollout, where +it would put a prose edit between two hosts. + ## The release The tag names the version. The commit it points at carries the rest: diff --git a/scripts/docs-check-settings.ts b/scripts/docs-check-settings.ts new file mode 100644 index 00000000..a36095a8 --- /dev/null +++ b/scripts/docs-check-settings.ts @@ -0,0 +1,43 @@ +import { readFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { Glob } from 'bun'; + +/* + Which settings sections actually exist, read from the screens that declare them. + + Separated from `docs-check.ts` because it is the only check that parses source rather than importing it — + the sections are JSX-adjacent object literals inside components, so importing them would drag React and a + browser environment into a CLI. A regex over `BASE_PATH` and `key:` is the honest tool for that, and it is + narrow enough to be obvious when it breaks. +*/ + +const ROOT = resolve(import.meta.dir, '..'); +const SETTINGS_DIR = join(ROOT, 'src/apps/officer-web/Screens/Dashboard/Settings'); + +/** `/settings/profile` + `change-password` → `/settings/profile/change-password`. */ +export async function declaredSettingsPaths(): Promise> { + const paths = new Set(); + const glob = new Glob('**/*.tsx'); + for await (const file of glob.scan({ cwd: SETTINGS_DIR, absolute: true })) { + const body = readFileSync(file, 'utf-8'); + const base = /BASE_PATH\s*=\s*'([^']+)'/.exec(body)?.[1]; + if (!base) continue; + paths.add(base); + for (const m of body.matchAll(/key:\s*'([^']+)'/g)) paths.add(`${base}/${m[1]}`); + } + return paths; +} + +/** + * Settings routes Spotlight offers, and which of them nothing declares. + * + * Parsed rather than imported for the same reason: `useSpotlightItems` is a hook and pulls in the dock, + * lucide and React. + */ +export async function SPOTLIGHT_SETTINGS_PATHS(): Promise<{ offered: string[]; missing: string[] }> { + const file = join(ROOT, 'src/apps/officer-web/Screens/Dashboard/Layout/Spotlight/useSpotlightItems.ts'); + const body = readFileSync(file, 'utf-8'); + const offered = [...body.matchAll(/to:\s*'(\/settings\/[^']+)'/g)].map((m) => m[1]!); + const declared = await declaredSettingsPaths(); + return { offered, missing: offered.filter((p) => !declared.has(p)) }; +} diff --git a/scripts/docs-check.ts b/scripts/docs-check.ts new file mode 100644 index 00000000..08031127 --- /dev/null +++ b/scripts/docs-check.ts @@ -0,0 +1,148 @@ +/* + Does the documentation still match the code? + + ── Why a script and not a habit ── + + "Update the docs before a deploy" is a step that gets skipped at two in the morning, and its absence is + invisible. Every piece of drift found on 2026-08-22 was MECHANICAL — a count, a route name, a process + list — so it is checkable, and a check that exits non-zero cannot be quietly not done. + + What it will not do is read prose and judge it. It looks only at claims with a machine-checkable + counterpart, which is a small fraction of the documentation and the fraction that rots fastest, because a + number is exactly the thing nobody re-derives while editing a sentence around it. + + Real examples this was written from, both live at the time: + + platform/CLAUDE.md "27 entries as of 2026-08-13" → the registry held 18 + officer-dev/CLAUDE.md "Officer is twenty PM2 entries" → there were 13 + Spotlight's settings list 9 rows against 21 sections, and one route that does not exist + + ── The rule for adding a check ── + + A check must compare a documented claim against a value DERIVED AT RUN TIME, never against a second + hard-coded copy. A check with the answer written into it is one more place to drift, and it will drift in + the same edit that misses the doc. +*/ + +import { readFileSync, existsSync } from 'node:fs'; +import { join, resolve } from 'node:path'; + +type Finding = { file: string; claim: string; documented: string; actual: string }; +type CheckResult = { findings: Finding[]; skipped?: string }; + +const ROOT = resolve(import.meta.dir, '..'); +const WORKSPACE = resolve(ROOT, '..'); + +const read = (path: string): string | null => (existsSync(path) ? readFileSync(path, 'utf-8') : null); + +/** Spelled-out numbers appear in prose more often than digits, and both need catching. */ +const WORDS: Record = { + ten: 10, + eleven: 11, + twelve: 12, + thirteen: 13, + fourteen: 14, + fifteen: 15, + sixteen: 16, + seventeen: 17, + eighteen: 18, + nineteen: 19, + twenty: 20, + 'twenty-one': 21, + 'twenty-two': 22, + 'twenty-three': 23, + 'twenty-four': 24, + 'twenty-five': 25, +}; +const asNumber = (raw: string): number | null => { + const digits = Number(raw); + if (Number.isFinite(digits)) return digits; + return WORDS[raw.toLowerCase()] ?? null; +}; + +/** + * A documented count, found by a pattern with one capture group, compared to the live value. + * + * Returns a finding per MISMATCH, not per occurrence: a doc stating the right number twice is fine. + */ +function checkCount(file: string, pattern: RegExp, actual: number, claim: string): CheckResult { + const body = read(file); + if (!body) return { findings: [], skipped: `${file} not found` }; + const findings: Finding[] = []; + for (const match of body.matchAll(pattern)) { + const stated = asNumber(match[1] ?? ''); + if (stated === null || stated === actual) continue; + findings.push({ file, claim, documented: String(stated), actual: String(actual) }); + } + return { findings }; +} + +async function main(): Promise { + const all: Finding[] = []; + const skipped: string[] = []; + const record = (r: CheckResult) => { + all.push(...r.findings); + if (r.skipped) skipped.push(r.skipped); + }; + + // 1. How many permissions the registry holds. The number appears in prose and is never recomputed. + const { PERMISSIONS } = await import('../src/servers/permissions/registry'); + record( + checkCount( + join(ROOT, 'CLAUDE.md'), + /([0-9]+|[a-z-]+) entries as of/gi, + PERMISSIONS.length, + 'permission registry entries', + ), + ); + + // 2. How many PM2 entries Officer is. Documented as going stale "every time a sidecar lands", which is an + // admission that it needs a check rather than a caveat. + const pm2 = Bun.spawnSync([process.env.SHELL || '/bin/sh', '-ic', 'pm2 jlist'], { stdout: 'pipe', stderr: 'pipe' }); + if (pm2.exitCode === 0) { + try { + const list = JSON.parse(pm2.stdout.toString()) as Array<{ name: string }>; + const officer = list.filter((p) => p.name === 'officer' || p.name.startsWith('officer-')).length; + record( + checkCount( + join(WORKSPACE, 'CLAUDE.md'), + /Officer is ([a-z-]+|[0-9]+) PM2 entries/gi, + officer, + 'officer PM2 entries', + ), + ); + } catch { + skipped.push('pm2 jlist did not parse'); + } + } else { + // Honest rather than silent: a machine with no pm2 cannot check this, and saying so is the point. + skipped.push('pm2 unavailable — PM2 entry count unchecked'); + } + + // 3. Every settings path Spotlight offers must exist as a declared section. This is the drift that shipped + // a dead link to /settings/users, and it is the class the other two cannot catch: a name, not a count. + const { SPOTLIGHT_SETTINGS_PATHS } = await import('./docs-check-settings'); + const declared = await SPOTLIGHT_SETTINGS_PATHS(); + for (const bad of declared.missing) { + all.push({ + file: 'Spotlight/useSpotlightItems.ts', + claim: 'settings route offered by Spotlight', + documented: bad, + actual: 'no screen declares that section', + }); + } + + for (const note of skipped) console.log(` skipped: ${note}`); + if (all.length === 0) { + console.log('docs:check — every checkable claim matches the code'); + return; + } + console.log(`\ndocs:check — ${all.length} claim(s) no longer match the code:\n`); + for (const f of all) { + console.log(` ${f.file}`); + console.log(` ${f.claim}: says ${f.documented}, actually ${f.actual}\n`); + } + process.exit(1); +} + +await main();