From 3fb439fabf5d1e1abda7e6a6e5f744bc9caff6d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Fri, 21 Aug 2026 02:44:25 +0000 Subject: [PATCH] plan: nightly visual regression on a second server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture every route of a JWT-authenticated dashboard each night, diff against a promoted baseline, and have an agent look only at what changed. The blocker is auth, and it fails silently. iris seeds a capture with Cookies, Cookies-wal, Login Data and Local State — a JWT lives in localStorage, which Chrome keeps in Default/Local Storage/leveldb and iris does not copy. So an authenticated capture renders the login screen: a real screenshot of a real page that is the wrong page. Three ways out, with a recommendation. Copying leveldb works and drags every site's localStorage into a temp profile. A token in the URL needs no profile at all and removes the desktop plugin from the plan entirely — that is the one to try first. Seeding through the relay works and needs a browser paired at 3am, which is the constraint iris exists to avoid. The rest is deliberate about where the model is spent: pixel diff first, mask what is known to move rather than raising the threshold, and only changed pairs reach an agent — with "if you cannot tell, say so" in the prompt, because a confident wrong verdict on night four is how this stops being trusted. Baselines are promoted by a person, never by a run succeeding, or a regression becomes the baseline on night two. Also records what Officer lacks: the queue has retries and no schedules, so this is cron. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 4 + docs/visual-regression-plan.md | 149 +++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+) create mode 100644 docs/visual-regression-plan.md diff --git a/CLAUDE.md b/CLAUDE.md index 6e3907c9..84308307 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -438,6 +438,10 @@ explaining why it was safe. mounted and what it knows, the URL-vs-channel split for panel-to-panel communication, and the persistence key families. Read before building a panel app. Its defect list is `docs/workspace-panel-todo.md`. +- `docs/visual-regression-plan.md` — **plan, not built**: nightly screenshot regression against a JWT + dashboard on a second server. The blocker is that a JWT lives in `localStorage` and iris copies only + cookies, so an authenticated capture silently renders the login page — plus why diffing must happen + before any agent is asked to look - `docs/plugin-dependency-scope.md` — a plugin declares WHO must be able to run each host binary. Why the owner's login shell is the wrong question for a `confined` plugin, the two-part reachability check (world-executable AND every parent traversable), and why members' binaries on alpha worked only because diff --git a/docs/visual-regression-plan.md b/docs/visual-regression-plan.md new file mode 100644 index 00000000..70fd9452 --- /dev/null +++ b/docs/visual-regression-plan.md @@ -0,0 +1,149 @@ +# Nightly visual regression, on a second server + +**Plan, not built.** Written 2026-08-21. + +Capture every route of a dashboard behind a JWT login, every night, on a machine nobody is sitting at — +and have an agent look only at what changed. + +The target is **not** Officer and **not** this machine: a separate box running Officer with the `iris` and +`browser` plugins, pointed at a dashboard the owner runs elsewhere. Its login is a JWT, like Officer's. + +## The shape + +``` +target server (Officer + iris + browser) + │ + ├─ cron, 03:00 ──▶ capture N routes with iris → runs//.png + │ compare against runs/baseline/ → changed[] + │ if changed is empty → record and stop, wake nobody + │ else → open a chat turn with the changed pairs attached + │ + └─ agent reads only the changed images, judges each: intended, or regression + → notification with the verdict, and a link to the run +``` + +The expensive, fuzzy step happens only where pixels actually moved. On a quiet night the run costs a few +seconds of Chrome and nothing else. + +## The hard part: a JWT lives in localStorage, and iris cannot put it there + +This is the thing to solve first, because everything else is plumbing. + +Cookie-authenticated sites work today: `iris-browser` copies `Cookies`, `Cookies-wal`, `Login Data` and +`Local State` out of a signed-in profile, so a capture rides the same session. + +**A JWT dashboard is not cookie-authenticated.** The token sits in `localStorage`, which Chrome keeps in +`Default/Local Storage/leveldb/` — a directory `iris-browser` does not copy. So a capture of a +token-authenticated app renders the login screen, and does it silently: the screenshot is real, the page +is real, and it is the wrong page. + +Three ways out, worth choosing between deliberately. + +### A. Copy `Local Storage/leveldb` too — smallest change, biggest blast radius + +One more entry in `seed_sessions`. It works, and it copies **every** site's localStorage into the +capture profile, not just the target's. That is someone's whole browser state in a temp directory, and +the directory is a LevelDB — copying it under a live browser can catch a compaction mid-write, which +Cookies-the-file mostly cannot. + +Viable, and the least attractive of the three. + +### B. Log in during the capture — no profile at all + +Most JWT dashboards accept a token in the URL, or can be given one by a first-run page. If the target +does, then: + +``` +iris 'https://dash.example.com/?token=' -o run/home.png +``` + +and there is no profile, no desktop, no signed-in browser anywhere. The token comes from a service +account minted for this, stored in Officer's secret store, and never appears in a screenshot. + +**This is the recommended path** when the target supports it. It removes desktop from the picture +entirely, which removes the only reason that plugin was in this plan. + +### C. Seed via the relay, capture via iris + +Pair a browser once, use the relay to `evaluate` the token into `localStorage`, and let iris capture from +that profile. Works without changing either plugin, and needs a browser paired at 3am — which is exactly +the constraint iris exists to avoid. + +Keep as the fallback for a target that supports neither a token URL nor a clean profile copy. + +## What to capture + +**Routes come from a list the repo owns**, not from crawling. A crawler finds what exists; a list says +what is supposed to exist, and a route disappearing is the regression you most want caught. + +```jsonc +// routes.json, in the dashboard's own repository +[ + { "path": "/", "name": "home" }, + { "path": "/reports", "name": "reports", "waitFor": "[data-ready]" }, + { "path": "/reports/42", "name": "report-detail" }, +] +``` + +`waitFor` maps to iris's `--wait-for`, which is what stops a spinner being the thing under test. + +Two viewports — one desktop, one phone — doubles the run and catches the majority of layout breakage. +`--jobs` renders them in parallel. + +## Diffing, before any agent is involved + +Compare each capture with the same route's baseline. A pixel count, a threshold, nothing clever: + +- **identical** → nothing happens +- **differs under threshold** (antialiasing, a clock, a random avatar) → nothing happens, but log it +- **differs over threshold** → into `changed[]` + +The threshold exists because something on the page always moves. **Mask what is known to move** rather +than raising the threshold until it passes — a timestamp region is a rectangle to blank before comparing, +not a reason to accept 3% drift everywhere. + +Officer has no image diff today. `sharp` is already a plugin dependency (atelier) and can do it; a raw +pixel loop over two decoded buffers is about thirty lines and needs nothing. + +**Baselines are promoted, never automatic.** A run does not overwrite the baseline because it succeeded — +it does so when someone says "yes, that is the new correct". Otherwise a regression becomes the baseline +on night two and disappears. + +## What reaches the agent + +Only changed pairs, and each one as **two images plus the route name**: baseline and current. The +question asked is narrow, because a broad one gets a broad answer: + +> This is `/reports` before and after. Describe what changed, then say whether it is a layout break, a +> content change, or noise. If you cannot tell, say so. + +"If you cannot tell, say so" earns its place — a model asked to judge will judge, and a confident wrong +verdict on night four is how the whole thing stops being trusted. + +The verdicts land in a notification (`officer-notify` is already core) with a count and a link, not the +images. Nobody wants twenty pictures on a phone at 3am. + +## What does not exist yet + +In rough order of how much they block: + +1. **`--wait-for` against a JWT app** — solve auth first (§B), everything else is downstream. +2. **An image diff.** Thirty lines, or `sharp`. +3. **A scheduler.** Officer's queue has retries, not schedules — `scheduleRetry` is the only timed thing + in it. So this is system cron calling a script, or a `setInterval` in a plugin sidecar the way the + nudger does it. Cron is the honest answer for something that runs once a night. +4. **A place for runs and baselines.** `$OFFICER_ROOT/data/visual-regression/{baseline,runs}/` — outside + any plugin directory, because a plugin update replaces that. +5. **Whatever holds this.** A plugin of its own on that server, with `routes.json` as its config and a + panel showing the last run. It is not iris and it is not the relay; it is the thing that uses them. + +## What this deliberately is not + +**Not a test runner.** It proves a page renders and looks like it did yesterday. It cannot tell you a +button is wired to the wrong handler, and it should not be stretched to try — that is a real test suite, +and the two answer different questions. + +**Not a crawler.** See routes, above. + +**Not the browser relay.** The relay drives a live browser someone is signed into; this runs when nobody +is. They overlap only in that both produce screenshots. If §B holds, the relay has no role in this at all.