platform/docs/visual-regression-plan.md

Nightly visual regression, on a second server

Plan, not built. Written 2026-08-21.

Capture every route of a dashboard behind a JWT login, every night, on a machine nobody is sitting at — and have an agent look only at what changed.

The target is not Officer and not this machine: a separate box running Officer with the iris and browser plugins, pointed at a dashboard the owner runs elsewhere. Its login is a JWT, like Officer's.

The shape

target server (Officer + iris + browser)
  │
  ├─ cron, 03:00 ──▶ capture N routes with iris        → runs/<date>/<route>.png
  │                  compare against runs/baseline/    → changed[]
  │                  if changed is empty → record and stop, wake nobody
  │                  else → open a chat turn with the changed pairs attached
  │
  └─ agent reads only the changed images, judges each: intended, or regression
                     → notification with the verdict, and a link to the run

The expensive, fuzzy step happens only where pixels actually moved. On a quiet night the run costs a few seconds of Chrome and nothing else.

The hard part: a JWT lives in localStorage, and iris cannot put it there

This is the thing to solve first, because everything else is plumbing.

Cookie-authenticated sites work today: iris-browser copies Cookies, Cookies-wal, Login Data and Local State out of a signed-in profile, so a capture rides the same session.

A JWT dashboard is not cookie-authenticated. The token sits in localStorage, which Chrome keeps in Default/Local Storage/leveldb/ — a directory iris-browser does not copy. So a capture of a token-authenticated app renders the login screen, and does it silently: the screenshot is real, the page is real, and it is the wrong page.

Three ways out, worth choosing between deliberately.

A. Copy Local Storage/leveldb too — smallest change, biggest blast radius

One more entry in seed_sessions. It works, and it copies every site's localStorage into the capture profile, not just the target's. That is someone's whole browser state in a temp directory, and the directory is a LevelDB — copying it under a live browser can catch a compaction mid-write, which Cookies-the-file mostly cannot.

Viable, and the least attractive of the three.

B. Log in during the capture — no profile at all

Most JWT dashboards accept a token in the URL, or can be given one by a first-run page. If the target does, then:

iris 'https://dash.example.com/?token=<jwt>' -o run/home.png

and there is no profile, no desktop, no signed-in browser anywhere. The token comes from a service account minted for this, stored in Officer's secret store, and never appears in a screenshot.

This is the recommended path when the target supports it. It removes desktop from the picture entirely, which removes the only reason that plugin was in this plan.

C. Seed via the relay, capture via iris

Pair a browser once, use the relay to evaluate the token into localStorage, and let iris capture from that profile. Works without changing either plugin, and needs a browser paired at 3am — which is exactly the constraint iris exists to avoid.

Keep as the fallback for a target that supports neither a token URL nor a clean profile copy.

What to capture

Routes come from a list the repo owns, not from crawling. A crawler finds what exists; a list says what is supposed to exist, and a route disappearing is the regression you most want caught.

// routes.json, in the dashboard's own repository
[
  { "path": "/", "name": "home" },
  { "path": "/reports", "name": "reports", "waitFor": "[data-ready]" },
  { "path": "/reports/42", "name": "report-detail" },
]

waitFor maps to iris's --wait-for, which is what stops a spinner being the thing under test.

Two viewports — one desktop, one phone — doubles the run and catches the majority of layout breakage. --jobs renders them in parallel.

Diffing, before any agent is involved

Compare each capture with the same route's baseline. A pixel count, a threshold, nothing clever:

• identical → nothing happens
• differs under threshold (antialiasing, a clock, a random avatar) → nothing happens, but log it
• differs over threshold → into changed[]

The threshold exists because something on the page always moves. Mask what is known to move rather than raising the threshold until it passes — a timestamp region is a rectangle to blank before comparing, not a reason to accept 3% drift everywhere.

Officer has no image diff today. sharp is already a plugin dependency (atelier) and can do it; a raw pixel loop over two decoded buffers is about thirty lines and needs nothing.

Baselines are promoted, never automatic. A run does not overwrite the baseline because it succeeded — it does so when someone says "yes, that is the new correct". Otherwise a regression becomes the baseline on night two and disappears.

What reaches the agent

Only changed pairs, and each one as two images plus the route name: baseline and current. The question asked is narrow, because a broad one gets a broad answer:

This is /reports before and after. Describe what changed, then say whether it is a layout break, a content change, or noise. If you cannot tell, say so.

"If you cannot tell, say so" earns its place — a model asked to judge will judge, and a confident wrong verdict on night four is how the whole thing stops being trusted.

The verdicts land in a notification (officer-notify is already core) with a count and a link, not the images. Nobody wants twenty pictures on a phone at 3am.

What does not exist yet

In rough order of how much they block:

1. --wait-for against a JWT app — solve auth first (§B), everything else is downstream.
2. An image diff. Thirty lines, or sharp.
3. A scheduler. Officer's queue has retries, not schedules — scheduleRetry is the only timed thing in it. So this is system cron calling a script, or a setInterval in a plugin sidecar the way the nudger does it. Cron is the honest answer for something that runs once a night.
4. A place for runs and baselines. $OFFICER_ROOT/data/visual-regression/{baseline,runs}/ — outside any plugin directory, because a plugin update replaces that.
5. Whatever holds this. A plugin of its own on that server, with routes.json as its config and a panel showing the last run. It is not iris and it is not the relay; it is the thing that uses them.

What this deliberately is not

Not a test runner. It proves a page renders and looks like it did yesterday. It cannot tell you a button is wired to the wrong handler, and it should not be stretched to try — that is a real test suite, and the two answer different questions.

Not a crawler. See routes, above.

Not the browser relay. The relay drives a live browser someone is signed into; this runs when nobody is. They overlap only in that both produce screenshots. If §B holds, the relay has no role in this at all.