From 19f558077eab211801f794da3f3113871bffaba8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Tue, 18 Aug 2026 04:55:30 +0000 Subject: [PATCH] document the usage panel and what testing it would mean MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit it has never run: db:push has not been applied and neither process has been restarted. docs/ai-usage-panel.md records where every number comes from and why officer cannot derive any of them itself, the two things the live payload settled (every model-scoped bucket is null while limits[] carries a real fable window; the window is named by kind, never by the model name), the measured costs, and the staleness rules — which are the part most likely to be wrong, because they assert things about data they did not measure. sixteen ordered tests in five groups. group C is the actual feature. test 10 is the one that would break silently: a refresh must not overwrite an account it could not measure. also records what the scan measurement found, since it decides the deferred work — 74.7% of message ids appear in more than one transcript file, so per-file aggregates cannot be summed and an incremental scan is not the one-liner it looks like. TODO.md gains an entry under "waiting on the owner to run it", beside plan mode, which is also still unexercised. Co-Authored-By: Claude Opus 5 --- TODO.md | 22 ++++ docs/ai-usage-panel.md | 245 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 docs/ai-usage-panel.md diff --git a/TODO.md b/TODO.md index f3e98a6a..b206f8f6 100644 --- a/TODO.md +++ b/TODO.md @@ -447,6 +447,28 @@ outside the example's new README: ## Waiting on the owner to run it +- [ ] **The AI usage panel has never run.** Landed 2026-08-18. Click the harness pill in the chat composer + and it shows Claude Code's own usage display — limits with countdowns, tokens by day and by model — + with one tab per Anthropic account, so a login you have switched away from still answers. + + **Two things must happen before it does anything**: `bun db:push` (it adds `ai_provider_usage`) and + `pm2 restart officer officer-claude-code`. Neither has been run. Without the push the panel opens and + shows the signed-in account with no tabs, which is the designed degradation rather than a failure. + + **`docs/ai-usage-panel.md` is the checklist** — sixteen ordered tests in five groups. Group C is the + feature: sign in as the second account and confirm the first one keeps its numbers and its countdown. + Test 10 is the one most likely to be wrong — a refresh must not overwrite an account it could not + measure, and that failure is silent. + + Safe to leave untested. Everything is read-only except one fire-and-forget upsert at turn end, and the + pill was already an inert label before this. The one thing worth watching is `pm2 logs officer` during + a turn: the snapshot is `void`ed and swallows its own errors, so a persistent failure shows up only + there and as a row that never updates. + + Shipped alongside it and also unexercised: the model dropdown no longer renders the literal string + `claude-code` for a chat that named no tier. That half needs `officer-claude-code` restarted, not just + `officer` — the parser in the sidecar is what emits it. + - [ ] **Plan mode is live and has never been exercised.** Landed `1b811dfa` on 2026-08-17; all three processes were restarted the same evening, so the code IS running. Not one part of it has been used: not the toggle, not the plan card, not the round trip from the sidecar's pending promise to a browser diff --git a/docs/ai-usage-panel.md b/docs/ai-usage-panel.md new file mode 100644 index 00000000..d7ef72db --- /dev/null +++ b/docs/ai-usage-panel.md @@ -0,0 +1,245 @@ +# The AI usage panel + +**Built 2026-08-18. Never run against a live server.** The parts with unit tests are noted below; nothing +here has been exercised through HTTP, through a real turn, or with two accounts. The test plan at the end is +what "tested" would mean. + +Click the harness pill in the chat composer — the label reading **Claude Code**, left of the background-job +button — and it opens what Claude Code's own usage display shows: limits with reset countdowns, tokens by +day, tokens by model, with one tab per Anthropic account this machine has seen. + +## Before it works + +```bash +bun db:push # creates ai_provider_usage +pm2 restart officer officer-claude-code +``` + +`db:push` first. Without the table the panel still opens and still shows the signed-in account — the stored +read catches and returns an empty list — but there are no tabs and nothing survives an account switch. + +Both processes are needed for the _other_ change that shipped alongside this (`model:resolved`, see +"Related" below). The usage panel itself only needs `officer`. + +## Where the numbers come from, and why not from Officer + +**Officer does not have these numbers and cannot derive them.** Per-session cost is accumulated in memory in +`websocket.ts` and dies with `pm2 restart officer`; nothing aggregates it across sessions, days or models. +More importantly the rate-limit windows are a property of the **Anthropic account**, spent by every `claude` +on the machine — this chat, a terminal, a task runner, a pipeline step. A figure derived from what the chat +socket happened to observe would be lower than the truth by however much was spent elsewhere, and silently so. + +So everything is read from the CLI's own record: + +| what | source | +| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | +| Session / Weekly / model-scoped windows | `GET https://api.anthropic.com/api/oauth/usage`, `Authorization: Bearer `, `anthropic-beta: oauth-2025-04-20` | +| The access token, and the plan label (`Max 20x`) | `~/.claude/.credentials.json` → `claudeAiOauth` | +| Which account is signed in | `~/.claude.json` → `oauthAccount.emailAddress` | +| Tokens by day, tokens by model | scanning `~/.claude/projects/**/*.jsonl` | + +`CLAUDE_CONFIG_DIR` is honoured for the first, third and fourth. + +The shape and the non-obvious correctness details are lifted from omarchy's `omarchy-agent-usage-claude` +collector (`basecamp/omarchy`, branch `quattro`) rather than rediscovered — specifically the dedup by message +id, the four token counters, the percent-versus-fraction scale detection, and reading model-scoped windows +out of `limits[]`. The account-email part comes from the owner's fork, `gitea.officer.dev/OMARCHI/pastilhas.agents`, +which added it for the same reason it matters here: percentages you cannot attribute to an account are +percentages you cannot act on. + +### Two things the live payload settled + +Probed 2026-08-18 against the real endpoint; the response is the fixture in `claude-usage.test.ts`. + +1. **Every model-scoped bucket is null.** `seven_day_opus`, `seven_day_sonnet` and friends are all `null`, + while `limits[]` carries a real `weekly_scoped` entry for Fable at 6%. A reader that trusts the flat + buckets silently drops a window the account is actually spending against. +2. **The window is named by `kind`, never by the model name.** A display name like `Opus 5 (1M context)` + contains "1M", and parsing a window out of the title turns that into a one-minute window. + +### Cost + +Measured on this machine, 30 transcript files totalling ~150MB: + +| | cold | note | +| --------------- | ---------------- | ----------------------------- | +| Transcript scan | ~440ms, one core | 43k lines, 18.6k `JSON.parse` | +| Limits probe | ~600ms | one HTTPS round trip | +| Account read | ~2ms | 50KB JSON | + +They run concurrently, so a cold read is **~600ms**; warm is ~1ms. Both halves are cached server-side for +60s, independently, so a failed probe never throws away a good scan. + +**The scan grows monotonically** — transcripts are never deleted. Today it is fine. It is the number that +eventually is not; see "Deferred". + +## The cache, and why it exists + +Not for speed. **Half a reading cannot be taken at all for an account you are not signed into**: the limits +need the credential the CLI holds for the _current_ login, and the token counts come from that login's +transcripts. Switch accounts and the previous one is not slow to read — it is unreadable. + +The owner holds a personal account and a company one and switches between them. The question worth answering +at a glance is _which of my accounts has room right now_, and only a stored reading per account can answer it. + +### The table + +`ai_provider_usage` (`src/databases/officer_db/src/ai-provider-usage/`) + +- Unique on `(user_id, ai_provider, email_account)` — so switching logins **accumulates** rather than + overwrites. Keyed on the Officer user as well, because one Officer account holds several Anthropic logins + and eventually several Officer accounts will each hold their own. +- `ai_provider` carries a CHECK constraint (`claude-code`, `opencode`), per the house rule — a provider the + code stops supporting must not sit there unnoticed. +- The reading itself is one `jsonb` document. The three sections are arrays whose shape is provider-specific + — a prepaid provider reports a credit balance and no windows at all — so columns would be mostly null for + everyone and a migration per provider. A snapshot is never edited, only replaced whole, which is what makes + a document safe here. +- `updated_at` is when the reading was **measured**, not when the row was touched. It is set explicitly on + the update half of the upsert, because a column default only applies to inserts. + +### The write + +`snapshotClaudeUsage` fires on the `result` message in `createMessageHandler` — the end of a turn, while +you are reading the answer. `void`, so nothing about the turn depends on it, and it swallows its own +failures: a missed snapshot costs one stale row until the next turn. + +Capped by the reader's own 60s cache, so a burst of short turns is not a burst of 150MB scans. + +**Owner-gated, and not as a permissions nicety.** `getOwnerHomeDir` ignores the email it is passed, so a +member's turn would write _the owner's_ usage under _the member's_ user id. That is not a leak, it is a +wrong row, and it would be invisible. + +### The read + +`GET /chat/usage` answers from Postgres alone — no scan, no network — so the panel paints instantly with +every account, **including the ones that cannot be measured**. `?refresh=1` re-measures the signed-in account, +upserts it, and returns the merged list. The client fires both and swaps the fresh tab in when it lands. + +One exception: on a first-ever open, where the signed-in account has no stored row, the plain call measures +inline rather than returning an empty panel. + +Owner-only in the handler rather than in the registry — `chat` is a `confined` permission, so a member +holding it reaches every route in this router. A 403, not an empty reading: members run `claude` as their own +Linux account with their own credential, so "what is my usage" for them is a different query against a +different login, and answering it with the owner's data would be the wrong answer confidently given. + +## Staleness: what a cached row means later + +This is the part most likely to be wrong in a way that matters, because it is the part that asserts things +about data it did not measure. + +**Countdowns survive staleness exactly.** `resetsAt` is an absolute instant, not a duration, so a row measured +six hours ago has an as-of percentage and a _perfectly correct_ "resets in 14h 20m", ticked client-side with +no network and no signed-in session. That asymmetry is why a cached row is worth showing rather than +apologising for. + +**Past its reset, the percentage is zero** (`currentLimit` in `useClaudeUsage.ts`). The stored figure is not +merely stale there — it is wrong in the dangerous direction: a company account snapshotted at 78% whose weekly +window rolled over overnight would still read 78% when in fact it is empty and available. + +- **Weekly rolls forward.** The window length is in the `kind`, so the next boundary is arithmetic rather + than a guess: add seven days until it lands in the future. This is the case the feature is for. +- **Session does not.** The five-hour window starts on first use rather than on a grid, so once it lapses + there is no clock running. 0%, and no countdown line — not because it cannot be computed, but because + there is nothing to count down to. + +A non-current tab carries a **"Not signed in · as of 20m ago"** line. The live tab does not: saying "just +now" under a number being measured as you look at it is noise. + +`fetchedAt` is the **older** of the limits and scan halves, not a stamp applied on the way out. A reading +served from cache was measured when the cache was filled; stamping on exit would make every stored row claim +to be current, which defeats the age label entirely. + +## Files + +| file | what | +| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| `src/servers/api/chat/claude-usage.ts` | reads credentials, account, limits, transcripts. Owner-only _structurally_ — takes no identity at all | +| `src/servers/api/chat/usage-store.ts` | snapshot / list / merge across accounts | +| `src/servers/api/chat/chat.ts` | `GET /chat/usage[?refresh=1]` | +| `src/servers/api/chat/websocket.ts` | the turn-end snapshot | +| `src/databases/officer_db/src/ai-provider-usage/` | schema + queries | +| `src/workspaces/officerdev/src/apps/Chat/useClaudeUsage.ts` | the two-query hook and every formatter | +| `src/workspaces/officerdev/src/apps/Chat/components/UsagePopover.tsx` | the pill and the panel | + +## What has unit tests, and what has not + +**Tested** — `limitsFromPayload` against the live payload (7 cases), `friendlyModelName`, `currentLimit`'s +staleness rules, `mergeUsage`'s invariants, and the formatters. + +**Not tested by anything** — every I/O path and the whole UI: the route, the owner refusal, the turn-end +write, the transcript scan, the tabs, the cache-first swap. + +## Test plan + +Do these in order; each depends on the last. `pastilhas` is the owner. + +### A. It works at all + +1. `bun db:push`, then `pm2 restart officer`. Confirm the server boots — `assertPermissionTotality` runs + before `serve()`, and a router without a registry entry refuses to boot. No registry line was added + (the route lives inside the existing `chat` router), so a boot failure here means something else. +2. `psql officer -c '\d ai_provider_usage'` — four columns plus the unique index and the CHECK. +3. Open a chat, click the **Claude Code** pill. Expect: plan and email in the header, three limit bars with + countdowns, seven days, models. Compare against the omarchy bar panel on the same machine — the + percentages should agree, the token totals may not (omarchy merges opencode and pi sessions; this does not). + +### B. The snapshot + +4. Send one prompt and let the turn finish. `psql officer -c 'select user_id, ai_provider, email_account, plan, updated_at from ai_provider_usage'` — one row, `updated_at` within the last minute. +5. Send a second prompt. `updated_at` moves. It should NOT move by more than the turn took plus 60s of cache. +6. `pm2 logs officer` during a turn — no error from `snapshotClaudeUsage`, and no added latency to the turn. + +### C. Two accounts — the actual feature + +7. `claude` → sign out → sign in as the _other_ account. Confirm `~/.claude.json` now names it. +8. Reopen the pill. Expect: **two tabs**, the new account selected, the old one showing + _"Not signed in · as of Nm ago"_ with its last-known percentages. +9. Click the old tab. Percentages are as-of; the **weekly countdown still ticks** and is correct. +10. Take a turn on the new account, reopen. The new account's row updates; **the old account's row is + untouched** — this is the one that would silently break, since a refresh must not overwrite an account + it could not measure. + +### D. Staleness + +11. Hardest to test honestly: it needs a row whose window has lapsed. Either wait, or + `update ai_provider_usage set payload = jsonb_set(payload, '{limits,0,resetsAt}', '"2026-08-01T01:00:00Z"')` + on a non-current row. Expect **0%** and a countdown rolled forward to a future date for a weekly window; + **0% and no countdown line** for a session window. +12. Leave the panel open for a minute — countdowns tick down (30s interval), and only while it is open. + +### E. Refusals and degradation + +13. As a **member** with the `chat` grant: the pill is a plain label, not a button. `curl` the route with + their token → 403. +14. Rename `~/.claude/.credentials.json` and reopen: limits section reports "not signed in", **token history + still renders**. Put it back. +15. Drop the table (`drop table ai_provider_usage`) and reopen: the panel still shows the signed-in account, + with no tabs. Then `bun db:push` again. +16. Open the pill on an **OpenCode** conversation: plain label, no button. + +## Deferred + +- **The scan is O(all history) and history only grows.** ~440ms today at 150MB. The files are append-only + JSONL, so an incremental scan keyed on `(path, size, mtime)` is the obvious fix — but dedup is by message + id and is _global_ across files, which is what makes it non-trivial. A measurement on 2026-08-18 found + **74.7% of message ids appear in more than one file** (19,696 usage entries → 5,869 unique), so per-file + aggregates cannot simply be summed. Not urgent; revisit when the cold read passes a second or two. +- **Members have no usage of their own.** They run their own `claude` with their own credential, and the + reader is structurally owner-only. The table is already keyed by `user_id`, so this is a reader change, + not a schema one. +- **`opencode` is in the provider CHECK but nothing writes it.** It has no usage endpoint and no readable + transcripts. The constraint names it so the day it does, this is a writer and not a migration. +- **No way to forget an account.** `forgetProviderUsage` exists and nothing calls it; a tab for a login you + no longer use is clutter with no delete affordance. + +## Related + +Shipped the same day, same area, also untested live: **`model:resolved`** — the model dropdown used to render +the literal string `claude-code` for any chat that named no tier, because `DEFAULT_MODEL` is the bare harness +and names no catalogue entry. The harness reports its actual model in `system.init`; the parser was throwing +it away. Now forwarded, but **only when the turn named no tier** — the alias map is lossy (`opus[1m]` and +`opusplan` both answer as `claude-opus-5`), so forwarding unconditionally would relabel the user's own choice. +Display only: not durable, not recorded, never written into `selectedModel`. That one needs +`officer-claude-code` restarted as well as `officer`.