officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

document the usage panel and what testing it would mean

Browse Source 
it has never run: db:push has not been applied and neither process has been
restarted. docs/ai-usage-panel.md records where every number comes from and
why officer cannot derive any of them itself, the two things the live
payload settled (every model-scoped bucket is null while limits[] carries a
real fable window; the window is named by kind, never by the model name),
the measured costs, and the staleness rules — which are the part most likely
to be wrong, because they assert things about data they did not measure.

sixteen ordered tests in five groups. group C is the actual feature. test 10
is the one that would break silently: a refresh must not overwrite an
account it could not measure.

also records what the scan measurement found, since it decides the deferred
work — 74.7% of message ids appear in more than one transcript file, so
per-file aggregates cannot be summed and an incremental scan is not the
one-liner it looks like.

TODO.md gains an entry under "waiting on the owner to run it", beside plan
mode, which is also still unexercised.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-18 04:55:30 +00:00
co-authored by Claude Opus 5
parent 9745f54fa1
commit 19f558077e
2 changed files with 267 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ai-usage-panel.md
+245
View File
@@ -0,0 +1,245 @@
# The AI usage panel
**Built 2026-08-18. Never run against a live server.** The parts with unit tests are noted below; nothing
here has been exercised through HTTP, through a real turn, or with two accounts. The test plan at the end is
what "tested" would mean.
Click the harness pill in the chat composer — the label reading **Claude Code**, left of the background-job
button — and it opens what Claude Code's own usage display shows: limits with reset countdowns, tokens by
day, tokens by model, with one tab per Anthropic account this machine has seen.
## Before it works
```bash
bun db:push # creates ai_provider_usage
pm2 restart officer officer-claude-code
```
`db:push` first. Without the table the panel still opens and still shows the signed-in account — the stored
read catches and returns an empty list — but there are no tabs and nothing survives an account switch.
Both processes are needed for the _other_ change that shipped alongside this (`model:resolved`, see
"Related" below). The usage panel itself only needs `officer`.
## Where the numbers come from, and why not from Officer
**Officer does not have these numbers and cannot derive them.** Per-session cost is accumulated in memory in
`websocket.ts` and dies with `pm2 restart officer`; nothing aggregates it across sessions, days or models.
More importantly the rate-limit windows are a property of the **Anthropic account**, spent by every `claude`
on the machine — this chat, a terminal, a task runner, a pipeline step. A figure derived from what the chat
socket happened to observe would be lower than the truth by however much was spent elsewhere, and silently so.
So everything is read from the CLI's own record:
| what | source |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| Session / Weekly / model-scoped windows | `GET https://api.anthropic.com/api/oauth/usage`, `Authorization: Bearer <accessToken>`, `anthropic-beta: oauth-2025-04-20` |
| The access token, and the plan label (`Max 20x`) | `~/.claude/.credentials.json` → `claudeAiOauth` |
| Which account is signed in | `~/.claude.json` → `oauthAccount.emailAddress` |
| Tokens by day, tokens by model | scanning `~/.claude/projects/**/*.jsonl` |
`CLAUDE_CONFIG_DIR` is honoured for the first, third and fourth.
The shape and the non-obvious correctness details are lifted from omarchy's `omarchy-agent-usage-claude`
collector (`basecamp/omarchy`, branch `quattro`) rather than rediscovered — specifically the dedup by message
id, the four token counters, the percent-versus-fraction scale detection, and reading model-scoped windows
out of `limits[]`. The account-email part comes from the owner's fork, `gitea.officer.dev/OMARCHI/pastilhas.agents`,
which added it for the same reason it matters here: percentages you cannot attribute to an account are
percentages you cannot act on.
### Two things the live payload settled
Probed 2026-08-18 against the real endpoint; the response is the fixture in `claude-usage.test.ts`.
1. **Every model-scoped bucket is null.** `seven_day_opus`, `seven_day_sonnet` and friends are all `null`,
while `limits[]` carries a real `weekly_scoped` entry for Fable at 6%. A reader that trusts the flat
buckets silently drops a window the account is actually spending against.
2. **The window is named by `kind`, never by the model name.** A display name like `Opus 5 (1M context)`
contains "1M", and parsing a window out of the title turns that into a one-minute window.
### Cost
Measured on this machine, 30 transcript files totalling ~150MB:
| | cold | note |
| --------------- | ---------------- | ----------------------------- |
| Transcript scan | ~440ms, one core | 43k lines, 18.6k `JSON.parse` |
| Limits probe | ~600ms | one HTTPS round trip |
| Account read | ~2ms | 50KB JSON |
They run concurrently, so a cold read is **~600ms**; warm is ~1ms. Both halves are cached server-side for
60s, independently, so a failed probe never throws away a good scan.
**The scan grows monotonically** — transcripts are never deleted. Today it is fine. It is the number that
eventually is not; see "Deferred".
## The cache, and why it exists
Not for speed. **Half a reading cannot be taken at all for an account you are not signed into**: the limits
need the credential the CLI holds for the _current_ login, and the token counts come from that login's
transcripts. Switch accounts and the previous one is not slow to read — it is unreadable.
The owner holds a personal account and a company one and switches between them. The question worth answering
at a glance is _which of my accounts has room right now_, and only a stored reading per account can answer it.
### The table
`ai_provider_usage` (`src/databases/officer_db/src/ai-provider-usage/`)
- Unique on `(user_id, ai_provider, email_account)` — so switching logins **accumulates** rather than
overwrites. Keyed on the Officer user as well, because one Officer account holds several Anthropic logins
and eventually several Officer accounts will each hold their own.
- `ai_provider` carries a CHECK constraint (`claude-code`, `opencode`), per the house rule — a provider the
code stops supporting must not sit there unnoticed.
- The reading itself is one `jsonb` document. The three sections are arrays whose shape is provider-specific
— a prepaid provider reports a credit balance and no windows at all — so columns would be mostly null for
everyone and a migration per provider. A snapshot is never edited, only replaced whole, which is what makes
a document safe here.
- `updated_at` is when the reading was **measured**, not when the row was touched. It is set explicitly on
the update half of the upsert, because a column default only applies to inserts.
### The write
`snapshotClaudeUsage` fires on the `result` message in `createMessageHandler` — the end of a turn, while
you are reading the answer. `void`, so nothing about the turn depends on it, and it swallows its own
failures: a missed snapshot costs one stale row until the next turn.
Capped by the reader's own 60s cache, so a burst of short turns is not a burst of 150MB scans.
**Owner-gated, and not as a permissions nicety.** `getOwnerHomeDir` ignores the email it is passed, so a
member's turn would write _the owner's_ usage under _the member's_ user id. That is not a leak, it is a
wrong row, and it would be invisible.
### The read
`GET /chat/usage` answers from Postgres alone — no scan, no network — so the panel paints instantly with
every account, **including the ones that cannot be measured**. `?refresh=1` re-measures the signed-in account,
upserts it, and returns the merged list. The client fires both and swaps the fresh tab in when it lands.
One exception: on a first-ever open, where the signed-in account has no stored row, the plain call measures
inline rather than returning an empty panel.
Owner-only in the handler rather than in the registry — `chat` is a `confined` permission, so a member
holding it reaches every route in this router. A 403, not an empty reading: members run `claude` as their own
Linux account with their own credential, so "what is my usage" for them is a different query against a
different login, and answering it with the owner's data would be the wrong answer confidently given.
## Staleness: what a cached row means later
This is the part most likely to be wrong in a way that matters, because it is the part that asserts things
about data it did not measure.
**Countdowns survive staleness exactly.** `resetsAt` is an absolute instant, not a duration, so a row measured
six hours ago has an as-of percentage and a _perfectly correct_ "resets in 14h 20m", ticked client-side with
no network and no signed-in session. That asymmetry is why a cached row is worth showing rather than
apologising for.
**Past its reset, the percentage is zero** (`currentLimit` in `useClaudeUsage.ts`). The stored figure is not
merely stale there — it is wrong in the dangerous direction: a company account snapshotted at 78% whose weekly
window rolled over overnight would still read 78% when in fact it is empty and available.
- **Weekly rolls forward.** The window length is in the `kind`, so the next boundary is arithmetic rather
than a guess: add seven days until it lands in the future. This is the case the feature is for.
- **Session does not.** The five-hour window starts on first use rather than on a grid, so once it lapses
there is no clock running. 0%, and no countdown line — not because it cannot be computed, but because
there is nothing to count down to.
A non-current tab carries a **"Not signed in · as of 20m ago"** line. The live tab does not: saying "just
now" under a number being measured as you look at it is noise.
`fetchedAt` is the **older** of the limits and scan halves, not a stamp applied on the way out. A reading
served from cache was measured when the cache was filled; stamping on exit would make every stored row claim
to be current, which defeats the age label entirely.
## Files
| file | what |
| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `src/servers/api/chat/claude-usage.ts` | reads credentials, account, limits, transcripts. Owner-only _structurally_ — takes no identity at all |
| `src/servers/api/chat/usage-store.ts` | snapshot / list / merge across accounts |
| `src/servers/api/chat/chat.ts` | `GET /chat/usage[?refresh=1]` |
| `src/servers/api/chat/websocket.ts` | the turn-end snapshot |
| `src/databases/officer_db/src/ai-provider-usage/` | schema + queries |
| `src/workspaces/officerdev/src/apps/Chat/useClaudeUsage.ts` | the two-query hook and every formatter |
| `src/workspaces/officerdev/src/apps/Chat/components/UsagePopover.tsx` | the pill and the panel |
## What has unit tests, and what has not
**Tested** — `limitsFromPayload` against the live payload (7 cases), `friendlyModelName`, `currentLimit`'s
staleness rules, `mergeUsage`'s invariants, and the formatters.
**Not tested by anything** — every I/O path and the whole UI: the route, the owner refusal, the turn-end
write, the transcript scan, the tabs, the cache-first swap.
## Test plan
Do these in order; each depends on the last. `pastilhas` is the owner.
### A. It works at all
1. `bun db:push`, then `pm2 restart officer`. Confirm the server boots — `assertPermissionTotality` runs
before `serve()`, and a router without a registry entry refuses to boot. No registry line was added
(the route lives inside the existing `chat` router), so a boot failure here means something else.
2. `psql officer -c '\d ai_provider_usage'` — four columns plus the unique index and the CHECK.
3. Open a chat, click the **Claude Code** pill. Expect: plan and email in the header, three limit bars with
countdowns, seven days, models. Compare against the omarchy bar panel on the same machine — the
percentages should agree, the token totals may not (omarchy merges opencode and pi sessions; this does not).
### B. The snapshot
4. Send one prompt and let the turn finish. `psql officer -c 'select user_id, ai_provider, email_account, plan, updated_at from ai_provider_usage'` — one row, `updated_at` within the last minute.
5. Send a second prompt. `updated_at` moves. It should NOT move by more than the turn took plus 60s of cache.
6. `pm2 logs officer` during a turn — no error from `snapshotClaudeUsage`, and no added latency to the turn.
### C. Two accounts — the actual feature
7. `claude` → sign out → sign in as the _other_ account. Confirm `~/.claude.json` now names it.
8. Reopen the pill. Expect: **two tabs**, the new account selected, the old one showing
_"Not signed in · as of Nm ago"_ with its last-known percentages.
9. Click the old tab. Percentages are as-of; the **weekly countdown still ticks** and is correct.
10. Take a turn on the new account, reopen. The new account's row updates; **the old account's row is
untouched** — this is the one that would silently break, since a refresh must not overwrite an account
it could not measure.
### D. Staleness
11. Hardest to test honestly: it needs a row whose window has lapsed. Either wait, or
`update ai_provider_usage set payload = jsonb_set(payload, '{limits,0,resetsAt}', '"2026-08-01T01:00:00Z"')`
on a non-current row. Expect **0%** and a countdown rolled forward to a future date for a weekly window;
**0% and no countdown line** for a session window.
12. Leave the panel open for a minute — countdowns tick down (30s interval), and only while it is open.
### E. Refusals and degradation
13. As a **member** with the `chat` grant: the pill is a plain label, not a button. `curl` the route with
their token → 403.
14. Rename `~/.claude/.credentials.json` and reopen: limits section reports "not signed in", **token history
still renders**. Put it back.
15. Drop the table (`drop table ai_provider_usage`) and reopen: the panel still shows the signed-in account,
with no tabs. Then `bun db:push` again.
16. Open the pill on an **OpenCode** conversation: plain label, no button.
## Deferred
- **The scan is O(all history) and history only grows.** ~440ms today at 150MB. The files are append-only
JSONL, so an incremental scan keyed on `(path, size, mtime)` is the obvious fix — but dedup is by message
id and is _global_ across files, which is what makes it non-trivial. A measurement on 2026-08-18 found
**74.7% of message ids appear in more than one file** (19,696 usage entries → 5,869 unique), so per-file
aggregates cannot simply be summed. Not urgent; revisit when the cold read passes a second or two.
- **Members have no usage of their own.** They run their own `claude` with their own credential, and the
reader is structurally owner-only. The table is already keyed by `user_id`, so this is a reader change,
not a schema one.
- **`opencode` is in the provider CHECK but nothing writes it.** It has no usage endpoint and no readable
transcripts. The constraint names it so the day it does, this is a writer and not a migration.
- **No way to forget an account.** `forgetProviderUsage` exists and nothing calls it; a tab for a login you
no longer use is clutter with no delete affordance.
## Related
Shipped the same day, same area, also untested live: **`model:resolved`** — the model dropdown used to render
the literal string `claude-code` for any chat that named no tier, because `DEFAULT_MODEL` is the bare harness
and names no catalogue entry. The harness reports its actual model in `system.init`; the parser was throwing
it away. Now forwarded, but **only when the turn named no tier** — the alias map is lossy (`opus[1m]` and
`opusplan` both answer as `claude-opus-5`), so forwarding unconditionally would relabel the user's own choice.
Display only: not durable, not recorded, never written into `selectedModel`. That one needs
`officer-claude-code` restarted as well as `officer`.