From 764ba23206ddd34e2f7b26ea1934b9da3573fea1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Tue, 18 Aug 2026 09:03:52 +0000 Subject: [PATCH] command hq: per-turn cost ledger, watcher fires as their own table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner asked for individual turn token/dollar costs, a running total, cost over time, and cost per trigger — derived, not stored, following the same rule `running` already uses. Two new tables carry it: - command_hq_watcher_fires: one row per trigger (sessionRowId, taskId, firedAt). Replaces the single watcherLastFiredAt column, which lost the previous fire's timestamp every time a new one landed — fine for "when did it last fire", useless for "how many times, and what did each one cost". - command_hq_turns: one row per completed turn (sessionRowId, fireId nullable, inputTokens, outputTokens, totalUsd). Nothing pre-summed — running total, cost-over-time and cost-per-trigger are all queries over this table. Turn count needed no column either: COUNT(*) already answers it, confirmed explicitly with the owner rather than assumed. command_hq_sessions gains watcherArmedAt/watcherStoppedAt in place of any "last fired" cache — set only on the false<->true transitions, so duration (current run, or the last completed one) is computable without a full transition history table. Two new read routes, GET /commanders/:sessionId/turns and .../fires, expose the raw series behind the cost summary and watcherLastFiredAt (now computed via MAX(fired_at), not read off a column — the field survives in the JSON response under the same name). Bug found and fixed while wiring this up, recorded in docs/command-hq.md: recordCommandHqWatcherFired's signature changed from (userId, sessionId) to (sessionRowId, taskId) and one call site kept the old argument order — tsgo stayed clean because number/string still matched structurally. Caught by re-reading every call site by hand, not by any tool. Needs bun db:push before any route answers with real data. Co-Authored-By: Claude Opus 5 --- docs/command-hq.md | 149 ++++++++++++--- .../officer_db/src/command-hq/index.ts | 16 +- .../officer_db/src/command-hq/queries.ts | 180 ++++++++++++++++-- .../officer_db/src/command-hq/schema.ts | 79 +++++++- src/servers/api/chat/websocket.ts | 40 +++- src/servers/api/command-hq/router.ts | 77 +++++++- 6 files changed, 477 insertions(+), 64 deletions(-) diff --git a/docs/command-hq.md b/docs/command-hq.md index f3a45bb8..c4b1c428 100644 --- a/docs/command-hq.md +++ b/docs/command-hq.md @@ -32,21 +32,27 @@ for exact request/response shapes; the table below is the map, not the full cont | GET | `/commanders[?category=\|none]` | — | omitted = all, `none` = uncategorised bucket. See below | | POST | `/commanders/new` | `{prompt, displayText?, categoryId?}` | creates a commander; `prompt` required. `displayText ?? prompt` titles it, same precedence as the websocket. `-> {sessionId, cwd}` — `sessionId` is Claude's real transcript id, not a placeholder | | GET | `/commanders/:sessionId` | — | Command HQ's own metadata only — NOT the transcript, see below | +| GET | `/commanders/:sessionId/turns` | — | `{turns: [...]}` — the raw per-turn cost ledger, oldest first. What "cost over time" plots from | +| GET | `/commanders/:sessionId/fires` | — | `{fires: [...]}` — every trigger, oldest first. The full history behind `watcherLastFiredAt` | | POST | `/commanders` | `{sessionId, categoryId?}` | files an EXISTING session; idempotent; 404 if the session isn't in the commanders directory. Not the creation path — see `/commanders/new` | -| PATCH | `/commanders/:sessionId` | `{categoryId?, name?}` | `name` proxies to `renameClaudeSession` — the same rename `/chat` uses | +| PATCH | `/commanders/:sessionId` | `{categoryId?, name?, watcherArmed?}` | `name` proxies to `renameClaudeSession`; `watcherArmed: false` disarms (arming has no route — needs a live task id) | | POST | `/purge` | — | sweeps rows whose transcript no longer resolves; returns `{removed}` | | POST | `/commanders/:sessionId/tags` | `{tagId}` | | | DELETE | `/commanders/:sessionId/tags/:tagId` | — | | **A commander's conversation is `GET /chat/sessions/:id`, unchanged.** This router deliberately does not -duplicate it — `GET /commanders/:sessionId` returns only `{sessionId, categoryId, tagIds, running, -createdAt}`. A client wanting the full picture calls both. +duplicate it — `GET /commanders/:sessionId` returns Command HQ's own metadata: `{sessionId, categoryId, +tagIds, running, watcherArmed, watcherArmedAt, watcherStoppedAt, watcherLastFiredAt, cost, createdAt}`. A +client wanting the full picture calls both; a client wanting the raw cost series calls `.../turns` as well +— `cost` here is only the summary (`{turnCount, inputTokens, outputTokens, totalUsd}`), not the list. -**`GET /commanders` is the grid endpoint** and returns the richest shape: every field `/chat/sessions` -already returns per session (`id`, `title`, `cwd`, `createdAt`, `updatedAt`, `messageCount`, …) plus -`categoryId`, `tagIds`, and `running` — a boolean, derived live from the same `task:started`/ -`task:notification` fold `/chat`'s own background-job tray uses, not a stored value. This is the field to -poll if a client wants to know "did my watcher die" without opening it. +**`GET /commanders` is the grid endpoint** and returns the richest per-card shape: every field +`/chat/sessions` already returns per session (`id`, `title`, `cwd`, `createdAt`, `updatedAt`, +`messageCount`, …) plus `categoryId`, `tagIds`, `running` (derived live from the same `task:started`/ +`task:notification` fold `/chat`'s own background-job tray uses), the watcher-arming fields, and the same +`cost` summary the detail route returns — one aggregate query per card, the same cost class `running` +already accepts for this grid. This is the field to poll if a client wants to know "did my watcher die" +without opening it, or "what has this one cost me so far" without a second request. **A row whose transcript is gone is silently absent from every response.** Never a tombstone, never a 404 with a special shape — it simply is not in the list. See "The /chat ↔ /command-hq relationship" below for @@ -183,19 +189,60 @@ an accident of copying a plugin schema). One row per commander. -| column | notes | -| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `user_id` | owner of the row, always the same account that owns the session | -| `session_id` | **Claude's transcript uuid**, not Officer's `sessionId` — the same id `chat_session_models` keys on, for the same reason: everything else that joins to a session (models, events) uses this id | -| `category_id` | FK to `command_hq_categories`, nullable — one category per commander (see below) | -| `created_at` | | +| column | notes | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `user_id` | owner of the row, always the same account that owns the session | +| `session_id` | **Claude's transcript uuid**, not Officer's `sessionId` — the same id `chat_session_models` keys on, for the same reason: everything else that joins to a session (models, events) uses this id | +| `category_id` | FK to `command_hq_categories`, nullable — one category per commander (see below) | +| `watcher_armed` | should the platform keep re-arming this watcher on every fire — see "The watcher loop" | +| `watcher_task_id` | the currently-tracked watcher task, nullable | +| `watcher_armed_at` | since when the CURRENT run has been going; set only on the false→true transition, untouched by re-arms | +| `watcher_stopped_at` | when the current run ended; set on disarm, cleared on the next arm — paired with `watcher_armed_at` | +| `created_at` | | No `name` column (see "Naming"). No `status` column, deliberately: liveness is **derived**, the same way the background-job tray already derives it — fold `chat_session_events` for the session, `task:started` minus `task:notification` (`collectRunningTasks` in `websocket.ts` is the existing implementation of exactly this fold; reuse it, do not reinvent it). A stored status would be a second source of truth that goes stale precisely in the case this feature is used for (see "Later" — silent death on sidecar restart) and nothing -would ever correct it. +would ever correct it. **No `watcher_last_fired_at` either, deliberately removed** — see +`command_hq_watcher_fires` below; the owner's call, once cost tracking made clear that a single overwritten +timestamp was losing information a real table should hold instead. + +### `command_hq_watcher_fires` — one row per trigger + +Added when cost tracking surfaced a real gap: the ORIGINAL design cached "last fired" as a single column on +the commander row, which meant every new fire destroyed the previous one's timestamp. Fine for "when did it +last fire", useless for "how many times, and what did each firing cost" — which is exactly what the owner +asked for. So this became a real table instead of a column, and the commander row lost its cache entirely +(the owner's call: derive `MAX(fired_at)` at read time rather than keep a copy that could only ever describe +the same table this row also has to insert into). + +| column | notes | +| ---------------- | ------------------------------------------ | +| `session_row_id` | FK to `command_hq_sessions.id` | +| `task_id` | which watcher task fired | +| `fired_at` | the only timestamp this row exists to hold | + +### `command_hq_turns` — the cost ledger + +One row per completed turn (every `result` message a commander's session produces, not only the ones that +follow a fire — the setup conversation before a watcher is ever armed costs something too). + +| column | notes | +| ---------------- | ------------------------------------------------------------------------------------------ | +| `session_row_id` | FK to `command_hq_sessions.id` | +| `fire_id` | FK to `command_hq_watcher_fires.id`, nullable — null for turns before the first-ever arm | +| `input_tokens` | | +| `output_tokens` | | +| `total_usd` | `doublePrecision` — a monitoring number, not a billing ledger; see the schema file for why | +| `created_at` | | + +**Nothing is pre-summed.** Running total, cost over time, cost per trigger and turn count are all queries +over this table — `getCommandHqCostSummary` (one `SUM`/`COUNT`), `listCommandHqTurns` (the raw series a +client plots), grouped by `fire_id` for per-trigger. Turn count in particular needed no column at all: +`COUNT(*)` on this table already answers it, the same "derive it" call the owner made for `watcher_armed_at` +and the dropped `watcher_last_fired_at` — raised explicitly when the schema was being designed, not assumed. ### `command_hq_categories` @@ -329,6 +376,21 @@ fixed:** either false start: it only WATCHES for this happening and can say so if it doesn't (see "Built" below) — it does not drive the re-arm and does not respawn anything itself. +**The evidence for step 3 is stronger than either side's first write-up said, and it is worth recording +precisely rather than as "ran fine for a while" (`COMMS/command-hq/mobile-4.md`).** Both notes originally +described the mobile side's own same-turn watcher as having run "for hours, this session." Andre's own +count, checked directly: **at least 10–15 runs, spanning before and after the current mechanism existed, +zero failures.** That is a different class of claim than "it worked tonight" — it is evidence against +_revisiting_ the decision, not just evidence for having made it. Worth stating why that distinction +matters: same-turn self-re-arm looks fragile on paper (it depends on the model correctly doing one specific +thing as the last step of every turn, unsupervised, forever), so the instinct on reading the design cold is +to add a safety net — a supervisor, a timeout, a platform-side check that re-arm actually happened. The +record here is that a dozen-plus real runs across two different surrounding mechanisms never once needed +one. It is also why the platform's role stays _noticing_ (`watcherArmed && !running`, "Deliberately not +built" below) rather than _driving_: noticing is the right amount of machinery for a failure with zero +observed instances, and building more than that would be solving a problem the evidence says does not +occur. + **Built:** - **A third dialog option**, `FireBehavior = 'stop' | 'handoff' | 'keep-going'` (was a `stopOnFire` @@ -354,13 +416,19 @@ fixed:** Known gap: this is in-memory client state, so it does not survive a page refresh between clicking Stop and the notification arriving — an acceptable, narrow window, not attempted to be closed tonight. -- **`command_hq_sessions` gained `watcherArmed`, `watcherTaskId`, `watcherLastFiredAt`.** Unlike `running`, - `watcherArmed` is allowed to be stored — it is the owner's INTENT ("keep re-arming this"), not a live - process fact the platform is reporting on, so nothing external can silently falsify it the way a sidecar - restart falsifies `running`. `watcherTaskId` is closer to that rejected shape and accepts the same known - gap `running` already lives with — see the schema file's docstring. - `armCommandHqWatcher` / `recordCommandHqWatcherFired` / `disarmCommandHqWatcher` in `queries.ts`; exposed - read-only on `GET /commanders` and `GET /commanders/:sessionId`, and `PATCH /commanders/:sessionId` accepts +- **`command_hq_sessions` gained `watcherArmed`, `watcherTaskId`, `watcherArmedAt`, `watcherStoppedAt`.** + Unlike `running`, `watcherArmed` is allowed to be stored — it is the owner's INTENT ("keep re-arming + this"), not a live process fact the platform is reporting on, so nothing external can silently falsify it + the way a sidecar restart falsifies `running`. `watcherTaskId` is closer to that rejected shape and + accepts the same known gap `running` already lives with — see the schema file's docstring. + `watcherArmedAt`/`watcherStoppedAt` are the current-run boundary pair, added a turn later during the + cost-tracking pass once "since when has this been going" was a real question — see "Data model" above. + There is no `watcherLastFiredAt` column any more: that fact moved into `command_hq_watcher_fires` (its own + table, one row per trigger) once cost tracking needed per-fire attribution, not a single overwritten + timestamp. `armCommandHqWatcher` / `recordCommandHqWatcherFired` / `disarmCommandHqWatcher` / + `recordCommandHqTurn` in `queries.ts`; exposed read-only on `GET /commanders` and + `GET /commanders/:sessionId` (`watcherLastFiredAt` survives as a name in the JSON response — it is just + computed there, `MAX(fired_at)`, not read off a column), and `PATCH /commanders/:sessionId` accepts `{watcherArmed: false}` to disarm (arming itself has no route — it needs a live task id, which only a running turn has). - **`armWatcher` threaded end-to-end, client to server.** `BackgroundJobDialog` computes it (`watching && @@ -377,8 +445,10 @@ onFire === 'handoff'`) → `sendWithHiddenPrefix` → `QueuedPrompt.armWatcher` task starting is what the agent's own self-re-arm produces, so the tracked `watcherTaskId` is simply updated to it — the platform is not driving anything here, only keeping up with what the agent already did. On `task:notification`: if it matches the tracked `watcherTaskId` for an armed commander, record the - fire (`watcherLastFiredAt`). Both read `getCommandHqSession` live rather than trusting any closure-local - state, for the same reason as above. + fire — a new row in `command_hq_watcher_fires`, not a column write. Both read `getCommandHqSession` live + rather than trusting any closure-local state, for the same reason as above. On `result`: every completed + turn writes a row to `command_hq_turns`, attributed to the commander's most recent fire (`fire_id`, null + before the first-ever arm) — see "Data model" for why nothing here is pre-summed. - **Known, accepted imprecision:** the "already armed → treat the next task:started as the re-arm" rule cannot tell a genuine self-re-arm apart from an unrelated background task the agent happens to start in the same commander conversation. A commander is one watcher by design, so this is a narrow gap rather @@ -389,10 +459,19 @@ the re-arm mechanism itself — landed as agent-driven, same-turn, after two rej — (4) arming state and the plumbing that watches for it. **Deliberately not built: the platform verifying re-arm happened.** Raised as a possible cheap addition — -since the platform already knows `watcherLastFiredAt` and can already compute `running` from the same fold -`GET /commanders` uses for every card, a client could flag "fired N ago, nothing has run since" without -any new mechanism, purely by reading what already exists. Not built tonight; noted here so it is not lost, -not because it was rejected. +since the platform can already derive last-fired (`MAX(fired_at)` in `command_hq_watcher_fires`) and already +computes `running` from the same fold `GET /commanders` uses for every card, a client could flag "fired N +ago, nothing has run since" without any new mechanism, purely by reading what already exists. Not built +tonight; noted here so it is not lost, not because it was rejected. + +**A real bug, caught by tracing rather than by the compiler, worth stating plainly.** When the cost-tracking +schema changed `recordCommandHqWatcherFired`'s signature from `(userId, sessionId)` to `(sessionRowId, +taskId)`, the ONE existing call site was not updated in the same pass — and `bunx tsgo` stayed clean, because +`userId: number` and `transcriptId: string` still satisfy `sessionRowId: number, taskId: string` +structurally. It would have written a fire row for a commander keyed on the wrong id entirely (the caller's +OWN user id, not the row it belonged to). Found by re-reading every call site the schema change touched +after the fact, not by any tool — worth remembering the next time a signature changes shape rather than +just its types: a compiler that stays quiet is not the same as a diff that is correct. ## Later — explicitly deferred, not forgotten @@ -410,9 +489,17 @@ scope for tonight."_ says running" against "pid check says running" is what turns a false "running" into a third, honest state ("vanished"). Cost matters here: `fuser` is a subprocess spawn per task, so this is a detail-pane check or a batched sweep on load, never a per-card check on every render of the grid. -- **No durable job record.** A commander's only record of its own history is `chat_session_events`, which is - swept after 7 days. The owner does not care how many times a watcher fired or what each firing found — the - chat transcript being the record is intentional, not a gap to fill. +- **No durable job record — REVISED.** True when written; no longer true for cost. The owner still does not + care how many times a watcher fired or WHAT each firing found (the chat transcript remains the only record + of content, and that is intentional), but does want per-turn token/dollar cost durably recorded — + `command_hq_turns` and `command_hq_watcher_fires`, added same session. See "The watcher loop" and the data + model section above. +- **Cost-based limits — explicitly out of scope for tonight, said directly by the owner.** "If this agent + reaches a certain cost in tokens, stop it, or compact the conversation and resend, or clear and resend." + Not built, and deliberately not designed around either: the ledger this session added + (`command_hq_turns`) is exactly the raw material a limit would need (`SUM` since a fire, or since armed), + so nothing here forecloses it — but which action, at what threshold, and whether it is a per-commander + setting or a global one, is all undecided. - **Detail pane contents beyond "the conversation".** The owner asked for logs, status, and transcript together in a split view; scoped down to "click a card, see its details" for v1, with the specifics (separate log tail vs transcript vs live status) left open. `/chat/sessions/:id` (transcript) and diff --git a/src/databases/officer_db/src/command-hq/index.ts b/src/databases/officer_db/src/command-hq/index.ts index aa86098b..47a99ef7 100644 --- a/src/databases/officer_db/src/command-hq/index.ts +++ b/src/databases/officer_db/src/command-hq/index.ts @@ -13,6 +13,11 @@ export { armCommandHqWatcher, recordCommandHqWatcherFired, disarmCommandHqWatcher, + recordCommandHqTurn, + getCommandHqCostSummary, + listCommandHqTurns, + listCommandHqWatcherFires, + getLastFiredAtBySessionRowIds, listCommandHqTags, createCommandHqTag, deleteCommandHqTag, @@ -20,4 +25,13 @@ export { addCommandHqSessionTag, removeCommandHqSessionTag, } from './queries'; -export type { CommandHqCategory, CommandHqSessionRow, CommandHqTag } from './queries'; +export type { + CommandHqCategory, + CommandHqSessionRow, + CommandHqTag, + CommandHqWatcherFire, + CommandHqTurnCost, + CommandHqCostSummary, + CommandHqTurnRow, + CommandHqFireRow, +} from './queries'; diff --git a/src/databases/officer_db/src/command-hq/queries.ts b/src/databases/officer_db/src/command-hq/queries.ts index c33e97f9..2786ab46 100644 --- a/src/databases/officer_db/src/command-hq/queries.ts +++ b/src/databases/officer_db/src/command-hq/queries.ts @@ -1,6 +1,13 @@ -import { and, eq, inArray, isNull } from 'drizzle-orm'; +import { and, desc, eq, inArray, isNull, sql } from 'drizzle-orm'; import { db } from '../db'; -import { commandHqCategories, commandHqSessionTags, commandHqSessions, commandHqTags } from './schema'; +import { + commandHqCategories, + commandHqSessionTags, + commandHqSessions, + commandHqTags, + commandHqTurns, + commandHqWatcherFires, +} from './schema'; // ── Categories ── @@ -50,7 +57,8 @@ export type CommandHqSessionRow = { categoryId: number | null; watcherArmed: boolean; watcherTaskId: string | null; - watcherLastFiredAt: Date | null; + watcherArmedAt: Date | null; + watcherStoppedAt: Date | null; createdAt: Date; }; @@ -61,7 +69,8 @@ const SESSION_ROW_COLUMNS = { categoryId: commandHqSessions.categoryId, watcherArmed: commandHqSessions.watcherArmed, watcherTaskId: commandHqSessions.watcherTaskId, - watcherLastFiredAt: commandHqSessions.watcherLastFiredAt, + watcherArmedAt: commandHqSessions.watcherArmedAt, + watcherStoppedAt: commandHqSessions.watcherStoppedAt, createdAt: commandHqSessions.createdAt, }; @@ -136,46 +145,175 @@ export async function getCommandHqSession(userId: number, sessionId: string): Pr * Arm (or re-arm) a commander's watcher — called on the first `task:started` after the dialog submits a * `handoff` job, and again on every SUBSEQUENT `task:started` while already armed, which is how the * AGENT's own self-re-arm (same turn, per `handoffContract`) is picked up: the platform does not drive - * the re-arm, it only notices the new task id the agent's own next watch produced. Idempotent in effect: - * arming with the same task id twice just writes the same values again. + * the re-arm, it only notices the new task id the agent's own next watch produced. + * + * `watcherArmedAt` is set ONLY on the false→true transition — a `CASE` inside the same UPDATE rather than + * a read-then-write, so there is no race between checking the current value and writing the new one. Every + * later re-arm call leaves it alone, which is what makes it "since when has the CURRENT run been going" + * rather than "since the last re-arm". `watcherStoppedAt` is unconditionally cleared, because arming from a + * previously-disarmed state means a stale stop time would otherwise sit there describing a run that is no + * longer the current one. */ export async function armCommandHqWatcher(userId: number, sessionId: string, taskId: string): Promise { const rows = await db .update(commandHqSessions) - .set({ watcherArmed: true, watcherTaskId: taskId }) + .set({ + watcherArmed: true, + watcherTaskId: taskId, + watcherArmedAt: sql`CASE WHEN ${commandHqSessions.watcherArmed} = false THEN now() ELSE ${commandHqSessions.watcherArmedAt} END`, + watcherStoppedAt: null, + }) .where(and(eq(commandHqSessions.userId, userId), eq(commandHqSessions.sessionId, sessionId))) .returning({ id: commandHqSessions.id }); return rows.length > 0; } +export type CommandHqWatcherFire = { id: number; firedAt: Date }; + /** - * The watcher fired. Records WHEN for display — does not touch `watcherArmed` or `watcherTaskId`, which - * stay exactly as they were through the fired-then-acting window, right up until the next `armCommandHqWatcher` - * replaces the task id on re-arm. + * The watcher fired — a "trigger". Inserts a row in `command_hq_watcher_fires` rather than updating a + * scalar, so cost can later be attributed per trigger rather than only in aggregate — see + * `commandHqWatcherFires`'s own docstring. Does not touch `watcherArmed` or `watcherTaskId` on the + * commander row, which stay exactly as they were through the fired-then-acting window, right up until the + * next `armCommandHqWatcher` replaces the task id on re-arm. + * + * Takes the commander's own row id, not `(userId, sessionId)` — every caller already has the row in hand + * (it had to check `watcherTaskId` matched before deciding this WAS the tracked watcher firing), so this + * avoids resolving it a second time. */ -export async function recordCommandHqWatcherFired(userId: number, sessionId: string): Promise { - const rows = await db - .update(commandHqSessions) - .set({ watcherLastFiredAt: new Date() }) - .where(and(eq(commandHqSessions.userId, userId), eq(commandHqSessions.sessionId, sessionId))) - .returning({ id: commandHqSessions.id }); - return rows.length > 0; +export async function recordCommandHqWatcherFired(sessionRowId: number, taskId: string): Promise { + const [row] = await db + .insert(commandHqWatcherFires) + .values({ sessionRowId, taskId }) + .returning({ id: commandHqWatcherFires.id, firedAt: commandHqWatcherFires.firedAt }); + return row!; } /** - * Disarm a commander's watcher. No product path calls this yet — there is no UI for it — but the - * mechanism needs an off switch to exist before it needs a button, and `watcherTaskId` must be cleared - * alongside `watcherArmed`, not left pointing at a task nothing will ever re-arm from again. + * Disarm a commander's watcher. No product path calls this yet on the web side — there is no UI for it — + * but the mechanism needs an off switch to exist before it needs a button, and `watcherTaskId` must be + * cleared alongside `watcherArmed`, not left pointing at a task nothing will ever re-arm from again. + * `watcherStoppedAt` is set here and ONLY here — the other end of the pair `watcherArmedAt` began. */ export async function disarmCommandHqWatcher(userId: number, sessionId: string): Promise { const rows = await db .update(commandHqSessions) - .set({ watcherArmed: false, watcherTaskId: null }) + .set({ watcherArmed: false, watcherTaskId: null, watcherStoppedAt: new Date() }) .where(and(eq(commandHqSessions.userId, userId), eq(commandHqSessions.sessionId, sessionId))) .returning({ id: commandHqSessions.id }); return rows.length > 0; } +export type CommandHqTurnCost = { inputTokens: number; outputTokens: number; totalUsd: number }; + +/** + * Record one completed turn's cost — called on every `result` a commander's session produces, not only + * the ones that follow a fire. The raw ledger `docs/command-hq.md` describes: running total, cost over + * time, cost per trigger and turn count are all queries over this table, nothing pre-summed here. + * + * `fireId` is resolved to whichever trigger is CURRENTLY open for this commander — the most recent fire + * row, if any — inside the same statement, so a turn during the setup conversation (before any fire has + * ever happened) correctly gets `fireId: null` rather than a query needing to special-case it. + */ +export async function recordCommandHqTurn(sessionRowId: number, cost: CommandHqTurnCost): Promise { + // Two statements rather than one INSERT ... SELECT: a fire landing in the gap between them would + // misattribute at most one turn to the setup conversation instead of the trigger that just started, + // which is a smaller and more honest error than the complexity of a correlated subquery buys back — + // fires happen far less often than turns complete, so the gap is rarely even open. + const [currentFire] = await db + .select({ id: commandHqWatcherFires.id }) + .from(commandHqWatcherFires) + .where(eq(commandHqWatcherFires.sessionRowId, sessionRowId)) + .orderBy(desc(commandHqWatcherFires.firedAt)) + .limit(1); + await db.insert(commandHqTurns).values({ + sessionRowId, + fireId: currentFire?.id ?? null, + inputTokens: cost.inputTokens, + outputTokens: cost.outputTokens, + totalUsd: cost.totalUsd, + }); +} + +export type CommandHqCostSummary = { + turnCount: number; + inputTokens: number; + outputTokens: number; + totalUsd: number; +}; + +/** The running total for a commander — every turn it has ever cost, summed. Zeroes, never null, for a commander with no turns recorded yet. */ +export async function getCommandHqCostSummary(sessionRowId: number): Promise { + const [row] = await db + .select({ + turnCount: sql`count(*)::int`, + inputTokens: sql`coalesce(sum(${commandHqTurns.inputTokens}), 0)::int`, + outputTokens: sql`coalesce(sum(${commandHqTurns.outputTokens}), 0)::int`, + totalUsd: sql`coalesce(sum(${commandHqTurns.totalUsd}), 0)::float8`, + }) + .from(commandHqTurns) + .where(eq(commandHqTurns.sessionRowId, sessionRowId)); + return row ?? { turnCount: 0, inputTokens: 0, outputTokens: 0, totalUsd: 0 }; +} + +export type CommandHqTurnRow = { + id: number; + fireId: number | null; + inputTokens: number; + outputTokens: number; + totalUsd: number; + createdAt: Date; +}; + +/** Every turn's cost for a commander, oldest first — the raw series a client plots cost-over-time from. */ +export async function listCommandHqTurns(sessionRowId: number): Promise { + return db + .select({ + id: commandHqTurns.id, + fireId: commandHqTurns.fireId, + inputTokens: commandHqTurns.inputTokens, + outputTokens: commandHqTurns.outputTokens, + totalUsd: commandHqTurns.totalUsd, + createdAt: commandHqTurns.createdAt, + }) + .from(commandHqTurns) + .where(eq(commandHqTurns.sessionRowId, sessionRowId)) + .orderBy(commandHqTurns.createdAt); +} + +export type CommandHqFireRow = { id: number; taskId: string; firedAt: Date }; + +/** Every trigger for a commander, oldest first. */ +export async function listCommandHqWatcherFires(sessionRowId: number): Promise { + return db + .select({ + id: commandHqWatcherFires.id, + taskId: commandHqWatcherFires.taskId, + firedAt: commandHqWatcherFires.firedAt, + }) + .from(commandHqWatcherFires) + .where(eq(commandHqWatcherFires.sessionRowId, sessionRowId)) + .orderBy(commandHqWatcherFires.firedAt); +} + +/** + * The most recent fire for each of the given commanders, batched — one query for a whole grid rather than + * one per card, the same shape `listCommandHqSessionTags` already uses. "Last fired" has no column of its + * own (dropped deliberately — see the schema file); this is where the grid gets it instead. + */ +export async function getLastFiredAtBySessionRowIds(sessionRowIds: number[]): Promise> { + if (!sessionRowIds.length) return new Map(); + const rows = await db + .select({ + sessionRowId: commandHqWatcherFires.sessionRowId, + lastFiredAt: sql`max(${commandHqWatcherFires.firedAt})`, + }) + .from(commandHqWatcherFires) + .where(inArray(commandHqWatcherFires.sessionRowId, sessionRowIds)) + .groupBy(commandHqWatcherFires.sessionRowId); + return new Map(rows.map((r) => [r.sessionRowId, r.lastFiredAt])); +} + /** Move a commander to a category, or `null` to uncategorise it explicitly (distinct from never having had one). */ export async function setCommandHqSessionCategory( userId: number, diff --git a/src/databases/officer_db/src/command-hq/schema.ts b/src/databases/officer_db/src/command-hq/schema.ts index fb1a94b4..d11ef069 100644 --- a/src/databases/officer_db/src/command-hq/schema.ts +++ b/src/databases/officer_db/src/command-hq/schema.ts @@ -1,4 +1,14 @@ -import { boolean, integer, pgTable, serial, text, timestamp, uniqueIndex } from 'drizzle-orm/pg-core'; +import { + boolean, + doublePrecision, + index, + integer, + pgTable, + serial, + text, + timestamp, + uniqueIndex, +} from 'drizzle-orm/pg-core'; import { users } from '../auth/schema'; /** @@ -88,13 +98,76 @@ export const commandHqSessions = pgTable( * exactly the window a card would show as "armed, acting on what just fired" rather than "watching". */ watcherTaskId: text('watcher_task_id'), - /** When the watcher last fired. Display only — "Armed, last fired 3h ago". Never read for logic. */ - watcherLastFiredAt: timestamp('watcher_last_fired_at', { withTimezone: true }), + /** + * When the CURRENT (or most recent) run started — set the moment `watcherArmed` first flips false→true, + * left untouched by every later self-re-arm, so a watcher that has been cycling for weeks still reports + * its true start rather than its last re-arm. Answers "how long has this been running" while armed, and + * "how long did that run last" (paired with `watcherStoppedAt`) once it isn't. Not a full arm/disarm + * history — only the boundaries of the current or most recent run, which is what a card needs; a table + * of every transition was rejected as more than this needs, the same call `watcherLastFiredAt` lost for + * the opposite reason (not enough — see `commandHqWatcherFires` above). + */ + watcherArmedAt: timestamp('watcher_armed_at', { withTimezone: true }), + /** + * When the current run ENDED — set on disarm, cleared back to null on the next arm. Paired with + * `watcherArmedAt` so a duration is computable even after the watcher stops: while armed this is null + * (`watcherArmedAt` to now); once disarmed both are set and the gap between them is how long that run + * lasted. Both null means never armed at all. + */ + watcherStoppedAt: timestamp('watcher_stopped_at', { withTimezone: true }), createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), }, (table) => [uniqueIndex('uq_command_hq_sessions_user_session').on(table.userId, table.sessionId)], ); +/** + * One row per time a commander's watcher fired — a "trigger". Nothing but the raw fact and when it + * happened: no cached "last fired" lives on `commandHqSessions` — that is `MAX(firedAt)` for the commander, + * computed at read time, the same "derive it, do not store a copy" rule `running` above already follows. + * + * This is what makes "cost per trigger" answerable at all. Before this table existed, a fire only ever + * updated a single timestamp column, which meant the PREVIOUS fire's time was gone the moment a new one + * landed — fine for "when did it last fire", useless for "how many times, and what did each one cost". + */ +export const commandHqWatcherFires = pgTable( + 'command_hq_watcher_fires', + { + id: serial('id').primaryKey(), + sessionRowId: integer('session_row_id') + .notNull() + .references(() => commandHqSessions.id, { onDelete: 'cascade' }), + /** Which watcher task fired — `commandHqSessions.watcherTaskId` at the moment this row was written. */ + taskId: text('task_id').notNull(), + firedAt: timestamp('fired_at', { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [index('idx_command_hq_watcher_fires_session').on(table.sessionRowId, table.firedAt)], +); + +/** + * One row per completed turn — every `result` message a commander's session produces, cost attached. + * The raw ledger: running total, cost over time, cost per trigger and turn count are all queries over + * this table, nothing pre-summed or cached elsewhere. `totalUsd` is a plain float (`doublePrecision`), + * matching `MessageCost.totalUSD` as the harness itself reports it — this is a monitoring number, not a + * billing ledger, and the accumulated float error over a realistic turn count is far below what the + * harness's own cost estimate is accurate to in the first place. + */ +export const commandHqTurns = pgTable( + 'command_hq_turns', + { + id: serial('id').primaryKey(), + sessionRowId: integer('session_row_id') + .notNull() + .references(() => commandHqSessions.id, { onDelete: 'cascade' }), + /** Which trigger this turn belongs to. Null for the setup conversation before the first-ever arm. */ + fireId: integer('fire_id').references(() => commandHqWatcherFires.id, { onDelete: 'set null' }), + inputTokens: integer('input_tokens').notNull(), + outputTokens: integer('output_tokens').notNull(), + totalUsd: doublePrecision('total_usd').notNull(), + createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [index('idx_command_hq_turns_session').on(table.sessionRowId, table.createdAt)], +); + /** A user-created label. Many-to-many with commanders, unlike categories — schema only for v1; no UI yet. */ export const commandHqTags = pgTable( 'command_hq_tags', diff --git a/src/servers/api/chat/websocket.ts b/src/servers/api/chat/websocket.ts index 71f9fe32..c3b8e391 100644 --- a/src/servers/api/chat/websocket.ts +++ b/src/servers/api/chat/websocket.ts @@ -33,6 +33,7 @@ import { getCommandHqSession, armCommandHqWatcher, recordCommandHqWatcherFired, + recordCommandHqTurn, } from 'officerdb'; import { resolveHomeDir } from '@@/user-home'; import { claudeLoginState } from '@@/os-user-claude'; @@ -570,8 +571,12 @@ function createMessageHandler( const taskId = msg.taskId; void getCommandHqSession(userId, capturedTranscriptId) .then((row) => { + // `row.id` — the commander's OWN row id, not `userId` — is the FK `command_hq_watcher_fires` + // actually takes. Passing `userId`/`transcriptId` here type-checks by coincidence (number, + // string) while writing a fire row for the wrong commander entirely; caught by tracing the + // call, not by the compiler. if (row?.watcherArmed && row.watcherTaskId === taskId) { - return recordCommandHqWatcherFired(userId, capturedTranscriptId); + return recordCommandHqWatcherFired(row.id, taskId); } }) .catch((err: unknown) => { @@ -584,6 +589,39 @@ function createMessageHandler( } } + /* + One turn's cost, recorded whenever it ends — not only on a firing's acting turn. The setup + conversation before a watcher is ever armed costs something too, and `fireId` is null for exactly + those turns (`recordCommandHqTurn` resolves it, not this call site). This is the raw ledger + `docs/command-hq.md` describes: running total, cost over time, cost per trigger and turn count are + all queries over it, nothing summed here. + */ + if (msg.type === 'result' && commandHq && transcriptId) { + const userId = session.userId; + if (userId) { + const capturedTranscriptId = transcriptId; + // Named explicitly, not spread — `MessageCost.totalUSD` and `CommandHqTurnCost.totalUsd` differ + // only in casing, which a spread would carry both keys of and TypeScript would wave through + // (excess-property checking does not catch a spread the way it catches a literal). + const cost = { + inputTokens: msg.cost.inputTokens, + outputTokens: msg.cost.outputTokens, + totalUsd: msg.cost.totalUSD, + }; + void getCommandHqSession(userId, capturedTranscriptId) + .then((row) => { + if (row) return recordCommandHqTurn(row.id, cost); + }) + .catch((err: unknown) => { + logger.warn('Could not record commander turn cost', { + sessionId, + transcriptId: capturedTranscriptId, + error: String(err), + }); + }); + } + } + /* A turn ended — take a usage snapshot while the person is reading the answer. diff --git a/src/servers/api/command-hq/router.ts b/src/servers/api/command-hq/router.ts index 038c087b..c37e0850 100644 --- a/src/servers/api/command-hq/router.ts +++ b/src/servers/api/command-hq/router.ts @@ -25,12 +25,17 @@ import { setCommandHqSessionCategory, purgeCommandHqSessions, disarmCommandHqWatcher, + getCommandHqCostSummary, + listCommandHqTurns, + listCommandHqWatcherFires, + getLastFiredAtBySessionRowIds, listCommandHqSessionTags, listCommandHqTags, createCommandHqTag, deleteCommandHqTag, addCommandHqSessionTag, removeCommandHqSessionTag, + type CommandHqCostSummary, } from 'officerdb'; /** @@ -155,7 +160,14 @@ type CommanderSummary = ClaudeSessionSummary & { * no display meaning, the join key `armCommandHqWatcher` uses and nothing a card needs to show. */ watcherArmed: boolean; + /** Since when the CURRENT run has been going, or null if never armed. Untouched by every self-re-arm. */ + watcherArmedAt: Date | null; + /** When the current run ended, or null while still armed (or never armed). Paired with `watcherArmedAt`. */ + watcherStoppedAt: Date | null; + /** `MAX(firedAt)` for this commander — no column of its own, see `docs/command-hq.md` → "The watcher loop". */ watcherLastFiredAt: Date | null; + /** The running total — every turn this commander has ever cost, summed. See `GET .../turns` for the raw series. */ + cost: CommandHqCostSummary; }; async function isRunning(sessionId: string): Promise { @@ -187,10 +199,19 @@ commandHqRouter.get('/commanders', async (ctx) => { const liveById = new Map(live.map((s) => [s.id, s])); const filed = rows.filter((row) => liveById.has(row.sessionId)); - const tagsByRow = await listCommandHqSessionTags(filed.map((row) => row.id)); - const runningById = new Map( - await Promise.all(filed.map(async (row) => [row.sessionId, await isRunning(row.sessionId)] as const)), - ); + const [tagsByRow, lastFiredByRow, runningById, costByRow] = await Promise.all([ + listCommandHqSessionTags(filed.map((row) => row.id)), + getLastFiredAtBySessionRowIds(filed.map((row) => row.id)), + Promise.all(filed.map(async (row) => [row.sessionId, await isRunning(row.sessionId)] as const)).then( + (pairs) => new Map(pairs), + ), + // One aggregate query per commander, the same cost class `isRunning` above already accepts for this + // grid — not batched, because a running SUM/COUNT is not a lookup on a small shared table the way tags + // and last-fired are. + Promise.all(filed.map(async (row) => [row.id, await getCommandHqCostSummary(row.id)] as const)).then( + (pairs) => new Map(pairs), + ), + ]); const commanders: CommanderSummary[] = filed .map((row) => ({ @@ -199,7 +220,10 @@ commandHqRouter.get('/commanders', async (ctx) => { tagIds: tagsByRow.get(row.id) ?? [], running: runningById.get(row.sessionId) ?? false, watcherArmed: row.watcherArmed, - watcherLastFiredAt: row.watcherLastFiredAt, + watcherArmedAt: row.watcherArmedAt, + watcherStoppedAt: row.watcherStoppedAt, + watcherLastFiredAt: lastFiredByRow.get(row.id) ?? null, + cost: costByRow.get(row.id) ?? { turnCount: 0, inputTokens: 0, outputTokens: 0, totalUsd: 0 }, })) .sort((a, b) => b.updatedAt.localeCompare(a.updatedAt)); @@ -304,18 +328,57 @@ commandHqRouter.get('/commanders/:sessionId', async (ctx) => { if (!row) throw errors.NOT_FOUND('not a commander'); if (!loadClaudeSessionById(who, sessionId)) throw errors.NOT_FOUND('not a commander'); - const [tagsByRow, running] = await Promise.all([listCommandHqSessionTags([row.id]), isRunning(sessionId)]); + const [tagsByRow, running, lastFiredByRow, cost] = await Promise.all([ + listCommandHqSessionTags([row.id]), + isRunning(sessionId), + getLastFiredAtBySessionRowIds([row.id]), + getCommandHqCostSummary(row.id), + ]); return ctx.json({ sessionId: row.sessionId, categoryId: row.categoryId, tagIds: tagsByRow.get(row.id) ?? [], running, watcherArmed: row.watcherArmed, - watcherLastFiredAt: row.watcherLastFiredAt, + watcherArmedAt: row.watcherArmedAt, + watcherStoppedAt: row.watcherStoppedAt, + watcherLastFiredAt: lastFiredByRow.get(row.id) ?? null, + cost, createdAt: row.createdAt, }); }); +/** + * GET /command-hq/commanders/:sessionId/turns + * + * The raw per-turn cost ledger, oldest first — what a client plots "cost over time" from, and what + * `cost` on the detail route above is the sum of. Not included inline on that route because it is + * unbounded (a commander running for months accumulates one row per turn), so a client that only wants + * the running total is not paying for the whole history on every open. + */ +commandHqRouter.get('/commanders/:sessionId/turns', async (ctx) => { + const user = ctx.get('user'); + const sessionId = ctx.req.param('sessionId'); + const row = await getCommandHqSession(user.id, sessionId); + if (!row) throw errors.NOT_FOUND('not a commander'); + return ctx.json({ turns: await listCommandHqTurns(row.id) }); +}); + +/** + * GET /command-hq/commanders/:sessionId/fires + * + * Every trigger this commander's watcher has ever fired, oldest first — the boundaries a client uses to + * group `.../turns` by trigger, and the full history behind the single `watcherLastFiredAt` the other + * routes report. + */ +commandHqRouter.get('/commanders/:sessionId/fires', async (ctx) => { + const user = ctx.get('user'); + const sessionId = ctx.req.param('sessionId'); + const row = await getCommandHqSession(user.id, sessionId); + if (!row) throw errors.NOT_FOUND('not a commander'); + return ctx.json({ fires: await listCommandHqWatcherFires(row.id) }); +}); + /** * POST /command-hq/commanders { sessionId: string, categoryId?: number } *