From aa5bacec4645fb4050135a3c47d9f32ecde2185d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Thu, 20 Aug 2026 02:33:31 +0000 Subject: [PATCH] email recall: the design, and which answers can be trusted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit the use case is someone mid-phone-call who needs a fact from eight years of correspondence now. everything in here is in service of that moment. the split that came out of the discussion, and it is the whole document: counting needs EXHAUSTION, retrieval needs SPEED, and they are different machinery. an agent asked "how many times did she ask X" will search, find some, and answer with a number bounded by the phrasings it happened to think of — "i found four" is indistinguishable from "there were nine", and nothing in the output says which. vector search does not fix that; top-k misses too, just differently. the only honest count is to judge every message once and store the verdict. which makes the compile half the one that answers the questions that prompted this, and it needs no vectors at all. embeddings still earn their place for the other row: keyword search finds "school run" and never finds it from "the pickup arrangement", and on a live call thirty seconds of searching is a lost moment. the stronger argument is whatsapp and transcripts sharing no vocabulary with email — but we do not have those, so they shape exactly one decision (the store is not email-shaped) and nothing else. two constraints stated as constraints rather than preferences: the embedding model runs LOCALLY, because an api means uploading eight years of private correspondence to a third party, and there is no ollama or embedding code on any machine here — that is the real new dependency, not the database. and it is sqlite-vec in the plugin's own file, not a vector database: 19,097 rows is small, and a separate service would be operational surface bought for nothing. the hard part is neither. it is quoted reply chains putting the same paragraph in forty messages of one thread, which naive embedding turns into forty identical results. step 1 of the build order is chunking with no embeddings at all, because if that is wrong nothing after it can be right. a stored verdict is a judgement a model made about someone's divorce, so it records what it was asked, what it read and when. a number with no provenance is worse than none, because it will be quoted. --- CLAUDE.md | 3 + docs/email-recall.md | 159 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+) create mode 100644 docs/email-recall.md diff --git a/CLAUDE.md b/CLAUDE.md index 8c6db9d9..cfbbdf11 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -421,6 +421,9 @@ explaining why it was safe. - `docs/navigation-audit.md` — **authoritative** on routing/navigation: the opaque-click anti-pattern, a severity-ranked findings table, the channel-selection map and the four-phase plan +- `docs/email-recall.md` — **design, not built**: recall over your own correspondence. The split that + matters is counting (needs every message judged) versus retrieval (needs the best few, fast) — and why + a number produced by searching is confidently wrong with no signal that it is - `docs/find-my-device.md` — **design, not built**: locating your own devices from Officer. What is assembly (most of it), what is impossible (offline finding — there is no crowd to relay it), and the question to settle first, which is retention rather than anything technical diff --git a/docs/email-recall.md b/docs/email-recall.md new file mode 100644 index 00000000..cba7210d --- /dev/null +++ b/docs/email-recall.md @@ -0,0 +1,159 @@ +# Recall over your own correspondence + +**Status: DESIGN, NOT BUILT.** Sketched 2026-08-20, for the email plugin. + +## The goal, which is not the technology + +Someone is on a phone call with their ex-wife. She says something about the weekend arrangement. He has +eight years of email with her, and he needs the fact **now** — not after a search, not after reading +twenty messages. + +That is the whole requirement. Everything below is in service of it, and any part of this that does not +make that moment faster or more accurate is decoration. + +The user will never write a query. They ask the agent a question in their own words, and the agent picks +its tools. So the design question is not "what can the user search" — it is **"what does the agent have +to work with, and which of its answers can be trusted"**. + +## Two mechanisms, because there are two kinds of question + +The examples that prompted this split cleanly, and they need different machinery: + +| the question | what it needs | +| ---------------------------------------------------------------- | ------------------------------------- | +| _"how many times did she ask me to break the weekend agreement"_ | **exhaustion** — every message judged | +| _"what did she say about the school run"_ | **retrieval** — the best few, fast | + +``` +COMPILE exhaustive, once, expensive → counting, timelines, "show me all of them" +RETRIEVE cheap, approximate, always → exploring, mid-call, "what did she say about…" +``` + +They are not competing designs. The agent should have both and choose. + +### Counting cannot be done by searching + +This is the load-bearing point, and it is the one most likely to be skipped. + +An agent asked "how many times" will search, find some, and answer with a number. That number is bounded +by the phrasings it happened to think of. **"I found four" is indistinguishable from "there were nine"**, +and nothing in the output signals which it was. + +Vector search does not fix this. Top-k also misses — it just misses differently. Ask for the top 20 and +the 21st is silently gone. + +The only honest way to count is to judge every message once and store the verdict. Then counting is +`SELECT count(*)`: exact, instant, and re-askable a hundred ways without re-reading anything. + +**A stored verdict is a judgement a model made about someone's divorce.** It records what it was asked, +which message it read, and when — so the answer can be inspected, disbelieved and redone. A number with +no provenance is worse than no number, because it will be quoted. + +### Retrieval is where embeddings earn their place + +Keyword search finds `school run`. It will never find it from _"the pickup arrangement"_ or _"when I +collect her on Fridays"_. Over eight years, the same thing is described twenty ways and never once in the +words you would search for. + +An agent closes some of that gap by itself — it knows those are synonyms and can fire several searches. +But that is **guess-bounded**: it finds the phrasings it thought of, and cannot know the ones it did not. + +Two things decide it in favour of embeddings anyway: + +- **Latency.** Five keyword searches and twenty candidate reads is thirty seconds. On a live call that is + a lost moment. A vector query is milliseconds plus one read. +- **What comes later.** WhatsApp and audio transcripts share no vocabulary, schema or structure with + email. Embeddings collapse all of it into one space. ASR output is noisy in a way keyword search is + brittle against and embeddings are not. + +**But we do not have WhatsApp or transcripts.** They shape exactly one decision — see the schema — and +nothing else. Build against email. + +## It has to run locally + +Every message passes through an embedding model. Doing that through an API means uploading eight years of +private correspondence to a third party. + +For this content that is not a trade-off, it is a disqualification. **Local embedding is a requirement, +not a preference.** + +Today there is no ollama and no embedding code on any of these machines. **That is the real new +dependency — not the database.** + +## Not a vector database + +`sqlite-vec` is an extension to the SQLite already in use. Same file as `emails.db`, no service, no +sidecar, no Docker, no port, nothing to supervise or back up separately. + +For a plugin that already owns that file, this is **a table, not an architecture**. A separate vector +database would add an operational surface for a corpus of 19,000 rows, which is small — vector stores +begin earning their keep in the millions. What is being bought here is recall on paraphrase and speed on +a phone call, not scale. + +## The genuinely hard part is chunking + +Not the vectors. **Quoted reply chains.** + +The same paragraph exists in forty messages of one thread. Embed naively and every query returns that +thread forty times, and it will feel worse than what exists today. + +So: strip quoted history, thread properly, embed the new text of each message. `emails` already has +`thread_id`. This is the work. + +Two other email-specific traps measured on the real 19,097-message store: + +- **Bodies are ~0.5 GB of HTML.** Embed the text, not the markup. +- **`labels` is a comma-separated string**, and `read` is populated on 93 rows. Neither is a filter to + build on. See the `email-db` skill. + +## Schema sketch + +Deliberately **not** email-shaped. That is the one concession to a future that does not exist yet, and it +costs nothing now: + +``` +recall_chunks + id integer pk + source text -- 'email' today. 'whatsapp', 'transcript' later. + source_id text -- the message id in ITS store — no foreign key across sources + account text -- which mailbox/number this came from + thread_id text -- for dedupe and for showing context + author text -- who said it + said_at timestamp + text text -- quoted history stripped + embedding float[] -- sqlite-vec + +recall_verdicts -- the COMPILE half + id integer pk + question text -- what the model was asked, verbatim + source text + source_id text + verdict boolean + reasoning text + model text -- which model, so a re-run is comparable + judged_at timestamp +``` + +Everything else assumes email until something else exists. + +## Build order + +1. **Chunking and dedupe**, with no embeddings at all. Strip quotes, thread, count what is left. If this + is wrong nothing after it can be right, and it is testable on its own. +2. **A local embedding model**, and a decision about which. The new dependency. +3. **`sqlite-vec` + backfill** of the existing corpus. One-off, slow, resumable. +4. **A retrieval tool** the agent can call, returning chunks with their message ids so it can go read the + original. +5. **The compile half** — a verdict table and a way to ask "judge every message against this question". +6. Multi-source, if and when WhatsApp or transcripts exist. + +Steps 1 and 5 are useful without any of the others. Step 5 in particular answers the questions that +prompted this, and needs no vectors at all. + +## Open questions + +- **Which local model**, and how big. Quality against a laptop-or-server-sized footprint. +- **Re-embedding.** Changing model means re-embedding everything. Store which model produced each vector. +- **Retention and deletion.** A deleted email should lose its chunks and its verdicts. Nothing here + currently cascades. +- **Who can query it.** This is the most sensitive data in the platform, and it is per-account.