Files

T

pastilhas aa5bacec46 email recall: the design, and which answers can be trusted

the use case is someone mid-phone-call who needs a fact from eight years of
correspondence now. everything in here is in service of that moment.

the split that came out of the discussion, and it is the whole document:
counting needs EXHAUSTION, retrieval needs SPEED, and they are different
machinery. an agent asked "how many times did she ask X" will search, find
some, and answer with a number bounded by the phrasings it happened to think
of — "i found four" is indistinguishable from "there were nine", and nothing
in the output says which. vector search does not fix that; top-k misses too,
just differently. the only honest count is to judge every message once and
store the verdict.

which makes the compile half the one that answers the questions that prompted
this, and it needs no vectors at all.

embeddings still earn their place for the other row: keyword search finds
"school run" and never finds it from "the pickup arrangement", and on a live
call thirty seconds of searching is a lost moment. the stronger argument is
whatsapp and transcripts sharing no vocabulary with email — but we do not have
those, so they shape exactly one decision (the store is not email-shaped) and
nothing else.

two constraints stated as constraints rather than preferences: the embedding
model runs LOCALLY, because an api means uploading eight years of private
correspondence to a third party, and there is no ollama or embedding code on
any machine here — that is the real new dependency, not the database. and it
is sqlite-vec in the plugin's own file, not a vector database: 19,097 rows is
small, and a separate service would be operational surface bought for nothing.

the hard part is neither. it is quoted reply chains putting the same paragraph
in forty messages of one thread, which naive embedding turns into forty
identical results. step 1 of the build order is chunking with no embeddings at
all, because if that is wrong nothing after it can be right.

a stored verdict is a judgement a model made about someone's divorce, so it
records what it was asked, what it read and when. a number with no provenance
is worse than none, because it will be quoted.

2026-08-20 02:33:31 +00:00

7.5 KiB

Raw Blame History

Recall over your own correspondence

Status: DESIGN, NOT BUILT. Sketched 2026-08-20, for the email plugin.

The goal, which is not the technology

Someone is on a phone call with their ex-wife. She says something about the weekend arrangement. He has eight years of email with her, and he needs the fact now — not after a search, not after reading twenty messages.

That is the whole requirement. Everything below is in service of it, and any part of this that does not make that moment faster or more accurate is decoration.

The user will never write a query. They ask the agent a question in their own words, and the agent picks its tools. So the design question is not "what can the user search" — it is "what does the agent have to work with, and which of its answers can be trusted".

Two mechanisms, because there are two kinds of question

The examples that prompted this split cleanly, and they need different machinery:

the question	what it needs
"how many times did she ask me to break the weekend agreement"	exhaustion — every message judged
"what did she say about the school run"	retrieval — the best few, fast

COMPILE    exhaustive, once, expensive   → counting, timelines, "show me all of them"
RETRIEVE   cheap, approximate, always    → exploring, mid-call, "what did she say about…"

They are not competing designs. The agent should have both and choose.

Counting cannot be done by searching

This is the load-bearing point, and it is the one most likely to be skipped.

An agent asked "how many times" will search, find some, and answer with a number. That number is bounded by the phrasings it happened to think of. "I found four" is indistinguishable from "there were nine", and nothing in the output signals which it was.

Vector search does not fix this. Top-k also misses — it just misses differently. Ask for the top 20 and the 21st is silently gone.

The only honest way to count is to judge every message once and store the verdict. Then counting is SELECT count(*): exact, instant, and re-askable a hundred ways without re-reading anything.

A stored verdict is a judgement a model made about someone's divorce. It records what it was asked, which message it read, and when — so the answer can be inspected, disbelieved and redone. A number with no provenance is worse than no number, because it will be quoted.

Retrieval is where embeddings earn their place

Keyword search finds school run. It will never find it from "the pickup arrangement" or "when I collect her on Fridays". Over eight years, the same thing is described twenty ways and never once in the words you would search for.

An agent closes some of that gap by itself — it knows those are synonyms and can fire several searches. But that is guess-bounded: it finds the phrasings it thought of, and cannot know the ones it did not.

Two things decide it in favour of embeddings anyway:

Latency. Five keyword searches and twenty candidate reads is thirty seconds. On a live call that is a lost moment. A vector query is milliseconds plus one read.
What comes later. WhatsApp and audio transcripts share no vocabulary, schema or structure with email. Embeddings collapse all of it into one space. ASR output is noisy in a way keyword search is brittle against and embeddings are not.

But we do not have WhatsApp or transcripts. They shape exactly one decision — see the schema — and nothing else. Build against email.

It has to run locally

Every message passes through an embedding model. Doing that through an API means uploading eight years of private correspondence to a third party.

For this content that is not a trade-off, it is a disqualification. Local embedding is a requirement, not a preference.

Today there is no ollama and no embedding code on any of these machines. That is the real new dependency — not the database.

Not a vector database

sqlite-vec is an extension to the SQLite already in use. Same file as emails.db, no service, no sidecar, no Docker, no port, nothing to supervise or back up separately.

For a plugin that already owns that file, this is a table, not an architecture. A separate vector database would add an operational surface for a corpus of 19,000 rows, which is small — vector stores begin earning their keep in the millions. What is being bought here is recall on paraphrase and speed on a phone call, not scale.

The genuinely hard part is chunking

Not the vectors. Quoted reply chains.

The same paragraph exists in forty messages of one thread. Embed naively and every query returns that thread forty times, and it will feel worse than what exists today.

So: strip quoted history, thread properly, embed the new text of each message. emails already has thread_id. This is the work.

Two other email-specific traps measured on the real 19,097-message store:

Bodies are ~0.5 GB of HTML. Embed the text, not the markup.
labels is a comma-separated string, and read is populated on 93 rows. Neither is a filter to build on. See the email-db skill.

Schema sketch

Deliberately not email-shaped. That is the one concession to a future that does not exist yet, and it costs nothing now:

recall_chunks
  id            integer pk
  source        text      -- 'email' today. 'whatsapp', 'transcript' later.
  source_id     text      -- the message id in ITS store — no foreign key across sources
  account       text      -- which mailbox/number this came from
  thread_id     text      -- for dedupe and for showing context
  author        text      -- who said it
  said_at       timestamp
  text          text      -- quoted history stripped
  embedding     float[]   -- sqlite-vec

recall_verdicts             -- the COMPILE half
  id            integer pk
  question      text      -- what the model was asked, verbatim
  source        text
  source_id     text
  verdict       boolean
  reasoning     text
  model         text      -- which model, so a re-run is comparable
  judged_at     timestamp

Everything else assumes email until something else exists.

Build order

Chunking and dedupe, with no embeddings at all. Strip quotes, thread, count what is left. If this is wrong nothing after it can be right, and it is testable on its own.
A local embedding model, and a decision about which. The new dependency.
sqlite-vec + backfill of the existing corpus. One-off, slow, resumable.
A retrieval tool the agent can call, returning chunks with their message ids so it can go read the original.
The compile half — a verdict table and a way to ask "judge every message against this question".
Multi-source, if and when WhatsApp or transcripts exist.

Steps 1 and 5 are useful without any of the others. Step 5 in particular answers the questions that prompted this, and needs no vectors at all.

Open questions

Which local model, and how big. Quality against a laptop-or-server-sized footprint.
Re-embedding. Changing model means re-embedding everything. Store which model produced each vector.
Retention and deletion. A deleted email should lose its chunks and its verdicts. Nothing here currently cascades.
Who can query it. This is the most sensitive data in the platform, and it is per-account.

7.5 KiB Raw Blame History