persist the wallet's view of the chain

Opening a wallet meant waiting for a full gap-limit scan before any number appeared, and a
restart threw that work away. Worse, an unreachable Esplora rendered identically to an empty
wallet — as a zero balance — which is alarming for the one case where it is not true.

The backend now keeps a snapshot and serves it stale-while-revalidate: a snapshot inside the
TTL is served as-is, an older one is served immediately with a refresh started behind it, and
only a wallet that has genuinely never been read blocks on the network. wallet_chain_cache
holds one row per wallet so a refresh is a single atomic upsert.

The snapshot, not the endpoint, is the unit of caching. Balances, UTXOs and history were
three fetches over a shared scan, so the three queries a wallet screen fires on mount could
each observe a different moment; building them together costs the same requests and fixes
that incidentally.

Only the chain's own facts are stored. Addresses, scripts and pubkeys are re-derived from the
account xpub on load — cheaper than persisting them, and it means a restored snapshot cannot
disagree with the wallet's actual keys. Stored coordinates are validated rather than trusted,
and a snapshot at an unknown version is discarded, not migrated.

Two reads deliberately opt out. sendCoins takes a fresh snapshot because selecting coins from
a cached UTXO set builds a transaction spending outputs that may already be gone, and that
failure arrives as a broadcast rejection after signing. nextUnused does too, because handing
out an address whose stale record says "unused" is silent address reuse — a privacy leak the
owner cannot see or undo. Receive-address generation is therefore the one read that stops
working while the upstream is down, on purpose.

Failures are recorded alongside the last good snapshot rather than replacing it; wiping data
on failure would reproduce the exact bug this exists to fix. Every cache operation is
best-effort, so a database problem degrades to a slow load and can never fail a wallet
request. A sync block on balances, transactions and utxos carries the age to the UI, which
now distinguishes "empty" from "never read".

Verified against three live mainnet wallets: snapshots persisted and reloaded, and two
wallets kept their data and age through a real Esplora rate-limit failure while recording the
error separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/servers/sidecar/wallet/backends/onchain.ts

@@ -16,6 +16,7 @@
 // There is no lightning here at all: the capability set omits every lightning flag, so invoices,
 // payments, channels and peers all fall through to BaseBackend's 501.
 
+import type { WalletChainSnapshot } from 'officerdb';
 import { HDKey } from '@scure/bip32';
 import * as bitcoin from 'bitcoinjs-lib';
 import { toXOnly } from 'bitcoinjs-lib/src/psbt/bip371';
@@ -49,6 +50,7 @@ import {
   type SendCoinsRequest,
   type SendCoinsResult,
   type SignMessageResult,
+  type SyncState,
   type Utxo,
   type VerifyMessageResult,
 } from '../types';
@@ -71,6 +73,22 @@ export interface WalletSigner {
   withRoot<T>(fn: (root: HDKey) => T): T;
 }
 
+// ── the persistence boundary ─────────────────────────────────────────────────────────────────────
+
+/**
+ * Durable storage for the last snapshot this backend read off the chain. Injected the same way the
+ * signer is, and for the same reason: this file stays a pure wallet with no idea which database it is
+ * attached to, or that it has a wallet id at all. resolve.ts binds the real implementation.
+ *
+ * Every method is best-effort by contract. A cache that is unreachable, corrupt or empty must degrade
+ * to today's behaviour — a slow first load — and must never fail a wallet request.
+ */
+export interface ChainCacheStore {
+  load(): Promise<{ snapshot: WalletChainSnapshot | null; syncedAt: Date | null; lastError: string | null } | null>;
+  save(snapshot: WalletChainSnapshot): Promise<void>;
+  recordError(message: string): Promise<void>;
+}
+
 // ── extended key parsing ─────────────────────────────────────────────────────────────────────────
 
 /**
@@ -143,12 +161,24 @@ type ScannedAddress = AddressEntry & {
   used: boolean;
 };
 
-type WalletScan = {
+/**
+ * One coherent read of the chain: the address scan and everything derived from it, taken together.
+ *
+ * Balances, UTXOs and history used to be fetched by three separate endpoints on top of a shared scan,
+ * which meant the three queries a wallet screen fires on mount could each observe a different moment.
+ * Building them as one snapshot costs the same requests — collectUtxos only asks about funded addresses
+ * and collectTxs only about touched ones — and makes the whole thing a single unit to cache, serve stale
+ * and persist.
+ */
+type Snapshot = {
+  /** Unix ms the chain was read. Survives a restart: a restored snapshot carries its original time. */
   at: number;
   tipHeight: number;
   addresses: ScannedAddress[];
-  /** scriptPubKey hex → the entry that owns it. Used to score transactions as ours. */
+  /** scriptPubKey hex → the entry that owns it. Used to score transactions as ours. Never persisted. */
   byScript: Map<string, ScannedAddress>;
+  utxos: SpendableUtxo[];
+  txs: OnchainTx[];
 };
 
 // ── tuning ───────────────────────────────────────────────────────────────────────────────────────
@@ -156,10 +186,12 @@ type WalletScan = {
 /** BIP44's standard gap limit: 20 consecutive unused addresses ends the scan for a chain. */
 const GAP_LIMIT = 20;
 
-/** How long a discovery scan stays fresh. Short enough to feel live, long enough that a dashboard
- *  polling getInfo/getBalances/getTransactions together costs one scan rather than three. */
+/** How long a snapshot is served without kicking off a refresh behind it. */
 const SCAN_TTL_MS = 30_000;
 
+/** Cap on the persisted history. A wallet with years of activity should not grow an unbounded jsonb blob. */
+const MAX_CACHED_TXS = 500;
+
 /** Parallel Esplora requests. Public instances rate-limit, so this stays modest. */
 const REQUEST_CONCURRENCY = 6;
 
@@ -179,6 +211,8 @@ export type OnchainBackendOptions = {
    */
   accountXpub: string | Partial<Record<AddressType, string>>;
   signer: WalletSigner;
+  /** Omit for a backend that should hold its chain view in memory only — tests do. */
+  cache?: ChainCacheStore;
 };
 
 export class OnchainBackend extends BaseBackend {
@@ -208,8 +242,13 @@ export class OnchainBackend extends BaseBackend {
    *  which is correct as soon as an issued address has actually been paid. */
   private readonly issued = new Map<string, number>();
 
-  private scanCache: WalletScan | null = null;
-  private scanInflight: Promise<WalletScan> | null = null;
+  private readonly cache: ChainCacheStore | null;
+
+  private current: Snapshot | null = null;
+  private inflight: Promise<Snapshot> | null = null;
+  /** Memoised so concurrent first reads do one cache load between them, not one each. */
+  private hydration: Promise<void> | null = null;
+  private lastError: string | null = null;
 
   constructor(opts: OnchainBackendOptions) {
     super();
@@ -218,6 +257,7 @@ export class OnchainBackend extends BaseBackend {
     this.network = opts.network;
     this.btcNetwork = networkFor(opts.network);
     this.signer = opts.signer;
+    this.cache = opts.cache ?? null;
 
     const coin = coinTypeFor(opts.network);
     const raw = typeof opts.accountXpub === 'string' ? { p2wpkh: opts.accountXpub } : opts.accountXpub;
@@ -286,39 +326,169 @@ export class OnchainBackend extends BaseBackend {
     return entry;
   }
 
-  // ── discovery scan ─────────────────────────────────────────────────────────────────────────────
+  // ── the snapshot: cache, refresh, persist ──────────────────────────────────────────────────────
+
+  getSyncState(): SyncState {
+    return {
+      syncedAt: this.current?.at ?? null,
+      stale: this.current !== null && Date.now() - this.current.at >= SCAN_TTL_MS,
+      lastError: this.lastError,
+    };
+  }
 
   /**
-   * Gap-limit scan of every configured account across both chains, memoised for SCAN_TTL_MS. Concurrent
-   * callers share one in-flight scan rather than each starting their own — without that, the three
-   * queries a wallet screen fires on mount would triple the request count against Esplora.
+   * The wallet's view of the chain, stale-while-revalidate.
+   *
+   * A snapshot within SCAN_TTL_MS is served as-is. An older one is served IMMEDIATELY with a refresh
+   * started behind it — that is the property the persistence exists for: after a restart, or while the
+   * Esplora upstream is slow or unreachable, the screen shows the last real numbers with their age
+   * rather than a blank wallet. Only a wallet that has genuinely never been read blocks on the network,
+   * and only that case can surface an error to the caller.
+   *
+   * `fresh` opts out for the two paths where stale data would be wrong rather than merely old.
    */
-  private async scan(): Promise<WalletScan> {
-    const fresh = this.scanCache;
-    if (fresh && Date.now() - fresh.at < SCAN_TTL_MS) return fresh;
-    if (this.scanInflight) return this.scanInflight;
+  private async snapshot(opts?: { fresh?: boolean }): Promise<Snapshot> {
+    if (opts?.fresh) return this.refresh();
 
-    const run = this.runScan().then(
-      (result) => {
-        this.scanCache = result;
-        this.scanInflight = null;
-        return result;
+    await this.hydrate();
+
+    const have = this.current;
+    if (have && Date.now() - have.at < SCAN_TTL_MS) return have;
+
+    const running = this.refresh();
+    if (!have) return running;
+
+    // The refresh outlives this response. Its failure is recorded on `lastError` and reported through
+    // getSyncState, so it is handled here only to keep it from surfacing as an unhandled rejection.
+    void running.catch(() => {});
+    return have;
+  }
+
+  /** Load the persisted snapshot into memory. Runs at most once, and never throws. */
+  private hydrate(): Promise<void> {
+    this.hydration ??= (async () => {
+      const store = this.cache;
+      if (!store || this.current) return;
+      try {
+        const row = await store.load();
+        if (!row) return;
+        this.lastError = row.lastError;
+        // A refresh may have completed while this load was in flight; live data always wins. A snapshot
+        // written by an older shape is ignored, not migrated — it is a cache, and the next refresh
+        // replaces it for free.
+        if (row.snapshot?.v === 1 && row.syncedAt && !this.current) {
+          this.current = this.fromPersisted(row.snapshot, row.syncedAt.getTime());
+        }
+      } catch (err) {
+        // A cache failure costs a slow load, nothing more. It must not become a wallet failure.
+        console.error('[wallet] chain cache load failed:', err instanceof Error ? err.message : err);
+      }
+    })();
+    return this.hydration;
+  }
+
+  /** Read the chain. Concurrent callers share one walk rather than each starting their own. */
+  private refresh(): Promise<Snapshot> {
+    this.inflight ??= this.buildSnapshot().then(
+      (snap) => {
+        this.inflight = null;
+        this.current = snap;
+        this.lastError = null;
+        void this.persist(snap);
+        return snap;
       },
       (err: unknown) => {
-        this.scanInflight = null;
+        this.inflight = null;
+        const message = err instanceof Error ? err.message : String(err);
+        this.lastError = message;
+        void this.cache?.recordError(message).catch((e: unknown) => {
+          console.error('[wallet] chain cache error write failed:', e instanceof Error ? e.message : e);
+        });
         throw err;
       },
     );
-    this.scanInflight = run;
-    return run;
+    return this.inflight;
   }
 
-  /** Drop the cache after a send, so the spent coins disappear from the next read immediately. */
+  private async persist(snap: Snapshot): Promise<void> {
+    if (!this.cache) return;
+    try {
+      await this.cache.save(toPersisted(snap));
+    } catch (err) {
+      console.error('[wallet] chain cache write failed:', err instanceof Error ? err.message : err);
+    }
+  }
+
+  /**
+   * Rebuild a snapshot from what was stored. Only the chain's own facts were persisted; every address,
+   * script and pubkey is re-derived here from the account xpub, so a restored snapshot cannot disagree
+   * with the wallet's actual keys. An entry naming a script type this wallet no longer has an xpub for
+   * is dropped rather than faked.
+   */
+  private fromPersisted(stored: WalletChainSnapshot, at: number): Snapshot {
+    const addresses: ScannedAddress[] = [];
+    for (const a of stored.addresses) {
+      const entry = this.tryDerive(a.type, a.chain, a.index);
+      if (!entry) continue;
+      addresses.push({
+        ...entry,
+        confirmedSats: a.confirmed,
+        unconfirmedSats: a.unconfirmed,
+        txCount: a.txCount,
+        used: a.txCount > 0,
+      });
+    }
+
+    const utxos: SpendableUtxo[] = [];
+    for (const u of stored.utxos) {
+      const entry = this.tryDerive(u.type, u.chain, u.index);
+      if (!entry) continue;
+      utxos.push({
+        txid: u.txid,
+        vout: u.vout,
+        amountSats: u.value,
+        address: entry.address,
+        addressType: entry.type,
+        confirmations: u.blockHeight === null ? 0 : Math.max(0, stored.tipHeight - u.blockHeight + 1),
+        derivationPath: entry.path,
+        frozen: false,
+        scriptPubKeyHex: entry.scriptPubKeyHex,
+        pubkeyHex: entry.pubkeyHex,
+      });
+    }
+
+    const byScript = new Map<string, ScannedAddress>();
+    for (const entry of addresses) byScript.set(entry.scriptPubKeyHex, entry);
+
+    return { at, tipHeight: stored.tipHeight, addresses, byScript, utxos, txs: stored.transactions };
+  }
+
+  /** Derive from untrusted stored coordinates — returns null rather than throwing on anything unusable. */
+  private tryDerive(type: string, chain: number, index: number): AddressEntry | null {
+    if (!isAddressType(type) || (chain !== 0 && chain !== 1)) return null;
+    if (!Number.isInteger(index) || index < 0 || index >= MAX_SCAN_INDEX) return null;
+    if (!this.accounts.has(type)) return null;
+    try {
+      return this.derive(type, chain, index);
+    } catch {
+      return null;
+    }
+  }
+
+  /** One full read of the chain: scan, then everything derived from it, all at the same moment. */
+  private async buildSnapshot(): Promise<Snapshot> {
+    const scan = await this.runScan();
+    // Both fan out over the scanned address set and neither depends on the other, so they overlap.
+    const [utxos, txs] = await Promise.all([this.collectUtxos(scan), this.collectTxs(scan)]);
+    return { ...scan, utxos, txs };
+  }
+
+  /** Drop everything after a send, so the spent coins cannot reappear from cache on the next read. */
   private invalidateScan(): void {
-    this.scanCache = null;
+    this.current = null;
   }
 
-  private async runScan(): Promise<WalletScan> {
+  private async runScan(): Promise<Omit<Snapshot, 'utxos' | 'txs'>> {
     const tipHeight = await this.chain.getTipHeight();
     const addresses: ScannedAddress[] = [];
 
@@ -394,10 +564,10 @@ export class OnchainBackend extends BaseBackend {
   }
 
   override async getBalances(): Promise<Balances> {
-    const scan = await this.scan();
+    const snap = await this.snapshot();
     let confirmed = 0;
     let unconfirmed = 0;
-    for (const entry of scan.addresses) {
+    for (const entry of snap.addresses) {
       confirmed += entry.confirmedSats;
       unconfirmed += entry.unconfirmedSats;
     }
@@ -415,9 +585,8 @@ export class OnchainBackend extends BaseBackend {
   }
 
   override async getUtxos(): Promise<Utxo[]> {
-    const scan = await this.scan();
-    const utxos = await this.collectUtxos(scan);
-    return utxos.map((u) => ({
+    const snap = await this.snapshot();
+    return snap.utxos.map((u) => ({
       txid: u.txid,
       vout: u.vout,
       amountSats: u.amountSats,
@@ -440,10 +609,12 @@ export class OnchainBackend extends BaseBackend {
   }
 
   override async getTransactions(opts?: { limit?: number }): Promise<OnchainTx[]> {
-    const scan = await this.scan();
-    const limit = opts?.limit ?? DEFAULT_TX_LIMIT;
+    const snap = await this.snapshot();
+    return snap.txs.slice(0, opts?.limit ?? DEFAULT_TX_LIMIT);
+  }
 
-    // Only addresses that have ever been touched can appear in history.
+  /** History for every address the scan found had been touched. Only those can appear in it. */
+  private async collectTxs(scan: Omit<Snapshot, 'utxos' | 'txs'>): Promise<OnchainTx[]> {
     const touched = scan.addresses.filter((a) => a.used);
     const pages = await mapLimit(touched, REQUEST_CONCURRENCY, (a) => this.chain.getAddressTxs(a.address));
 
@@ -460,11 +631,11 @@ export class OnchainBackend extends BaseBackend {
       if (ah !== bh) return bh - ah;
       return (b.timestamp ?? 0) - (a.timestamp ?? 0);
     });
-    return txs.slice(0, limit);
+    return txs.slice(0, MAX_CACHED_TXS);
   }
 
   /** Score one Esplora transaction against the wallet's own scripts. */
-  private toOnchainTx(tx: EsploraTx, scan: WalletScan): OnchainTx {
+  private toOnchainTx(tx: EsploraTx, scan: Omit<Snapshot, 'utxos' | 'txs'>): OnchainTx {
     let credit = 0;
     let debit = 0;
     const ours: string[] = [];
@@ -518,11 +689,12 @@ export class OnchainBackend extends BaseBackend {
     const recipientScript = outputScriptFor(req.address, this.btcNetwork);
     const recipientType = scriptType(recipientScript);
 
-    const scan = await this.scan();
-    const spendable = await this.collectUtxos(scan);
+    // Never stale. Selecting coins from a cached UTXO set would build a transaction spending outputs
+    // that may already be gone, and the failure would arrive as a broadcast rejection after signing.
+    const snap = await this.snapshot({ fresh: true });
 
     const selection = selectCoins({
-      utxos: spendable,
+      utxos: snap.utxos,
       targetSats: req.amountSats ?? 0,
       sendAll: req.sendAll === true,
       satPerVbyte: req.satPerVbyte,
@@ -621,7 +793,7 @@ export class OnchainBackend extends BaseBackend {
   // ── shared internals ───────────────────────────────────────────────────────────────────────────
 
   /** Fetch UTXOs for every scanned address that still holds a balance. */
-  private async collectUtxos(scan: WalletScan): Promise<SpendableUtxo[]> {
+  private async collectUtxos(scan: Omit<Snapshot, 'utxos' | 'txs'>): Promise<SpendableUtxo[]> {
     // An address whose funded and spent counts match holds nothing; asking Esplora about it is a
     // wasted round trip, and on a wallet with long history that is most of the address set.
     const funded = scan.addresses.filter((a) => a.confirmedSats + a.unconfirmedSats > 0);
@@ -652,13 +824,17 @@ export class OnchainBackend extends BaseBackend {
    * different address even before the current one is paid.
    */
   private async nextUnused(type: AddressType, chain: ChainIndex, peek: boolean): Promise<AddressEntry> {
-    const scan = await this.scan();
+    // Deliberately never served from cache. Handing out an address whose stale record says "unused" but
+    // which has been paid since is silent address reuse — a privacy leak the owner cannot see and cannot
+    // undo. Failing loudly while the upstream is down is the better of the two, so this is the one read
+    // that stops working offline, and on purpose.
+    const snap = await this.snapshot({ fresh: true });
     const key = `${type}:${chain}`;
     const hint = this.issued.get(key) ?? 0;
 
     const used = new Set<number>();
     let highest = -1;
-    for (const entry of scan.addresses) {
+    for (const entry of snap.addresses) {
       if (entry.type !== type || entry.chain !== chain) continue;
       highest = Math.max(highest, entry.index);
       if (entry.used) used.add(entry.index);
@@ -675,6 +851,46 @@ export class OnchainBackend extends BaseBackend {
 
 // ── helpers ──────────────────────────────────────────────────────────────────────────────────────
 
+function isAddressType(value: string): value is AddressType {
+  return value === 'p2wpkh' || value === 'p2tr' || value === 'p2sh-p2wpkh' || value === 'p2pkh';
+}
+
+/**
+ * Reduce a snapshot to the chain facts worth storing. Addresses, scripts and pubkeys are dropped: they
+ * are pure derivation from the account xpub, so re-deriving them on load is both cheaper than storing
+ * them and the only way a restored snapshot is guaranteed to match the wallet's real keys.
+ */
+function toPersisted(snap: Snapshot): WalletChainSnapshot {
+  return {
+    v: 1,
+    tipHeight: snap.tipHeight,
+    addresses: snap.addresses.map((a) => ({
+      type: a.type,
+      chain: a.chain,
+      index: a.index,
+      confirmed: a.confirmedSats,
+      unconfirmed: a.unconfirmedSats,
+      txCount: a.txCount,
+    })),
+    utxos: snap.utxos.map((u) => {
+      // The last two path elements are the BIP44 chain and address index — the same tail getUtxos
+      // reports as the relative path.
+      const tail = u.derivationPath.split('/');
+      return {
+        txid: u.txid,
+        vout: u.vout,
+        value: u.amountSats,
+        type: u.addressType,
+        chain: Number(tail.at(-2)),
+        index: Number(tail.at(-1)),
+        // Stored as a height, not a count, so confirmations stay correct as the tip moves on.
+        blockHeight: u.confirmations > 0 ? snap.tipHeight - u.confirmations + 1 : null,
+      };
+    }),
+    transactions: snap.txs,
+  };
+}
+
 function toScannedAddress(entry: AddressEntry, stat: EsploraAddress): ScannedAddress {
   const chainBalance = stat.chain_stats.funded_txo_sum - stat.chain_stats.spent_txo_sum;
   // The mempool delta is signed: an unconfirmed spend of a confirmed coin reads negative here, which is