stop losing the answer when immich rejects mid-upload

An upload over a few MB came back as a 400 with an empty body, no message anywhere, and nothing logged
by Immich, the sidecar or officer. It was a race, not a size limit. Immich judges an asset from its
first few KB and rejects immediately, then closes; both our hops were still writing the body; Node
treats the leftover bytes as a protocol violation and replaces the application's answer with a bodyless
`400 Bad Request` + `Connection: close`. The real message never reached the wire.

Measured before the change: streamed lost the message 1/4 at 8 MB and 4/4 at 32 MB — probability rising
with size, which is why small photos usually worked and a phone's video never did.

Both hops needed it. Fixing only the sidecar took 32 MB from 4/4 failing to 2/4, because the platform
proxy was losing it one hop up.

Bounded at 512 MB, above which the body streams exactly as before. That ceiling is not a refusal and is
deliberately not a 413: a file Immich ACCEPTS is read to the end and never races, so a 4 GB video is
unaffected. All that is given up above the cap is the error message on a file that was going to be
rejected anyway. A first attempt refused over-cap uploads outright and would have broken the working
4 GB case to improve diagnosis of the doomed one.

`bufferRequestBody` is opt-in and off by default: the vault and wallet proxies must keep streaming so a
passphrase or macaroon never lands in the platform's heap.

Also adds the proxy error logging that made this findable at all — status and two byte counts from
headers, never the bodies. `responseBytes: "unknown"` is what exposed the stripped response.

Verified live at 32/256 MB (buffered) and 640 MB (streamed, passes through).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/servers/sidecar/photos/routes.ts

@@ -83,11 +83,86 @@ export function relayResponse(res: Response): Response {
   return new Response(body, { status: res.status, headers });
 }
 
+/**
+ * The largest request body held in memory. Above this the body is STREAMED instead — never refused.
+ *
+ * The cap exists to bound heap, not to bound uploads: a 4 GB video must still go through. What is given
+ * up above the cap is only the guarantee that a REJECTION carries its message, and an upload that big
+ * which Immich rejects was going to fail either way — see `readBodyCapped`.
+ */
+const MAX_BUFFERED_BODY = 512 * 1024 * 1024;
+
+/**
+ * Read the whole request body into memory, refusing above `MAX_BUFFERED_BODY`.
+ *
+ * ── Why this is buffered rather than streamed ──
+ *
+ * It used to stream, which is the obvious thing for an upload and was wrong for one specific reason:
+ * when the body is still being written and Immich answers early, the response is lost. Immich rejects
+ * an asset as soon as it has enough of the file to judge it, then closes; our writer is still going;
+ * Node's HTTP server calls the leftover bytes a protocol violation and replaces the app's answer with a
+ * bodyless `400 Bad Request` + `Connection: close`. The app's real message never reaches the wire and
+ * Immich logs nothing, so the client sees an unexplained 400 and the server has no record of it.
+ *
+ * Measured on this machine (COMMS/UPLOAD_EMPTY_400*.md): streamed lost the response in 2 of 4 attempts
+ * at 8 MB, 32 MB and 64 MB alike; buffered lost it in 0 of 4 at every size. It is a race, not a size
+ * limit — its probability just approaches certainty as the body grows, which is why a phone's video
+ * failed every time while a small photo usually got through.
+ *
+ * A size threshold was considered and rejected: buffering only the small bodies would leave exactly the
+ * large uploads that always fail on the broken path. Cancelling the write once the response arrives was
+ * tried and does not help — by the time `fetch` resolves, the teardown has already happened.
+ *
+ * The cost is real and was accepted deliberately: a 300 MB video is 300 MB of this process's heap for
+ * the duration of the upload. That is the price of the upload working at all.
+ *
+ * ── And why there is a ceiling on it ──
+ *
+ * Buffering everything would mean a 4 GB video sitting in this heap (and another copy in officer's), so
+ * above MAX_BUFFERED_BODY the body is streamed as it always was. That is not a refusal and not a
+ * degradation of uploads: a file Immich ACCEPTS never triggers the race, because it is read to the end
+ * before anything is answered. The only thing lost above the cap is the error message on a file Immich
+ * rejects — and an upload that size which Immich rejects fails either way. Refusing it outright would
+ * break the legitimate case to improve the diagnosis of the doomed one.
+ */
+async function readBodyCapped(
+  req: Request,
+): Promise<{ ok: true; bytes: Uint8Array<ArrayBuffer> } | { ok: false; seen: number }> {
+  // Decided from `content-length` alone. Officer sends one whenever IT buffered, so the two hops agree
+  // without coordinating: a body small enough for officer to hold is small enough to hold here. When the
+  // header is absent officer streamed, which means it was over the cap there too.
+  const declared = Number(req.headers.get('content-length') ?? '');
+  if (!Number.isFinite(declared) || declared <= 0 || declared > MAX_BUFFERED_BODY) return { ok: false, seen: declared };
+
+  const reader = req.body!.getReader();
+  const chunks: Uint8Array[] = [];
+  let total = 0;
+  for (;;) {
+    const { done, value } = await reader.read();
+    if (done) break;
+    total += value.length;
+    if (total > MAX_BUFFERED_BODY) {
+      await reader.cancel().catch(() => {});
+      return { ok: false, seen: total };
+    }
+    chunks.push(value);
+  }
+
+  const bytes = new Uint8Array(new ArrayBuffer(total));
+  let at = 0;
+  for (const chunk of chunks) {
+    bytes.set(chunk, at);
+    at += chunk.length;
+  }
+  return { ok: true, bytes };
+}
+
 /**
  * Forward one request to Immich and stream the answer back.
  *
- * The body is a stream, never an ArrayBuffer: a full-resolution original or a `download/archive` zip is
- * hundreds of megabytes and buffering it would hold all of it in the sidecar's heap for no reason.
+ * The RESPONSE is still a stream, never an ArrayBuffer: a full-resolution original or a
+ * `download/archive` zip is hundreds of megabytes and buffering it would hold all of it in the
+ * sidecar's heap for no reason. Only the request body is held, and only for the reason above.
  */
 export async function handleOfficerRoute(cfg: UpstreamConfig, req: Request, url: URL): Promise<Response | null> {
   const rest = url.pathname.slice('/_officer/'.length);
@@ -109,12 +184,20 @@ export async function handleOfficerRoute(cfg: UpstreamConfig, req: Request, url:
   query.delete('token');
   const search = query.toString();
 
+  // Buffered, not streamed — see readBodyCapped. A body that is still being written when Immich answers
+  // loses the answer, which is what made every large upload fail with an unexplained empty 400.
+  let body: BodyInit | null = null;
+  if (hasBody && req.body) {
+    const read = await readBodyCapped(req);
+    // Over the cap (or no declared length): stream it, exactly as this always did. Not an error.
+    body = read.ok ? read.bytes : req.body;
+  }
+
   const res = await callUpstream(cfg, {
     path: `/api/${rest}`,
     method: req.method,
     query: search ? `?${search}` : '',
-    // Streamed, not buffered: an asset upload can be gigabytes and this layer never reads the bytes.
-    body: hasBody ? req.body : null,
+    body,
     contentType: req.headers.get('content-type'),
     range: req.headers.get('range'),
     ifNoneMatch: req.headers.get('if-none-match'),

src/servers/sidecar/photos/upstream.ts

@@ -165,12 +165,15 @@ type CallOptions = {
   /** Raw search string including the leading `?`, or empty. */
   query?: string;
   /**
-   * A string for the JSON callers, or the caller's own request body as a STREAM for the two forwarders.
+   * A string for the JSON callers, or the forwarder's already-buffered request body.
    *
-   * Streamed rather than buffered because a phone's video upload is bounded by `maxRequestBodySize`
-   * (4 GB) and buffering put all of it in this process's heap for a hop that never looks at the bytes.
-   * Streaming to Immich means chunked transfer-encoding instead of Content-Length — which is already how
-   * the body reaches us, since the platform proxy forwards no content-length either.
+   * This used to be a stream, for the obvious reason that a phone's video should not sit in this
+   * process's heap. It was changed back deliberately: a request body still being written when Immich
+   * answers loses the answer entirely — Node replaces the app's response with a bodyless 400 and closes
+   * — which is what made every large upload fail inexplicably. `readBodyCapped` in routes.ts has the
+   * measurements and the alternatives that were tried; the heap cost is the accepted price.
+   *
+   * A buffer also carries a Content-Length, so the request to Immich is no longer chunked.
    */
   body?: BodyInit | null;
   contentType?: string | null;