docs: phone photo backup contract, and the file-sync decision
photo sync turned out to be mostly built already. officer proxies immich through
officer-photos, and that sidecar's allow-list already permits the `assets`
resource for GET/POST/PUT/DELETE — so immich's own upload and bulk-upload-check
endpoints are already reachable with officer's auth in front and the immich key
never leaving the sidecar. the deliverable is therefore the contract, not a new
ingest service.
three gaps are written down rather than papered over:
- immich is not currently connected in officer (_health says configured:false),
so none of it could be verified live. the api key moved out of .env and was
never re-entered in the ui. owner action.
- upload bodies are buffered twice, once in createSidecarProxy and once in the
photos sidecar. nothing fails at phone-photo sizes; a video library would be
unpleasant. fixing it touches the shared factory, so it is a decision.
- no resumable upload. immich's own app has the same limitation.
file sync is design-only, as agreed. the recommendation is syncthing supervised
as a sidecar rather than reimplementing nextcloud's sync protocol — a mount is
not a sync, and the local-first replica is the whole feature.
the iOS answer is stated plainly because it changes the design: continuous
background sync is not possible there, which is why nextcloud's own iOS app is
manual too. if iOS matters, webdav becomes first-class rather than optional, and
that is the owner's call to make before any code is written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,128 @@
|
||||
# File sync
|
||||
|
||||
Written 2026-08-04. The "final boss" of the NextCloud replacement — separated from
|
||||
`nextcloud-replacement.md` because the answer is genuinely different from calendar and contacts, and
|
||||
conflating them is how this becomes a six-month project.
|
||||
|
||||
The ask: _"the synced files functionality that I can run on each computer and phone"_, with the same
|
||||
acceptance test as the rest — **it has to just work.**
|
||||
|
||||
---
|
||||
|
||||
## The distinction that decides everything
|
||||
|
||||
**A mount is not a sync.**
|
||||
|
||||
| | Mount (WebDAV, SMB, sshfs) | Sync (NextCloud client, Syncthing, Dropbox) |
|
||||
| ---------------- | -------------------------- | ------------------------------------------- |
|
||||
| Offline | Nothing works | Everything works |
|
||||
| Open a 2 GB file | Streams over the network | Local disk read |
|
||||
| Flaky connection | Failed saves, hung apps | Queues, retries |
|
||||
| Conflicts | Last writer wins, silently | Detected and surfaced |
|
||||
| Cost to build | Days | The rest of the year |
|
||||
|
||||
What NextCloud's desktop client actually gives you is a **local-first replica**: a full copy on every
|
||||
device, a local database of file state, change detection via filesystem watching, transfer scheduling,
|
||||
and explicit conflict handling. That is the thing being asked for, and it is a genuinely hard,
|
||||
well-studied distributed-systems problem.
|
||||
|
||||
---
|
||||
|
||||
## Decision: run Syncthing, do not write a sync engine
|
||||
|
||||
Same reasoning as choosing Radicale over writing CalDAV, only more so.
|
||||
|
||||
| Option | Verdict |
|
||||
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Syncthing** | **Chosen.** Mature (since 2013), open source, peer-to-peer, real clients on Linux/macOS/Windows/Android, explicit conflict files, block-level transfer, no server account model to build. |
|
||||
| WebDAV from the caldav sidecar | Radicale is calendars and contacts only, so this would be a second server anyway — and it solves "see my files remotely", not "have my files offline". **Complementary, not a substitute.** Cheap to add later. |
|
||||
| Reimplement NextCloud's sync protocol | Chunked upload, ETag bookkeeping, a local state DB, conflict resolution, and a client on three platforms. The single largest item on the whole list, to arrive at something worse than what exists. |
|
||||
|
||||
### Why Syncthing fits this platform specifically
|
||||
|
||||
It is peer-to-peer, so there is **no server-side account model to build** — devices authenticate each
|
||||
other by device ID. That removes the entire class of work that a Dropbox-alike would need. Officer's
|
||||
job shrinks to: run it, own its config, show its state, and point the file browser at the folder.
|
||||
|
||||
It also composes with what is already here: the machine already runs Headscale, so devices are already
|
||||
on a private network and Syncthing's discovery/relay layer can be turned down or off entirely.
|
||||
|
||||
---
|
||||
|
||||
## Shape of the work
|
||||
|
||||
### `officer-syncthing` sidecar
|
||||
|
||||
Same pattern as `officer-caldav` (supervise a real server) crossed with `officer-memos` (hold a
|
||||
credential, proxy an API).
|
||||
|
||||
- Supervise the `syncthing` binary on a **loopback** port, like Radicale.
|
||||
- Generate its config at boot rather than hand-maintaining it — same reasoning as
|
||||
`caldav/radicale.ts`: config is derived state, and a hand-edit that disagrees with what the code
|
||||
believes is very hard to debug.
|
||||
- Syncthing's own REST API is authenticated with an API key it writes into its config; the sidecar
|
||||
reads that key and holds it. The platform never sees it.
|
||||
- Storage under `DATA_PATH/sync/`, so it is inside the existing backup story.
|
||||
|
||||
### Platform side
|
||||
|
||||
Sixteen lines of `createSidecarProxy` at `/api/sync`, exactly like memos and transmission. No sync
|
||||
knowledge in the platform, ever.
|
||||
|
||||
### UI — replicate Syncthing's own UI first
|
||||
|
||||
Per the standing rule: reproduce what the service already ships, then extend. Syncthing's own web UI
|
||||
covers folders, devices, sync status/percentage, and conflicts. Those are the four panels. The
|
||||
extension Officer can add that Syncthing cannot is the interesting part: **the synced tree is just a
|
||||
directory**, so the existing file browser, code editor and file viewer all work on it for free. That is
|
||||
the integration nothing else offers.
|
||||
|
||||
### Build order
|
||||
|
||||
1. Sidecar supervising syncthing + config generation + registration.
|
||||
2. `/api/sync` proxy.
|
||||
3. Status panel (folders, devices, completion) — read-only.
|
||||
4. Add/remove folder and device pairing from the UI.
|
||||
5. Surface the synced root in the file browser as a first-class location.
|
||||
|
||||
---
|
||||
|
||||
## The honest problem: iOS
|
||||
|
||||
**There is no good Syncthing on iOS, and there will not be one.** iOS does not allow a
|
||||
general-purpose background daemon doing continuous filesystem work; the only always-on sync clients
|
||||
that exist there are ones Apple grants entitlements for. This is also why NextCloud's own iOS app is a
|
||||
manual, foreground, tap-to-upload experience rather than a real sync client — the platform, not the
|
||||
software, is the limit.
|
||||
|
||||
So "on each computer and phone" splits:
|
||||
|
||||
- **Linux/macOS/Windows** — solved by Syncthing, properly, today.
|
||||
- **Android** — solved by Syncthing (there are maintained forks; verify which is current before
|
||||
recommending one).
|
||||
- **iOS** — _not_ solvable as continuous sync. The realistic shape is a **WebDAV mount for browsing and
|
||||
on-demand download, plus explicit upload from the Officer mobile app**. That is a different feature
|
||||
with a different backend, and it is a decision for the owner rather than something to assume.
|
||||
|
||||
This is worth deciding before step 1, because "iOS matters" changes the answer: if it does, a WebDAV
|
||||
server becomes a first-class part of the design rather than an optional later addition, and it may be
|
||||
worth serving both from one place.
|
||||
|
||||
---
|
||||
|
||||
## What this does NOT do
|
||||
|
||||
- **It is not a replacement for the file browser.** Officer already has one, and it already reaches the
|
||||
whole filesystem. Syncthing adds replication, not access.
|
||||
- **It is not backup.** Sync propagates deletions. A synced folder is not a backup of itself, and
|
||||
anyone who believes otherwise finds out at the worst moment. Versioning (Syncthing has several
|
||||
strategies) should be enabled and surfaced in the UI precisely so this is not confused.
|
||||
- **It is not sharing.** Single-user remains a hard platform invariant.
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
Design only. Nothing built — this was written before implementation on the night of 2026-08-04 and the
|
||||
implementation ran out of night. The two things to settle before writing code are **the iOS question
|
||||
above** and whether the synced root should live under `DATA_PATH` or somewhere the owner picks.
|
||||
@@ -0,0 +1,167 @@
|
||||
# Phone photo auto-backup — API contract
|
||||
|
||||
Written 2026-08-04, for the mobile developer. This is the server-side contract for "photos taken on the
|
||||
phone appear on the server automatically", i.e. what the Immich app does today.
|
||||
|
||||
**The mobile app is not built here.** This document is the boundary: everything below is either already
|
||||
live or explicitly listed as missing, so the app can be built against it rather than reverse-engineered
|
||||
from the web client.
|
||||
|
||||
---
|
||||
|
||||
## The short version
|
||||
|
||||
**Almost all of this already exists.** Officer proxies a self-hosted Immich through the `officer-photos`
|
||||
sidecar, and the sidecar's route allow-list already permits the `assets` resource for `GET POST PUT
|
||||
DELETE` (`src/servers/sidecar/photos/routes.ts:22`). Immich's own upload and dedupe endpoints are
|
||||
therefore already reachable through Officer, with Officer's auth in front and the Immich API key never
|
||||
leaving the sidecar.
|
||||
|
||||
So the work is not "build a photo ingest service". It is: use the endpoints below, and close the three
|
||||
gaps at the bottom.
|
||||
|
||||
---
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
phone app officer officer-photos Immich
|
||||
───────── ─────── ────────────── ──────
|
||||
Authorization: → userMiddleware (JWT) → allow-list by first → x-api-key
|
||||
Bearer <platform createSidecarProxy path segment + (held ONLY
|
||||
JWT> injects X-Officer-User method in the sidecar)
|
||||
```
|
||||
|
||||
The phone holds a **platform JWT** — the same credential `monorepo-mobile/packages/core/src/services/
|
||||
api.ts:58` already sends as `Authorization: Bearer <token>`. It never holds an Immich key and never
|
||||
talks to Immich directly.
|
||||
|
||||
Base path for everything below: **`/api/photos/_officer/…`**, which maps to `<immich>/api/…`.
|
||||
So `/api/photos/_officer/assets` is Immich's `POST /api/assets`.
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### 1. Is the library reachable at all
|
||||
|
||||
```
|
||||
GET /api/photos/_health
|
||||
→ 200 { ok: true, configured: true, account: "<label>", version: "…", ms: 42 }
|
||||
→ 503 { ok: false, configured: false, error: "not connected" }
|
||||
→ 502 { ok: false, configured: true, error: "…" }
|
||||
```
|
||||
|
||||
`configured: false` means the owner has not connected an Immich account in Officer yet. Show that as a
|
||||
setup prompt, not an error — it is not something the app can fix.
|
||||
|
||||
### 2. Ask what the server already has ← this is what makes backup cheap
|
||||
|
||||
```
|
||||
POST /api/photos/_officer/assets/bulk-upload-check
|
||||
Content-Type: application/json
|
||||
|
||||
{ "assets": [ { "id": "<deviceAssetId>", "checksum": "<base64 sha1 of the file bytes>" }, … ] }
|
||||
|
||||
→ 200 { "results": [ { "id": "…", "action": "accept" | "reject", "reason": "duplicate" | …,
|
||||
"assetId": "<existing server id, when rejected as duplicate>" }, … ] }
|
||||
```
|
||||
|
||||
Call this **before** uploading anything. It is the difference between a backup pass costing a few KB and
|
||||
costing the whole camera roll. Batch generously — a few hundred ids per call.
|
||||
|
||||
`checksum` is base64-encoded **SHA-1** of the raw file bytes. Not SHA-256, not hex.
|
||||
|
||||
### 3. Upload one asset
|
||||
|
||||
```
|
||||
POST /api/photos/_officer/assets
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
assetData the file bytes (required)
|
||||
deviceAssetId stable per-device id for this photo (required)
|
||||
deviceId stable id for this device (required)
|
||||
fileCreatedAt ISO 8601 (required)
|
||||
fileModifiedAt ISO 8601 (required)
|
||||
isFavorite "true" | "false"
|
||||
duration for video
|
||||
livePhotoVideoId when pairing a live photo's video part
|
||||
|
||||
→ 201 { "id": "<asset id>", "status": "created" }
|
||||
→ 200 { "id": "<existing id>", "status": "duplicate" }
|
||||
```
|
||||
|
||||
`deviceAssetId` + `deviceId` is the identity pair Immich dedupes on, alongside the checksum. Keep both
|
||||
stable across app reinstalls if you can — if they change, the checksum still prevents duplicate _files_,
|
||||
but the server will not recognise the asset as one it has already seen from this device.
|
||||
|
||||
### 4. Albums (optional, for "put my backups in an album")
|
||||
|
||||
```
|
||||
GET /api/photos/_officer/albums
|
||||
POST /api/photos/_officer/albums { albumName, assetIds? }
|
||||
PUT /api/photos/_officer/albums/<id>/assets { ids: [...] }
|
||||
```
|
||||
|
||||
### 5. Reading the library back
|
||||
|
||||
```
|
||||
GET /api/photos/_officer/assets/<id>/thumbnail?size=preview|thumbnail
|
||||
GET /api/photos/_officer/assets/<id>/original
|
||||
POST /api/photos/_officer/search/metadata { … }
|
||||
GET /api/photos/_officer/timeline/…
|
||||
```
|
||||
|
||||
Range and `If-None-Match` are forwarded, so thumbnails cache and videos seek properly.
|
||||
|
||||
---
|
||||
|
||||
## The backup loop the app should implement
|
||||
|
||||
1. Enumerate new local assets since the last successful run (by creation date, per album/bucket).
|
||||
2. Compute SHA-1 for each; batch into `bulk-upload-check`.
|
||||
3. Upload only the `accept` ones, one at a time, honouring a user setting for "wi-fi only".
|
||||
4. Record the returned server id against the local asset id so step 1 can skip it next time.
|
||||
5. On failure, retry with backoff. **Do not** assume a failed upload means the asset is absent — re-run
|
||||
the check first; a request can succeed server-side and fail on the way back.
|
||||
|
||||
Immich's own app does essentially this, which is the reason to match it rather than invent something.
|
||||
|
||||
---
|
||||
|
||||
## Gaps — what is NOT done, and who has to close it
|
||||
|
||||
### A. Immich is not currently connected in Officer _(owner action, blocks live testing)_
|
||||
|
||||
`GET /api/photos/_health` returns `configured: false` right now. The Immich container is running
|
||||
(`immich_server`, port 2283) but Officer has no API key stored for it — the key used to live in `.env`
|
||||
as `IMMICH_API_KEY` and was correctly removed when credentials moved into the sidecar's own storage.
|
||||
|
||||
To close: in Immich, Account Settings → API Keys → new key; in Officer, `/photos` → connection settings
|
||||
→ paste URL + key. **Until this is done none of the endpoints above can be tested end to end**, which is
|
||||
why this document describes them from the code rather than from a live capture.
|
||||
|
||||
### B. Upload bodies are fully buffered, twice _(server work, needs a decision)_
|
||||
|
||||
`createSidecarProxy` does `await ctx.req.arrayBuffer()` (`create-proxy.ts:103`) and the photos sidecar
|
||||
does it again (`photos/routes.ts:93`). A 4K video is therefore held in memory twice per upload. Limits
|
||||
are generous (server 50 GB, photos sidecar 4 GB) so nothing will _fail_ at phone-photo sizes, but a
|
||||
backup pass over a large video library will make the platform's memory graph unpleasant.
|
||||
|
||||
The fix is to stream the body through instead of buffering, which touches the shared proxy factory and
|
||||
so affects every sidecar. Worth doing before heavy use; not worth blocking the app on.
|
||||
|
||||
### C. No resumable upload _(server work, needs a decision)_
|
||||
|
||||
An interrupted upload restarts from zero. On mobile data that is a real cost for videos. Immich's own
|
||||
app has the same limitation, so matching it is defensible for a first version — but if it turns out to
|
||||
matter, this is the piece that has to be designed rather than borrowed.
|
||||
|
||||
---
|
||||
|
||||
## Things deliberately not exposed
|
||||
|
||||
The sidecar's allow-list refuses Immich's `admin/*`, `auth/*`, `oauth/*`, `api-keys/*`, `sessions/*`,
|
||||
`jobs/*`, `system-config/*`, `system-metadata/*` and `libraries/*`. A proxy that can mint its own
|
||||
credentials is not a proxy. If the app genuinely needs something behind that line, it should be a
|
||||
decision, not a widened regex.
|
||||
Reference in New Issue
Block a user