officerdev/platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: phone photo backup contract, and the file-sync decision

Browse Source 
photo sync turned out to be mostly built already. officer proxies immich through
officer-photos, and that sidecar's allow-list already permits the `assets`
resource for GET/POST/PUT/DELETE — so immich's own upload and bulk-upload-check
endpoints are already reachable with officer's auth in front and the immich key
never leaving the sidecar. the deliverable is therefore the contract, not a new
ingest service.

three gaps are written down rather than papered over:
  - immich is not currently connected in officer (_health says configured:false),
    so none of it could be verified live. the api key moved out of .env and was
    never re-entered in the ui. owner action.
  - upload bodies are buffered twice, once in createSidecarProxy and once in the
    photos sidecar. nothing fails at phone-photo sizes; a video library would be
    unpleasant. fixing it touches the shared factory, so it is a decision.
  - no resumable upload. immich's own app has the same limitation.

file sync is design-only, as agreed. the recommendation is syncthing supervised
as a sidecar rather than reimplementing nextcloud's sync protocol — a mount is
not a sync, and the local-first replica is the whole feature.

the iOS answer is stated plainly because it changes the design: continuous
background sync is not possible there, which is why nextcloud's own iOS app is
manual too. if iOS matters, webdav becomes first-class rather than optional, and
that is the owner's call to make before any code is written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
pastilhas
2026-08-04 03:27:01 +00:00
co-authored by Claude Opus 5
parent edf26323da
commit bc52ce6361
2 changed files with 295 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/file-sync.md
+128
View File
@@ -0,0 +1,128 @@
# File sync
Written 2026-08-04. The "final boss" of the NextCloud replacement — separated from
`nextcloud-replacement.md` because the answer is genuinely different from calendar and contacts, and
conflating them is how this becomes a six-month project.
The ask: _"the synced files functionality that I can run on each computer and phone"_, with the same
acceptance test as the rest — **it has to just work.**
---
## The distinction that decides everything
**A mount is not a sync.**
| | Mount (WebDAV, SMB, sshfs) | Sync (NextCloud client, Syncthing, Dropbox) |
| ---------------- | -------------------------- | ------------------------------------------- |
| Offline | Nothing works | Everything works |
| Open a 2 GB file | Streams over the network | Local disk read |
| Flaky connection | Failed saves, hung apps | Queues, retries |
| Conflicts | Last writer wins, silently | Detected and surfaced |
| Cost to build | Days | The rest of the year |
What NextCloud's desktop client actually gives you is a **local-first replica**: a full copy on every
device, a local database of file state, change detection via filesystem watching, transfer scheduling,
and explicit conflict handling. That is the thing being asked for, and it is a genuinely hard,
well-studied distributed-systems problem.
---
## Decision: run Syncthing, do not write a sync engine
Same reasoning as choosing Radicale over writing CalDAV, only more so.
| Option | Verdict |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Syncthing** | **Chosen.** Mature (since 2013), open source, peer-to-peer, real clients on Linux/macOS/Windows/Android, explicit conflict files, block-level transfer, no server account model to build. |
| WebDAV from the caldav sidecar | Radicale is calendars and contacts only, so this would be a second server anyway — and it solves "see my files remotely", not "have my files offline". **Complementary, not a substitute.** Cheap to add later. |
| Reimplement NextCloud's sync protocol | Chunked upload, ETag bookkeeping, a local state DB, conflict resolution, and a client on three platforms. The single largest item on the whole list, to arrive at something worse than what exists. |
### Why Syncthing fits this platform specifically
It is peer-to-peer, so there is **no server-side account model to build** — devices authenticate each
other by device ID. That removes the entire class of work that a Dropbox-alike would need. Officer's
job shrinks to: run it, own its config, show its state, and point the file browser at the folder.
It also composes with what is already here: the machine already runs Headscale, so devices are already
on a private network and Syncthing's discovery/relay layer can be turned down or off entirely.
---
## Shape of the work
### `officer-syncthing` sidecar
Same pattern as `officer-caldav` (supervise a real server) crossed with `officer-memos` (hold a
credential, proxy an API).
- Supervise the `syncthing` binary on a **loopback** port, like Radicale.
- Generate its config at boot rather than hand-maintaining it — same reasoning as
`caldav/radicale.ts`: config is derived state, and a hand-edit that disagrees with what the code
believes is very hard to debug.
- Syncthing's own REST API is authenticated with an API key it writes into its config; the sidecar
reads that key and holds it. The platform never sees it.
- Storage under `DATA_PATH/sync/`, so it is inside the existing backup story.
### Platform side
Sixteen lines of `createSidecarProxy` at `/api/sync`, exactly like memos and transmission. No sync
knowledge in the platform, ever.
### UI — replicate Syncthing's own UI first
Per the standing rule: reproduce what the service already ships, then extend. Syncthing's own web UI
covers folders, devices, sync status/percentage, and conflicts. Those are the four panels. The
extension Officer can add that Syncthing cannot is the interesting part: **the synced tree is just a
directory**, so the existing file browser, code editor and file viewer all work on it for free. That is
the integration nothing else offers.
### Build order
1. Sidecar supervising syncthing + config generation + registration.
2. `/api/sync` proxy.
3. Status panel (folders, devices, completion) — read-only.
4. Add/remove folder and device pairing from the UI.
5. Surface the synced root in the file browser as a first-class location.
---
## The honest problem: iOS
**There is no good Syncthing on iOS, and there will not be one.** iOS does not allow a
general-purpose background daemon doing continuous filesystem work; the only always-on sync clients
that exist there are ones Apple grants entitlements for. This is also why NextCloud's own iOS app is a
manual, foreground, tap-to-upload experience rather than a real sync client — the platform, not the
software, is the limit.
So "on each computer and phone" splits:
- **Linux/macOS/Windows** — solved by Syncthing, properly, today.
- **Android** — solved by Syncthing (there are maintained forks; verify which is current before
recommending one).
- **iOS** — _not_ solvable as continuous sync. The realistic shape is a **WebDAV mount for browsing and
on-demand download, plus explicit upload from the Officer mobile app**. That is a different feature
with a different backend, and it is a decision for the owner rather than something to assume.
This is worth deciding before step 1, because "iOS matters" changes the answer: if it does, a WebDAV
server becomes a first-class part of the design rather than an optional later addition, and it may be
worth serving both from one place.
---
## What this does NOT do
- **It is not a replacement for the file browser.** Officer already has one, and it already reaches the
whole filesystem. Syncthing adds replication, not access.
- **It is not backup.** Sync propagates deletions. A synced folder is not a backup of itself, and
anyone who believes otherwise finds out at the worst moment. Versioning (Syncthing has several
strategies) should be enabled and surfaced in the UI precisely so this is not confused.
- **It is not sharing.** Single-user remains a hard platform invariant.
---
## Status
Design only. Nothing built — this was written before implementation on the night of 2026-08-04 and the
implementation ran out of night. The two things to settle before writing code are **the iOS question
above** and whether the synced root should live under `DATA_PATH` or somewhere the owner picks.
docs/mobile-photo-sync-api.md
+167
View File
@@ -0,0 +1,167 @@
# Phone photo auto-backup — API contract
Written 2026-08-04, for the mobile developer. This is the server-side contract for "photos taken on the
phone appear on the server automatically", i.e. what the Immich app does today.
**The mobile app is not built here.** This document is the boundary: everything below is either already
live or explicitly listed as missing, so the app can be built against it rather than reverse-engineered
from the web client.
---
## The short version
**Almost all of this already exists.** Officer proxies a self-hosted Immich through the `officer-photos`
sidecar, and the sidecar's route allow-list already permits the `assets` resource for `GET POST PUT
DELETE` (`src/servers/sidecar/photos/routes.ts:22`). Immich's own upload and dedupe endpoints are
therefore already reachable through Officer, with Officer's auth in front and the Immich API key never
leaving the sidecar.
So the work is not "build a photo ingest service". It is: use the endpoints below, and close the three
gaps at the bottom.
---
## Topology
```
phone app officer officer-photos Immich
───────── ─────── ────────────── ──────
Authorization: → userMiddleware (JWT) → allow-list by first → x-api-key
Bearer <platform createSidecarProxy path segment + (held ONLY
JWT> injects X-Officer-User method in the sidecar)
```
The phone holds a **platform JWT** — the same credential `monorepo-mobile/packages/core/src/services/
api.ts:58` already sends as `Authorization: Bearer <token>`. It never holds an Immich key and never
talks to Immich directly.
Base path for everything below: **`/api/photos/_officer/…`**, which maps to `<immich>/api/…`.
So `/api/photos/_officer/assets` is Immich's `POST /api/assets`.
---
## Endpoints
### 1. Is the library reachable at all
```
GET /api/photos/_health
→ 200 { ok: true, configured: true, account: "<label>", version: "…", ms: 42 }
→ 503 { ok: false, configured: false, error: "not connected" }
→ 502 { ok: false, configured: true, error: "…" }
```
`configured: false` means the owner has not connected an Immich account in Officer yet. Show that as a
setup prompt, not an error — it is not something the app can fix.
### 2. Ask what the server already has ← this is what makes backup cheap
```
POST /api/photos/_officer/assets/bulk-upload-check
Content-Type: application/json
{ "assets": [ { "id": "<deviceAssetId>", "checksum": "<base64 sha1 of the file bytes>" }, … ] }
→ 200 { "results": [ { "id": "…", "action": "accept" | "reject", "reason": "duplicate" | …,
"assetId": "<existing server id, when rejected as duplicate>" }, … ] }
```
Call this **before** uploading anything. It is the difference between a backup pass costing a few KB and
costing the whole camera roll. Batch generously — a few hundred ids per call.
`checksum` is base64-encoded **SHA-1** of the raw file bytes. Not SHA-256, not hex.
### 3. Upload one asset
```
POST /api/photos/_officer/assets
Content-Type: multipart/form-data
assetData the file bytes (required)
deviceAssetId stable per-device id for this photo (required)
deviceId stable id for this device (required)
fileCreatedAt ISO 8601 (required)
fileModifiedAt ISO 8601 (required)
isFavorite "true" | "false"
duration for video
livePhotoVideoId when pairing a live photo's video part
→ 201 { "id": "<asset id>", "status": "created" }
→ 200 { "id": "<existing id>", "status": "duplicate" }
```
`deviceAssetId` + `deviceId` is the identity pair Immich dedupes on, alongside the checksum. Keep both
stable across app reinstalls if you can — if they change, the checksum still prevents duplicate _files_,
but the server will not recognise the asset as one it has already seen from this device.
### 4. Albums (optional, for "put my backups in an album")
```
GET /api/photos/_officer/albums
POST /api/photos/_officer/albums { albumName, assetIds? }
PUT /api/photos/_officer/albums/<id>/assets { ids: [...] }
```
### 5. Reading the library back
```
GET /api/photos/_officer/assets/<id>/thumbnail?size=preview|thumbnail
GET /api/photos/_officer/assets/<id>/original
POST /api/photos/_officer/search/metadata { … }
GET /api/photos/_officer/timeline/…
```
Range and `If-None-Match` are forwarded, so thumbnails cache and videos seek properly.
---
## The backup loop the app should implement
1. Enumerate new local assets since the last successful run (by creation date, per album/bucket).
2. Compute SHA-1 for each; batch into `bulk-upload-check`.
3. Upload only the `accept` ones, one at a time, honouring a user setting for "wi-fi only".
4. Record the returned server id against the local asset id so step 1 can skip it next time.
5. On failure, retry with backoff. **Do not** assume a failed upload means the asset is absent — re-run
the check first; a request can succeed server-side and fail on the way back.
Immich's own app does essentially this, which is the reason to match it rather than invent something.
---
## Gaps — what is NOT done, and who has to close it
### A. Immich is not currently connected in Officer _(owner action, blocks live testing)_
`GET /api/photos/_health` returns `configured: false` right now. The Immich container is running
(`immich_server`, port 2283) but Officer has no API key stored for it — the key used to live in `.env`
as `IMMICH_API_KEY` and was correctly removed when credentials moved into the sidecar's own storage.
To close: in Immich, Account Settings → API Keys → new key; in Officer, `/photos` → connection settings
→ paste URL + key. **Until this is done none of the endpoints above can be tested end to end**, which is
why this document describes them from the code rather than from a live capture.
### B. Upload bodies are fully buffered, twice _(server work, needs a decision)_
`createSidecarProxy` does `await ctx.req.arrayBuffer()` (`create-proxy.ts:103`) and the photos sidecar
does it again (`photos/routes.ts:93`). A 4K video is therefore held in memory twice per upload. Limits
are generous (server 50 GB, photos sidecar 4 GB) so nothing will _fail_ at phone-photo sizes, but a
backup pass over a large video library will make the platform's memory graph unpleasant.
The fix is to stream the body through instead of buffering, which touches the shared proxy factory and
so affects every sidecar. Worth doing before heavy use; not worth blocking the app on.
### C. No resumable upload _(server work, needs a decision)_
An interrupted upload restarts from zero. On mobile data that is a real cost for videos. Immich's own
app has the same limitation, so matching it is defensible for a first version — but if it turns out to
matter, this is the piece that has to be designed rather than borrowed.
---
## Things deliberately not exposed
The sidecar's allow-list refuses Immich's `admin/*`, `auth/*`, `oauth/*`, `api-keys/*`, `sessions/*`,
`jobs/*`, `system-config/*`, `system-metadata/*` and `libraries/*`. A proxy that can mint its own
credentials is not a proxy. If the app genuinely needs something behind that line, it should be a
decision, not a widened regex.