From 1597fccd7505f1710f32d87da85d38fd12f3a8a1 Mon Sep 17 00:00:00 2001 From: Pastilhas Date: Thu, 20 Aug 2026 02:37:57 +0100 Subject: [PATCH] The HTTP service: assets, probe, and the full sharp surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exposes sharp and ffprobe with their own vocabulary — operations are named what sharp names them, ffprobe fields come back spelled the way ffprobe spells them — so the tools' own documentation applies and there is no house dialect to learn. Safety comes from one architectural invariant rather than from filtering: the caller never supplies a path, a URL or a protocol. Input is an id we issued, output is a name we chose. That removes the entire class of attack that makes "just pass the flags through" a remote shell, which is precisely what makes it safe to open up everything else. docs/SECURITY.md has the reasoning and the specific attacks. The vocabulary is a data table, so the validator and docs/SHARP.md are generated from one source and cannot drift. Two things are deliberately absent from it: withMetadata, because that is how a client's GPS coordinates get published, and limitInputPixels, because a caller who could raise it could turn a 1 KB PNG into a 40 GB allocation. Verified against a running server, not asserted: traversal ids rejected, unknown operations and typo'd parameters rejected by name, out-of-range values rejected with the bound, a GPS-tagged photo coming out clean end to end, video probe, Range requests, and a rejected upload leaving nothing on disk. Found while building: config threw at import if AV_CLUB_TOKEN was unset, so generating documentation required a production secret. The token is read at boot in server.ts now — still a refusal to start, no longer a module-scope landmine. ffmpeg is designed in docs/API.md and not built yet. Co-Authored-By: Claude Opus 5 --- .env.example | 17 ++ README.md | 39 +++- docs/API.md | 208 +++++++++++++++++ docs/SECURITY.md | 144 ++++++++++++ docs/SHARP.md | 491 ++++++++++++++++++++++++++++++++++++++++ package.json | 3 + scripts/gen-docs.ts | 122 ++++++++++ src/assets/store.ts | 149 ++++++++++++ src/config.ts | 71 ++++++ src/errors.ts | 56 +++++ src/probe/ffprobe.ts | 62 +++++ src/server.ts | 188 +++++++++++++++ src/sharp/run.ts | 134 +++++++++++ src/sharp/validate.ts | 137 +++++++++++ src/sharp/vocabulary.ts | 340 ++++++++++++++++++++++++++++ 15 files changed, 2153 insertions(+), 8 deletions(-) create mode 100644 .env.example create mode 100644 docs/API.md create mode 100644 docs/SECURITY.md create mode 100644 docs/SHARP.md create mode 100644 scripts/gen-docs.ts create mode 100644 src/assets/store.ts create mode 100644 src/config.ts create mode 100644 src/errors.ts create mode 100644 src/probe/ffprobe.ts create mode 100644 src/server.ts create mode 100644 src/sharp/run.ts create mode 100644 src/sharp/validate.ts create mode 100644 src/sharp/vocabulary.ts diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..28bf85a --- /dev/null +++ b/.env.example @@ -0,0 +1,17 @@ +# Copy to .env and fill in. The service refuses to start without AV_CLUB_TOKEN. + +# The bearer token every request must carry. Generate with: +# openssl rand -hex 32 +AV_CLUB_TOKEN= + +AV_CLUB_PORT=3020 + +# Where stored assets live. Deliberately NOT inside the repo: a deploy that replaces the working +# tree would otherwise delete every original ever uploaded, silently. +# AV_CLUB_ASSETS_DIR=/var/lib/av-club/assets + +# Limits. Each exists because its absence is a denial of service — see docs/SECURITY.md. +# AV_CLUB_MAX_UPLOAD_BYTES=536870912 +# AV_CLUB_JOB_TIMEOUT_MS=120000 +# AV_CLUB_MAX_OUTPUT_DIMENSION=16383 +# AV_CLUB_MAX_CONCURRENT_JOBS=2 diff --git a/README.md b/README.md index f04e473..e311a17 100644 --- a/README.md +++ b/README.md @@ -6,27 +6,50 @@ People upload photos and videos straight off a phone or a camera: a 12 MB HEIC, Safari refuses to play, a JPEG carrying the GPS coordinates of the room it was shot in. None of that belongs on a page as-is. This is the thing in between. +It is an HTTP service that exposes as much of **sharp**, **ffmpeg** and **ffprobe** as can be exposed +safely, keeping each tool's own vocabulary — sharp operations are named what sharp names them, +ffprobe fields come back spelled the way ffprobe spells them. If you know the tool, you know the API. + +What it will not let you do is name a file. That single restriction is what makes the rest safe to +open up; see [`docs/SECURITY.md`](docs/SECURITY.md). + ## Status -**Early.** Right now this repo contains the tooling requirements, an installer, and the -documentation of what the pipeline is meant to become. The processing code is not written yet. +Assets, probing and the full sharp surface work. **ffmpeg is designed but not built** — that is the +next piece. | | | |---|---| -| `scripts/install.sh` | Brings a machine to the state this needs, then verifies it | -| `scripts/verify.sh` | Checks the tooling by running it — usable standalone | -| `docs/TOOLING.md` | What each tool is for, and the two things that bite | -| `docs/PIPELINE.md` | What the pipeline is supposed to do, and what is still undecided | +| [`docs/API.md`](docs/API.md) | The HTTP surface — start here if you are calling it | +| [`docs/SHARP.md`](docs/SHARP.md) | Every sharp operation and its bounds. Generated from the validator, so it cannot drift | +| [`docs/SECURITY.md`](docs/SECURITY.md) | The threat model and the invariants. Read before adding an endpoint | +| [`docs/TOOLING.md`](docs/TOOLING.md) | What is installed, and the two things that bite | +| [`docs/PIPELINE.md`](docs/PIPELINE.md) | What this is for, and what is still undecided | ## Quick start ```bash -./scripts/install.sh # system packages + bun deps, then verifies -./scripts/verify.sh # verification on its own +./scripts/install.sh # system packages + bun deps, then verifies +cp .env.example .env # then set AV_CLUB_TOKEN +bun dev ``` Verified on Ubuntu 24.04. The installer says so rather than guessing if it finds something else. +```bash +# upload something, then resize it +curl -X POST localhost:3020/v1/assets -H "Authorization: Bearer $AV_CLUB_TOKEN" -F file=@photo.jpg + +curl -X POST localhost:3020/v1/sharp -H "Authorization: Bearer $AV_CLUB_TOKEN" \ + -H 'Content-Type: application/json' -d '{ + "input": "as_…", + "pipeline": [{ "rotate": {} }, { "resize": { "width": 1200, "withoutEnlargement": true } }], + "output": { "format": "avif", "options": { "quality": 55 } } + }' +``` + +`bun run verify` checks the tooling; `bun run docs` regenerates `docs/SHARP.md` from the validator. + ## Why this is its own repo **It has system dependencies a web app should not have to assume.** ffmpeg, ImageMagick, libvips and diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..4d718af --- /dev/null +++ b/docs/API.md @@ -0,0 +1,208 @@ +# The HTTP API + +av-club is a Bun HTTP service. It exposes as much of sharp, ffmpeg and ffprobe as can be exposed +safely, and it keeps each tool's own vocabulary: sharp operations are named what sharp names them, +ffmpeg options are spelled the way ffmpeg spells them. If you know the tool, you know this API. + +What it does **not** let you do is name a file. See `docs/SECURITY.md` — that single restriction is +what makes the rest of the surface safe to open up. + +## Status + +| Endpoint | | +|---|---| +| `POST /v1/assets`, `GET /v1/assets/:id`, `GET /v1/assets/:id/content`, `DELETE /v1/assets/:id` | implemented | +| `GET /v1/assets/:id/probe` | implemented | +| `POST /v1/sharp` | implemented | +| `POST /v1/ffmpeg` | **not yet** — designed at the end of this document | +| `GET /health` | implemented | + +## The shape of everything + +**Nothing takes a path or a URL.** You upload bytes, you get an **asset id**, and every subsequent +operation refers to that id. Operations produce new assets and return new ids; inputs are never +modified. + +``` +upload ──> as_a1b2… ──sharp──> as_c3d4… ──download +``` + +Ids look like `as_` followed by 32 hex characters. + +## Authentication + +Every endpoint except `/health` requires: + +``` +Authorization: Bearer +``` + +The token is `AV_CLUB_TOKEN` in the service's environment. There are no user accounts — this is an +internal service between things we run. + +## Errors + +One envelope, always, so a client can parse a failure the same way it parses a success: + +```json +{ "error": { "message": "resize.width must be an integer between 1 and 16383", "type": "invalid_request", "param": "pipeline[1].resize.width" } } +``` + +| `type` | Status | Meaning | +|---|---|---| +| `invalid_request` | 400 | The request is malformed or an argument is out of range. `param` says where. | +| `unauthorized` | 401 | Missing or wrong bearer token. | +| `not_found` | 404 | No such asset. | +| `payload_too_large` | 413 | Upload exceeds the size cap. | +| `unprocessable` | 422 | Valid request, but the file could not be processed — corrupt, or not the format it claimed. | +| `timeout` | 504 | The job exceeded its wall clock limit and was killed. | +| `server_error` | 500 | Our fault. | + +Messages are written to be readable by a person, because the thing on the other end is often +relaying them to one. + +--- + +## Assets + +### `POST /v1/assets` + +`multipart/form-data` with a `file` field. + +```bash +curl -X POST http://localhost:3020/v1/assets \ + -H "Authorization: Bearer $AV_CLUB_TOKEN" \ + -F file=@photo.heic +``` + +```json +{ + "id": "as_9f86d081884c7d659a2feaa0c55ad015", + "bytes": 2847293, + "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "kind": "image", + "format": "heif", + "width": 4032, + "height": 3024, + "createdAt": "2026-08-20T01:40:00.000Z" +} +``` + +`kind` is one of `image`, `video`, `audio`. It is determined by **inspecting the bytes**, never from +the filename or the browser's `Content-Type` — see `docs/SECURITY.md`. + +The upload is rejected with `422` if it is not media we can decode, and with `413` if it exceeds +`AV_CLUB_MAX_UPLOAD_BYTES`. + +### `GET /v1/assets/:id` + +The same JSON as above. + +### `GET /v1/assets/:id/content` + +The bytes, with the right `Content-Type`. Supports `Range` requests, which matters because a browser +will not scrub — often will not even play — a `