extract speech-to-text into its own plugin

Moves platform STT out into a plugin that owns the whole whisper.cpp contract: sidecar,
config (a JSON file under DATA_PATH/transcribe/, validated against the server before
storing), a thin proxy router, a skill so an agent knows the contract without being told,
and a script-mode file-browser task (not agentic — no model turn per audio file).

sidecar/transcribe.ts and its test moved unchanged from src/servers/api/stt/ — every
constant in it has a reason attached and none of it was touched.

The platform-side removal (routes, dead useAudioRecording path, Settings screen, locale
keys) lands in a separate, uncommitted change in the platform repo for review.

Not yet reachable from the file browser: the platform does not load tasks from a plugin's
directory yet (skills/affordances/notifications do). Ships anyway per instructions.

Officer-Session: 656f5c85-e49a-4f1f-b9f3-042a04b4d043
This commit is contained in:
2026-08-24 03:28:04 +00:00
commit 3da6d11c00
11 changed files with 1192 additions and 0 deletions
+69
View File
@@ -0,0 +1,69 @@
---
name: Transcribe
description: Use when you need to turn a recording into text — a voice memo, a meeting, dictation on disk. Carries the route, the request shape, and what to do when nothing is configured.
version: 1
---
Officer's speech-to-text runs entirely inside the `transcribe` plugin, over your own whisper.cpp server.
Nothing else on the platform reads audio for you — a request that mentions a recording, a voice memo, a
call, a lecture, means this plugin.
**There is no model turn here.** The plugin does the whole job itself: detect the spoken language, then
transcribe faithfully in it. Do not narrate the audio yourself, do not summarize it in place of
transcribing, and do not chain a translation step — the plugin is already faithful to the source language,
and Whisper's translate mode is English-only, so anything past it would be a rough loss rather than a
feature.
## Is it set up?
```
GET /api/transcribe/config
→ { "configured": true, "url": "http://192.168.1.50:64201" }
→ { "configured": false, "url": null }
```
If `configured` is `false`, say so plainly and stop — "speech-to-text isn't set up yet; point it at a
whisper.cpp server with `PUT /api/transcribe/config`" — rather than attempting the transcription and
reporting a generic failure. Setting it up is a one-time, deliberate act (it validates the URL against the
server before storing it), not something to do silently on the caller's behalf without being asked.
```
PUT /api/transcribe/config
{ "url": "http://192.168.1.50:64201" }
→ { "configured": true, "url": "http://192.168.1.50:64201" }
→ 400 { "error": "…" } — url missing, malformed, or the server did not answer
```
## Transcribing
```
POST /api/transcribe/transcribe
multipart/form-data:
file — the audio file (mp3, wav, m4a, whatever whisper.cpp accepts)
spokenLanguages — zero or more, e.g. spokenLanguages=en&spokenLanguages=pt
→ { "text": "…", "detectedLanguage": "en" }
→ 400 { "error": "Whisper not configured — PUT /api/transcribe/config first" }
→ 502 { "error": "…" } — the whisper.cpp server rejected or failed the request
```
`spokenLanguages` only matters when Whisper's own detection is genuinely unsure — it breaks the tie
toward a language you already know the speaker uses. Pass what you know; pass nothing and detection still
works, it is just less informed on an ambiguous clip.
**This can take minutes** on a long recording — whisper.cpp runs at roughly 8x realtime, so a 40-minute
memo is a couple of minutes of waiting, not a couple of seconds. Do not treat a slow response as a failure
and retry; wait for it.
## The file-browser task
`tasks/transcribe-audio-file` in this plugin ships a right-click task for `.mp3`/`.wav`/`.m4a` files that
calls this same route and files the result next to the original — a **script** task, deliberately, with no
model turn per audio file. If you are asked to transcribe something already sitting in the file browser,
prefer pointing the user at that over doing it yourself turn-by-turn.
## What this is not
Not a live dictation microphone — there is no streaming endpoint, only a file upload. Not a translator —
it transcribes in whatever language was spoken, never into a different one. And it holds no memory of past
transcriptions; every call is independent.