344 lines
10 KiB
Markdown
344 lines
10 KiB
Markdown
---
|
|
name: mlx.audio
|
|
description: Generate speech from text and transcribe audio using mlx-audio. Use when the user wants text-to-speech synthesis, speech-to-text transcription, voice cloning, audio separation, or speech-to-speech processing on Apple Silicon.
|
|
---
|
|
|
|
# MLX-Audio
|
|
|
|
A speech processing library built on Apple's MLX framework, providing TTS, STT, speech-to-speech (STS), and audio separation optimized for Apple Silicon.
|
|
|
|
- **Repository:** https://github.com/Blaizzy/mlx-audio
|
|
- **License:** MIT
|
|
|
|
## CLI Tools
|
|
|
|
### Text-to-Speech (TTS)
|
|
|
|
```bash
|
|
mlx_audio.tts.generate --model <model> --text '<text>' [options]
|
|
```
|
|
|
|
| Flag | Type | Default | Description |
|
|
|------|------|---------|-------------|
|
|
| `--model` | string | required | HuggingFace model ID |
|
|
| `--text` | string | required | Text to synthesize |
|
|
| `--voice` | string | — | Voice preset (model-specific) |
|
|
| `--speed` | float | 1.0 | Speech speed multiplier |
|
|
| `--lang_code` | string | `a` | Language code |
|
|
| `--play` | flag | — | Play audio immediately |
|
|
| `--output_path` | string | — | Directory to save audio |
|
|
| `--ref_audio` | string | — | Reference audio for voice cloning (CSM) |
|
|
|
|
#### Language Codes
|
|
|
|
| Code | Language |
|
|
|------|----------|
|
|
| `a` | American English |
|
|
| `b` | British English |
|
|
| `j` | Japanese |
|
|
| `z` | Mandarin Chinese |
|
|
| `e` | Spanish |
|
|
| `f` | French |
|
|
|
|
#### Kokoro Voices
|
|
|
|
| Voice | Description |
|
|
|-------|-------------|
|
|
| `af_heart`, `af_bella`, `af_nova`, `af_sky` | American female |
|
|
| `am_adam`, `am_echo` | American male |
|
|
| `bf_alice`, `bf_emma` | British female |
|
|
| `bm_daniel`, `bm_george` | British male |
|
|
| `jf_alpha`, `jm_kumo` | Japanese |
|
|
| `zf_xiaobei`, `zm_yunxi` | Chinese |
|
|
|
|
#### Examples
|
|
|
|
```bash
|
|
# Basic generation
|
|
mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 --text 'Hello, world!' --lang_code a
|
|
|
|
# With voice and speed
|
|
mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 --text 'Hello!' --voice af_heart --speed 1.2 --lang_code a
|
|
|
|
# Play immediately
|
|
mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 --text 'Hello!' --play --lang_code a
|
|
|
|
# Voice cloning with CSM
|
|
mlx_audio.tts.generate --model mlx-community/csm-1b --text "Hello from Sesame." --ref_audio ./reference_voice.wav --play
|
|
```
|
|
|
|
### Speech-to-Text (STT)
|
|
|
|
```bash
|
|
python -m mlx_audio.stt.generate --model <model> --audio <file> [options]
|
|
```
|
|
|
|
| Flag | Type | Default | Description |
|
|
|------|------|---------|-------------|
|
|
| `--model` | string | required | HuggingFace model ID |
|
|
| `--audio` | string | required | Input audio file |
|
|
| `--language` | string | — | Language code |
|
|
| `--max-tokens` | int | 1024 | Maximum output tokens |
|
|
| `--temperature` | float | 0.0 | Sampling temperature |
|
|
| `--context` | string | — | Hotwords/metadata for context |
|
|
| `--output-path` | string | — | Output directory |
|
|
| `--format` | string | — | Output format (e.g. `json`) |
|
|
| `--stream` | flag | — | Enable streaming mode |
|
|
| `--verbose` | flag | — | Detailed logging |
|
|
|
|
#### Examples
|
|
|
|
```bash
|
|
# Basic transcription
|
|
python -m mlx_audio.stt.generate --model mlx-community/whisper-large-v3-turbo-asr-fp16 --audio speech.wav --verbose
|
|
|
|
# With context for technical terms
|
|
python -m mlx_audio.stt.generate --model mlx-community/VibeVoice-ASR-bf16 --audio meeting.wav --context "MLX, Apple Silicon, PyTorch" --max-tokens 8192 --format json --verbose
|
|
|
|
# Parakeet model
|
|
python -m mlx_audio.stt.generate --model mlx-community/parakeet-tdt-0.6b-v3 --audio speech.wav --format json --verbose
|
|
```
|
|
|
|
---
|
|
|
|
## Python API
|
|
|
|
### TTS
|
|
|
|
```python
|
|
from mlx_audio.tts.utils import load_model
|
|
|
|
model = load_model("mlx-community/Kokoro-82M-bf16")
|
|
for result in model.generate("Hello from MLX-Audio!", voice="af_heart"):
|
|
audio = result.audio # mx.array waveform
|
|
```
|
|
|
|
### STT
|
|
|
|
```python
|
|
from mlx_audio.stt.generate import generate_transcription
|
|
|
|
result = generate_transcription(
|
|
model="mlx-community/whisper-large-v3-turbo-asr-fp16",
|
|
audio="audio.wav",
|
|
)
|
|
print(result.text)
|
|
```
|
|
|
|
### STT with Streaming
|
|
|
|
```python
|
|
from mlx_audio.stt import load
|
|
|
|
# VibeVoice-ASR streaming
|
|
model = load("mlx-community/VibeVoice-ASR-bf16")
|
|
for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
|
|
print(text, end="", flush=True)
|
|
|
|
# Parakeet streaming
|
|
model = load("mlx-community/parakeet-tdt-0.6b-v3")
|
|
for chunk in model.generate("long_audio.wav", stream=True):
|
|
print(chunk.text, end="", flush=True)
|
|
```
|
|
|
|
### Forced Alignment (Qwen3)
|
|
|
|
```python
|
|
from mlx_audio.stt import load
|
|
|
|
aligner = load("mlx-community/Qwen3-ForcedAligner-0.6B-8bit")
|
|
result = aligner.generate("audio.wav", text="I have a dream", language="English")
|
|
for item in result:
|
|
print(f"[{item.start_time:.2f}s - {item.end_time:.2f}s] {item.text}")
|
|
```
|
|
|
|
---
|
|
|
|
## REST API Server (OpenAI-compatible)
|
|
|
|
### Starting the Server
|
|
|
|
```bash
|
|
python -m mlx_audio.server [OPTIONS]
|
|
```
|
|
|
|
| Flag | Type | Default | Description |
|
|
|------|------|---------|-------------|
|
|
| `--host` | string | `localhost` | Server host |
|
|
| `--port` | int | `8000` | Server port |
|
|
| `--allowed-origins` | string | `*` | CORS allowed origins |
|
|
| `--workers` | int/float | `2` | Number of workers |
|
|
| `--reload` | flag | — | Enable auto-reload |
|
|
| `--start-ui` | flag | — | Launch Studio UI alongside API |
|
|
| `--log-dir` | string | `logs` | Directory for server logs |
|
|
|
|
### Endpoints
|
|
|
|
#### GET /v1/models
|
|
|
|
List available models.
|
|
|
|
```bash
|
|
curl http://localhost:8000/v1/models
|
|
```
|
|
|
|
#### POST /v1/models?model_name=\<name\>
|
|
|
|
Add a model to the server.
|
|
|
|
#### DELETE /v1/models?model_name=\<name\>
|
|
|
|
Remove a model from the server.
|
|
|
|
#### POST /v1/audio/speech
|
|
|
|
Generate speech from text.
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8000/v1/audio/speech \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "mlx-community/Kokoro-82M-bf16",
|
|
"input": "Hello, world!",
|
|
"voice": "af_heart",
|
|
"speed": 1.0,
|
|
"lang_code": "a",
|
|
"response_format": "mp3"
|
|
}' --output speech.mp3
|
|
```
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `model` | string | required | Model ID |
|
|
| `input` | string | required | Text to synthesize |
|
|
| `voice` | string | — | Voice preset |
|
|
| `speed` | float | 1.0 | Speech speed |
|
|
| `lang_code` | string | `a` | Language code |
|
|
| `ref_audio` | string | — | Reference audio path (voice cloning) |
|
|
| `ref_text` | string | — | Reference transcript |
|
|
| `response_format` | string | `mp3` | Output format |
|
|
| `stream` | bool | false | Enable streaming |
|
|
| `streaming_interval` | float | 2.0 | Streaming chunk interval |
|
|
| `temperature` | float | 0.7 | Sampling temperature |
|
|
| `top_p` | float | 0.95 | Nucleus sampling |
|
|
| `top_k` | int | 40 | Top-k sampling |
|
|
| `repetition_penalty` | float | 1.0 | Repetition penalty |
|
|
| `max_tokens` | int | 1200 | Maximum tokens |
|
|
| `gender` | string | `male` | Gender hint |
|
|
| `pitch` | float | 1.0 | Pitch adjustment |
|
|
| `instruct` | string | — | Instruction text |
|
|
|
|
#### POST /v1/audio/transcriptions
|
|
|
|
Transcribe an audio file (multipart/form-data).
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
|
-F file=@audio.wav \
|
|
-F model=mlx-community/whisper-large-v3-turbo-asr-fp16
|
|
```
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `file` | file | required | Audio file |
|
|
| `model` | string | required | Model ID |
|
|
| `language` | string | — | Language code |
|
|
| `max_tokens` | int | 1024 | Maximum tokens |
|
|
| `chunk_duration` | float | 30.0 | Chunk duration (seconds) |
|
|
| `stream` | bool | false | Enable streaming |
|
|
| `context` | string | — | Hotwords/context |
|
|
| `text` | string | — | Reference text |
|
|
| `verbose` | bool | false | Detailed output |
|
|
|
|
Response (NDJSON stream):
|
|
|
|
```json
|
|
{"text": "chunk text", "accumulated": "full text so far"}
|
|
```
|
|
|
|
#### WebSocket /v1/audio/transcriptions/realtime
|
|
|
|
Real-time transcription via WebSocket. Send initial config as JSON, then stream int16 PCM audio as binary frames.
|
|
|
|
```json
|
|
{
|
|
"model": "mlx-community/whisper-large-v3-turbo-asr-fp16",
|
|
"sample_rate": 16000,
|
|
"streaming": true
|
|
}
|
|
```
|
|
|
|
#### POST /v1/audio/separations
|
|
|
|
Separate audio sources (multipart/form-data).
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8000/v1/audio/separations \
|
|
-F file=@audio.wav \
|
|
-F model=mlx-community/sam-audio-large-fp16 \
|
|
-F description="speech"
|
|
```
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `file` | file | required | Audio file |
|
|
| `model` | string | `mlx-community/sam-audio-large-fp16` | Model ID |
|
|
| `description` | string | `speech` | Target description |
|
|
| `method` | string | `midpoint` | ODE method (`midpoint` or `euler`) |
|
|
| `steps` | int | 16 | ODE steps (2/4/8/16/32) |
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"target": "<base64 WAV>",
|
|
"residual": "<base64 WAV>",
|
|
"sample_rate": 44100
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Supported Models
|
|
|
|
### TTS Models
|
|
|
|
| Model | Languages | Notes |
|
|
|-------|-----------|-------|
|
|
| Kokoro | EN, JA, ZH, FR, ES, IT, PT, HI | Fast, high-quality multilingual |
|
|
| Qwen3-TTS | ZH, EN, JA, KO, + more | Voice design via instruction |
|
|
| CSM | EN | Voice cloning with reference audio |
|
|
| Dia | EN | Dialogue-focused |
|
|
| OuteTTS | EN | Efficient |
|
|
| Spark | EN, ZH | SparkTTS |
|
|
| Chatterbox | EN, ES, FR, DE, IT, PT, PL, TR, RU, NL, CS, AR, ZH, JA, HU, KO | Expressive multilingual |
|
|
| Soprano | EN | High-quality |
|
|
|
|
### STT Models
|
|
|
|
| Model | Languages | Notes |
|
|
|-------|-----------|-------|
|
|
| Whisper | 99+ languages | OpenAI's robust model |
|
|
| Qwen3-ASR | ZH, EN, JA, KO, + more | Alibaba multilingual |
|
|
| Qwen3-ForcedAligner | ZH, EN, JA, KO, + more | Word-level alignment |
|
|
| Parakeet | EN (v2), 25 EU languages (v3) | NVIDIA, high accuracy |
|
|
| Voxtral | Multiple | Mistral speech model |
|
|
| Voxtral Realtime | Multiple | 4B streaming STT |
|
|
| VibeVoice-ASR | Multiple | Microsoft 9B, supports diarization and context |
|
|
|
|
### Other Models
|
|
|
|
| Model | Type | Description |
|
|
|-------|------|-------------|
|
|
| Sortformer v1/v2.1 | VAD/Diarization | Speaker diarization (up to 4 speakers) |
|
|
| SAM-Audio | Separation | Text-guided source separation |
|
|
| Liquid2.5-Audio | STS | Speech/text-to-speech and STT |
|
|
| MossFormer2 SE | Enhancement | Speech enhancement / noise removal |
|
|
|
|
Models are available from `mlx-community` on HuggingFace with various quantization levels (3-bit through 8-bit and fp16/bf16).
|
|
|
|
## Source
|
|
|
|
- Repository: https://github.com/Blaizzy/mlx-audio
|
|
- HuggingFace: https://huggingface.co/mlx-community
|