feat(huddle): pluggable STT/TTS backends — let self-hosted communities point voice at their own endpoints
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Problem
Huddle voice is hardwired to two bundled on-device models:
- STT: sherpa-onnx Parakeet TDT-CTC 110M (`desktop/src-tauri/src/huddle/stt.rs`) — English-only
- TTS: pocket-tts (`desktop/src-tauri/src/huddle/pocket.rs`)
For English-speaking teams that's a great zero-config default. But for non-English and code-switching teams the STT output is garbled (#2478 reports this for Korean), which poisons the whole agent-in-huddle experience: the agent "hears" garbage, replies to garbage, and the reply is spoken by a small TTS voice. Users experience all of this as "voice quality is bad" when the real issue is the fixed models.
We run a self-hosted relay for a Malaysian team (English/Malay/Mandarin code-switching in a single sentence is normal here). We already operate a local ASR service on our LAN (Qwen3-ASR behind a simple HTTP endpoint) that handles code-switching well — but there is no way to point huddles at it: no config, no env var, no endpoint override. The models are compiled into the pipeline.
## Proposal
Make the huddle STT (and ideally TTS) backend pluggable, keeping the bundled models as the zero-config default:
- A per-community or per-app setting like `huddle.stt.endpoint` accepting an OpenAI-compatible `POST /v1/audio/transcriptions` URL (the de-facto standard most self-hosted ASR servers — whisper.cpp servers, speaches, vLLM audio, custom — already expose).
- The existing worker already produces 16 kHz mono PCM at utterance boundaries (VAD-segmented), so the integration point is narrow: ship the utterance to the endpoint instead of sherpa-onnx when configured, with the local model as fallback on error.
- Same shape for TTS if feasible: `huddle.tts.endpoint` accepting `POST /v1/audio/speech`.
## Why this fits Buzz
Self-hosting your relay but not your voice models is an odd asymmetry — the community that cares enough to run its own infrastructure is exactly the community whose language needs won't be covered by a single bundled English model. A pluggable endpoint keeps the default experience untouched, adds no new bundled dependencies, and turns #2478 from a model-quality bug into a solved configuration.
Happy to test against our self-hosted relay + LAN ASR and report back if this lands behind a flag.
Contributor guide
Assessment
This issue has not been assessed yet.