Feature request: let users add or swap local STT and TTS in conversation mode
- Dominant language
- TypeScript
- Stars
- 909
- Forks
- 116
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 156
Description
### Before filing
- [x] I searched [open and closed issues](https://github.com/block/berd/issues?q=is%3Aissue) for duplicates.
- [x] I am using a recent [release](https://github.com/block/berd/releases) and this doesn't already exist.
- [x] This is one request, not several bundled together.
### Closest existing issue
none found (searched voice, parakeet, pocket, stt, tts, conversation, nemotron, speech). Closest in spirit is #110 (closed ACP agent catalog). Different layer.
### Is this new, or an improvement?
New capability — Berd can't do this at all today
### The problem, in your terms
I use Berd conversation mode as the spoken loop into a live agent session (Grok 4.6 via Goose). That loop is the product I want: talk, the agent hears it in the current chat, the agent speaks back, I can barge in.
What I can adjust today:
- the agent and chat model (the brain)
- which Pocket speaker to use (Azelma, Mary, and the rest)
- playback speed
- Remove model for Pocket or Parakeet, then download that same pair again
What I cannot adjust:
- the speech recognizer
- the speech synthesizer family
- any second local STT or TTS pack
I wanted to try a streaming speech model (NVIDIA Nemotron speech streaming is the one I had in hand) inside the same conversation mode. Settings only offered Remove. There is no Add, no catalog, no path picker.
I then read the 0.6.2 source. Conversation STT is a pinned Offline CTC Parakeet 110M worker (`native_voice.rs` loads `model.int8.onnx` through sherpa-onnx `nemo_ctc`). Install and remove only know two kinds: Pocket and Parakeet (`pocket_voice.rs`, `VoiceModelKind`). That matches the UI. The missing piece is not a hidden button. The catalog is closed.
### What you do today
I keep using the shipped Parakeet plus Pocket pair in Berd, because that is the only path that injects transcripts into the live Goose session and speaks deltas with barge-in and the speaker echo guard.
Separately I built a local sidecar that reuses the same Parakeet weights, then calls Grok headless, then macOS `say`. It works as a walkie-talkie. It is not conversation mode. It does not steer the open Berd session, it does not stream assistant audio from Pocket, and each turn pays a full new-process cost (about 20 seconds in my smoke).
Forking Berd to hardcode a third model would work once and then fight every Berd release. I am not treating a private fork as the daily app.
### What you'd like to see
A supported way to register another local STT and/or TTS backend for conversation mode, without editing Berd source.
Minimum that would unblock me:
- Settings next to Remove: Add, or pick from a small list, or point at a local model directory plus a recognizer kind
- Conversation mode uses that backend through the same events it already has (user transcript, speaking or idle, error, clean shutdown)
- Install stays checksum-pinned, same as today's Parakeet and Pocket packs
- The shipped Parakeet plus Pocket pair remains the default
Nemotron speech streaming is one backend I would try first. I am not asking you to ship or maintain that model.
### Why this belongs in Berd itself
Conversation mode is a Berd runtime, not a Goose skill. A skill, agent, extension, or automation cannot replace the Tauri STT worker, the Pocket playback path, the microphone claim, or the Settings voice install or remove surface.
`berdctl` can create and send sessions. It cannot swap the recognizer that feeds those sessions.
This is the same class of closed catalog as #110, on the ears and mouth instead of the agent process.
### Non-goals
- No Nemotron-specific integration, and no request that Berd download or vendor NVIDIA speech models
- No cloud STT or TTS, no API keys, no Grok Voice SKU
- No change to the agent or chat-model picker (the brain already switches)
- No replacement of Parakeet or Pocket as the default
- No plugin marketplace, no unsigned model URLs, no request to accept an outside PR
- No change to telemetry defaults
### Alternatives you considered
- Use the Remove button, then hope Add appears: Remove only retires the pinned pack. Reinstall puts the same two kinds back.
- Drop a different ONNX into `~/Library/Application Support/xyz.block.berd/pocket-tts/native-voice-v2/stt/`: checksums reject it. The worker is Offline CTC, not a streaming RNNT graph.
- Skill or automation: runs after an agent is already selected. It cannot own the mic pipeline.
- Local sidecar (what I run now): useful for eval, missing conversation mode.
- Fork Berd and hardcode a third `VoiceModelKind`: works until the next release. CONTRIBUTING says not to send that patch.
A small generic local-backend registration point is cleaner than one-off model integrations.
### Mockups, prior art, or other context
Verified on Berd 0.6.2, macOS, Settings → About. Native voice pack is installed:
- STT: `sherpa-onnx-nemo-parakeet_tdt_ctc_110m-en-36000-int8` as `native-voice-v2/stt/model.int8.onnx`
- TTS: Pocket ONNX plus the Kyutai speaker wavs; selected voice Azelma; speed 1.5
Source read on `block/berd` main:
- `src-tauri/src/commands/native_voice.rs`
- `src-tauri/src/commands/pocket_voice.rs`
- `src/features/voice-conversation/`
Conversation mode today:
```
mic
-> pinned Parakeet Offline CTC
-> transcript events into the live Goose session
-> agent tokens
-> pinned Pocket TTS
-> speakers, with capture suppressed while it talks
```
The ask is a supported third box on the STT and TTS steps, not a new conversation product.
Contributor guide
Research direction
Start by tracing installation and removal in src-tauri/src/commands/native_voice.rs and pocket_voice.rs, then follow the conversation flow in src/features/voice-conversation/. Identify how model kinds, settings, and transcript or playback events connect; done means an additional local backend can be registered and used without changing the shipped Parakeet and Pocket defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, tauri, typescript
- Domain
- audio-video-rtc, desktop
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100