Enhance /voice dictation to a Wispr Flow-class experience (AI cleanup, custom dictionary, voice commands, streaming)
- Lingua principale
- Shell
- Stelle
- 11.2k
- Fork
- 1.9k
- Merge medio
- 14h 16m
- PR unite (30g)
- 6
Descrizione
### Describe the feature or problem you'd like to solve
`/voice` (dictation transcription via Foundry Local) currently does **raw speech-to-text** — the literal transcript lands directly in the prompt. That's a great foundation, but it's well behind what dedicated AI dictation tools like **[Wispr Flow](https://wisprflow.ai/)** have made the baseline expectation.
In a terminal you're dictating code identifiers, file paths, shell commands, and multi-sentence prompts, so raw STT produces messy output — fillers ("um", "uh", "like"), run-on sentences with no punctuation, and mangled jargon (e.g. "bolt module pa portal" instead of `bolt.module.paportal`, or scrambled GUIDs) — which you then fix by hand. That hand-cleanup defeats the whole speed advantage of talking instead of typing.
The ask: close the gap between `/voice` and a **Wispr Flow–class** dictation experience. The CLI has an edge standalone dictation apps don't — **there's already an LLM in the loop** that can clean up and structure the transcript locally/in-session.
### Proposed solution
Layer the following on top of the existing Foundry Local STT pipeline (enhancement, not a rewrite):
1. **AI transcript cleanup pass (highest value).** Route the raw STT output through a fast model pass before it lands in the prompt: strip fillers, add punctuation & capitalization, fix sentence boundaries, and honor self-corrections ("set the timeout to two — no, three seconds" → "three seconds"). The session already has a model in the loop, so this is mostly wiring. Make it a toggle (`/voice cleanup on|off`) for anyone who wants a verbatim transcript.
2. **Custom dictionary / bias terms.** Let users register domain terms, command names, and identifiers so STT stops mangling them (`kubectl`, `pnpm`, `OAuth`, repo module names, product names). Auto-seed it from context the CLI already has — the open files, the repo, recent commands, and the conversation — the same way community tooling (the `mic` helper) builds a Whisper bias prompt from the live conversation. Persist learned corrections.
3. **Voice commands / command mode.** Recognize a small set of spoken control words distinct from dictated text: "submit"/"send", "new line", "scratch that" (delete the last utterance), "clear", "cancel", "code block". This is Wispr's Command Mode adapted to the CLI prompt.
4. **Streaming partial transcripts + low latency.** Show interim words as you speak (near-real-time insertion) instead of committing only on end-of-utterance, with VAD-based auto-stop on silence. Makes long prompts feel responsive.
5. **Push-to-talk + hands-free modes.** A held-key push-to-talk plus a continuous/VAD hands-free mode — both common patterns for terminal dictation.
6. **Languages & code-switching.** Surface Foundry Local's multilingual models and allow mixed-language dictation, matching Wispr's 100+ language coverage.
**Benefit:** voice becomes a genuinely faster input path for prompts and code rather than a novelty — and it's a differentiator GitHub is well-positioned for, because the CLI can do the AI cleanup in-loop that Wispr does as a cloud service.
### Example prompts or workflows
- **Dictating a prompt:** "um, refactor the auth module to use, like, JWT instead of sessions, and add tests" → inserts **"Refactor the auth module to use JWT instead of sessions, and add tests."** (fillers gone, punctuated).
- **Self-correction:** "rename the variable to user I-D, no — make it userId camelCase" → inserts **`userId`**.
- **Jargon via custom dictionary:** "run bolt module paportal upload" resolves to the registered identifier **`bolt.module.paportal`** instead of "bolt module pa portal".
- **Command mode (hands-free):** dictate a long multi-line prompt, say "new line" between thoughts, then "submit" to send — no keyboard.
- **Scratch that:** "create a new branch called feature slash voice… scratch that… feature/voice-enhancements" leaves only the corrected text.
### Additional context
- Builds directly on the existing `/voice` (Foundry Local) pipeline — this is an **enhancement, not a rewrite**, and most of the "AI" pieces reuse the model already in the session.
- Related issues: **#3635** (original native `/voice` request, now shipped) and **#3636** (voice catalog load failures). This request is about the **quality/feature depth** of dictation once it's running.
- Reference bar: Wispr Flow's feature set — AI auto-punctuation/formatting, filler removal, personal & team custom dictionaries, command mode, near-real-time latency, and 100+ languages.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start by tracing the existing /voice Foundry Local pipeline and review issue #3635 for the shipped implementation and #3636 for related failures. Done would require defining and implementing the requested cleanup, dictionary, command-mode, streaming, input-mode, and language enhancements, with validation for the resulting dictation workflows.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- shell
- Ambito
- ai, cli
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100