github / github/copilot-cli

Enhance /voice dictation to a Wispr Flow-class experience (AI cleanup, custom dictionary, voice commands, streaming)

未关闭
#3,806 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
area:input-keyboard
主要语言
Shell
星标
11.2k
派生
1.9k
平均合并
14 小时 16 分钟
30 天内合并 PR
6

描述

### Describe the feature or problem you'd like to solve

`/voice` (dictation transcription via Foundry Local) currently does **raw speech-to-text** — the literal transcript lands directly in the prompt. That's a great foundation, but it's well behind what dedicated AI dictation tools like **[Wispr Flow](https://wisprflow.ai/)** have made the baseline expectation.

In a terminal you're dictating code identifiers, file paths, shell commands, and multi-sentence prompts, so raw STT produces messy output — fillers ("um", "uh", "like"), run-on sentences with no punctuation, and mangled jargon (e.g. "bolt module pa portal" instead of `bolt.module.paportal`, or scrambled GUIDs) — which you then fix by hand. That hand-cleanup defeats the whole speed advantage of talking instead of typing.

The ask: close the gap between `/voice` and a **Wispr Flow–class** dictation experience. The CLI has an edge standalone dictation apps don't — **there's already an LLM in the loop** that can clean up and structure the transcript locally/in-session.

### Proposed solution

Layer the following on top of the existing Foundry Local STT pipeline (enhancement, not a rewrite):

1. **AI transcript cleanup pass (highest value).** Route the raw STT output through a fast model pass before it lands in the prompt: strip fillers, add punctuation & capitalization, fix sentence boundaries, and honor self-corrections ("set the timeout to two — no, three seconds" → "three seconds"). The session already has a model in the loop, so this is mostly wiring. Make it a toggle (`/voice cleanup on|off`) for anyone who wants a verbatim transcript.

2. **Custom dictionary / bias terms.** Let users register domain terms, command names, and identifiers so STT stops mangling them (`kubectl`, `pnpm`, `OAuth`, repo module names, product names). Auto-seed it from context the CLI already has — the open files, the repo, recent commands, and the conversation — the same way community tooling (the `mic` helper) builds a Whisper bias prompt from the live conversation. Persist learned corrections.

3. **Voice commands / command mode.** Recognize a small set of spoken control words distinct from dictated text: "submit"/"send", "new line", "scratch that" (delete the last utterance), "clear", "cancel", "code block". This is Wispr's Command Mode adapted to the CLI prompt.

4. **Streaming partial transcripts + low latency.** Show interim words as you speak (near-real-time insertion) instead of committing only on end-of-utterance, with VAD-based auto-stop on silence. Makes long prompts feel responsive.

5. **Push-to-talk + hands-free modes.** A held-key push-to-talk plus a continuous/VAD hands-free mode — both common patterns for terminal dictation.

6. **Languages & code-switching.** Surface Foundry Local's multilingual models and allow mixed-language dictation, matching Wispr's 100+ language coverage.

**Benefit:** voice becomes a genuinely faster input path for prompts and code rather than a novelty — and it's a differentiator GitHub is well-positioned for, because the CLI can do the AI cleanup in-loop that Wispr does as a cloud service.

### Example prompts or workflows

- **Dictating a prompt:** "um, refactor the auth module to use, like, JWT instead of sessions, and add tests" → inserts **"Refactor the auth module to use JWT instead of sessions, and add tests."** (fillers gone, punctuated).
- **Self-correction:** "rename the variable to user I-D, no — make it userId camelCase" → inserts **`userId`**.
- **Jargon via custom dictionary:** "run bolt module paportal upload" resolves to the registered identifier **`bolt.module.paportal`** instead of "bolt module pa portal".
- **Command mode (hands-free):** dictate a long multi-line prompt, say "new line" between thoughts, then "submit" to send — no keyboard.
- **Scratch that:** "create a new branch called feature slash voice… scratch that… feature/voice-enhancements" leaves only the corrected text.

### Additional context

- Builds directly on the existing `/voice` (Foundry Local) pipeline — this is an **enhancement, not a rewrite**, and most of the "AI" pieces reuse the model already in the session.
- Related issues: **#3635** (original native `/voice` request, now shipped) and **#3636** (voice catalog load failures). This request is about the **quality/feature depth** of dictation once it's running.
- Reference bar: Wispr Flow's feature set — AI auto-punctuation/formatting, filler removal, personal & team custom dictionaries, command mode, near-real-time latency, and 100+ languages.

贡献指南

打开贡献指南

调研方向

首先跟踪现有的 /voice Foundry Local pipeline,并查看 issue #3635 了解已发布的实现,查看 #3636 了解相关故障。完成这项工作需要定义并实现所请求的清理、词典、命令模式、流式处理、输入模式和语言增强功能,并对由此产生的听写工作流进行验证。

由索引模型根据 Issue 内容生成。

评估

技术栈
shell
领域
ai, cli
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
冷清
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。