github / github/copilot-cli

Enhance /voice dictation to a Wispr Flow-class experience (AI cleanup, custom dictionary, voice commands, streaming)

Đang mở
#3,806 0 bình luận 1 reaction 0 người được giao Xem trên GitHub
area:input-keyboard
Ngôn ngữ chính
Shell
Star
11.2k
Fork
1.9k
Merge trung bình
14 giờ 16 phút
Pull request đã merge (30 ngày)
6

Mô tả

### Describe the feature or problem you'd like to solve

`/voice` (dictation transcription via Foundry Local) currently does **raw speech-to-text** — the literal transcript lands directly in the prompt. That's a great foundation, but it's well behind what dedicated AI dictation tools like **[Wispr Flow](https://wisprflow.ai/)** have made the baseline expectation.

In a terminal you're dictating code identifiers, file paths, shell commands, and multi-sentence prompts, so raw STT produces messy output — fillers ("um", "uh", "like"), run-on sentences with no punctuation, and mangled jargon (e.g. "bolt module pa portal" instead of `bolt.module.paportal`, or scrambled GUIDs) — which you then fix by hand. That hand-cleanup defeats the whole speed advantage of talking instead of typing.

The ask: close the gap between `/voice` and a **Wispr Flow–class** dictation experience. The CLI has an edge standalone dictation apps don't — **there's already an LLM in the loop** that can clean up and structure the transcript locally/in-session.

### Proposed solution

Layer the following on top of the existing Foundry Local STT pipeline (enhancement, not a rewrite):

1. **AI transcript cleanup pass (highest value).** Route the raw STT output through a fast model pass before it lands in the prompt: strip fillers, add punctuation & capitalization, fix sentence boundaries, and honor self-corrections ("set the timeout to two — no, three seconds" → "three seconds"). The session already has a model in the loop, so this is mostly wiring. Make it a toggle (`/voice cleanup on|off`) for anyone who wants a verbatim transcript.

2. **Custom dictionary / bias terms.** Let users register domain terms, command names, and identifiers so STT stops mangling them (`kubectl`, `pnpm`, `OAuth`, repo module names, product names). Auto-seed it from context the CLI already has — the open files, the repo, recent commands, and the conversation — the same way community tooling (the `mic` helper) builds a Whisper bias prompt from the live conversation. Persist learned corrections.

3. **Voice commands / command mode.** Recognize a small set of spoken control words distinct from dictated text: "submit"/"send", "new line", "scratch that" (delete the last utterance), "clear", "cancel", "code block". This is Wispr's Command Mode adapted to the CLI prompt.

4. **Streaming partial transcripts + low latency.** Show interim words as you speak (near-real-time insertion) instead of committing only on end-of-utterance, with VAD-based auto-stop on silence. Makes long prompts feel responsive.

5. **Push-to-talk + hands-free modes.** A held-key push-to-talk plus a continuous/VAD hands-free mode — both common patterns for terminal dictation.

6. **Languages & code-switching.** Surface Foundry Local's multilingual models and allow mixed-language dictation, matching Wispr's 100+ language coverage.

**Benefit:** voice becomes a genuinely faster input path for prompts and code rather than a novelty — and it's a differentiator GitHub is well-positioned for, because the CLI can do the AI cleanup in-loop that Wispr does as a cloud service.

### Example prompts or workflows

- **Dictating a prompt:** "um, refactor the auth module to use, like, JWT instead of sessions, and add tests" → inserts **"Refactor the auth module to use JWT instead of sessions, and add tests."** (fillers gone, punctuated).
- **Self-correction:** "rename the variable to user I-D, no — make it userId camelCase" → inserts **`userId`**.
- **Jargon via custom dictionary:** "run bolt module paportal upload" resolves to the registered identifier **`bolt.module.paportal`** instead of "bolt module pa portal".
- **Command mode (hands-free):** dictate a long multi-line prompt, say "new line" between thoughts, then "submit" to send — no keyboard.
- **Scratch that:** "create a new branch called feature slash voice… scratch that… feature/voice-enhancements" leaves only the corrected text.

### Additional context

- Builds directly on the existing `/voice` (Foundry Local) pipeline — this is an **enhancement, not a rewrite**, and most of the "AI" pieces reuse the model already in the session.
- Related issues: **#3635** (original native `/voice` request, now shipped) and **#3636** (voice catalog load failures). This request is about the **quality/feature depth** of dictation once it's running.
- Reference bar: Wispr Flow's feature set — AI auto-punctuation/formatting, filler removal, personal & team custom dictionaries, command mode, near-real-time latency, and 100+ languages.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start by tracing the existing /voice Foundry Local pipeline and review issue #3635 for the shipped implementation and #3636 for related failures. Done would require defining and implementing the requested cleanup, dictionary, command-mode, streaming, input-mode, and language enhancements, with validation for the resulting dictation workflows.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
shell
Lĩnh vực
ai, cli
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.