github / github/app

Optimize voice dictation prompts before sending to expensive models

未关闭
#943 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Requests and ideas
主要语言
没有语言数据
星标
2.1k
派生
153
PR 合并指标
30 天内没有已合并 PR

描述

🤖 *AI response on behalf of Casey*

### Feature summary

Add an option for GitHub Copilot app voice dictation to rewrite the transcribed prompt into a concise, agent/LLM-friendly form before submitting it to the selected model.

This is broader than simple filler-word removal: the goal is to transform conversational dictated speech into a clearer task prompt that removes duplication, tightens wording, and preserves the user intent.

### What problem are you trying to solve?

When using voice-to-text, users naturally speak conversationally. They pause, repeat themselves, restate requirements, add verbal scaffolding, and use phrases that help them think aloud but do not help the agent act.

That can lead to prompts like:

> Um, I think what we need to do is, uh, look at the parser tests, and maybe, you know, update them for the new behavior. I guess what I'm saying is, make sure the agent understands that the parser should preserve quoted text but normalize whitespace elsewhere.

A more useful prompt for an agent would be:

> Update the parser tests for the new behavior: quoted text should be preserved exactly, while whitespace outside quoted text should be normalized.

The current conversational transcript can make the request harder for the agent to follow and can waste input tokens. This matters more when the target model is a large or expensive model, where every unnecessary token has higher cost.

### Proposed solution

Provide an optional voice-dictation post-processing mode that rewrites the raw transcript before submission. It could use a small, cheap, or local model rather than the selected high-capability model.

The rewrite step should:

- Remove repeated ideas, filler language, and conversational scaffolding.
- Convert spoken, meandering instructions into a direct agent task.
- Preserve technical terms, file names, code identifiers, quoted text, and explicit constraints.
- Keep the user's meaning intact rather than inventing new requirements.
- Show the rewritten prompt before send, or make the behavior configurable for users who prefer exact transcription.

### Model selection and local inference

When choosing the model for this rewrite step, the app should consider that small, capable local/on-device models are likely to become practical for this kind of prompt-cleanup task. If the rewrite can run locally, users may get the clarity and token-saving benefits without paying for a separate hosted inference request.

It may also be worth considering whether the voice pipeline could eventually combine dictation and lightweight summarization/rewriting in one model or ensemble, so the transcript is captured and normalized into an agent-ready prompt as part of the same flow.
### Workflow impact

This would make voice input more effective for agentic workflows, especially for long dictated prompts. It should improve model comprehension, reduce unnecessary context, and lower token usage when prompts are ultimately sent to larger/more expensive models.

### Related context

Related app feature request for simpler filler-word cleanup: https://github.com/github/app/issues/942

Related VS Code backlog request for speech-input post-processing: https://github.com/microsoft/vscode/issues/315508

贡献指南

打开贡献指南

调研方向

未指定实现文件、测试或入口点。首先梳理应用的语音听写提交流程,并查看 github/app#942 和 microsoft/vscode#315508 中的相关请求。完成后应包括一个可选的重写步骤,该步骤保留技术细节和意图,支持审查或配置路径,并考虑小型或本地模型。

由索引模型根据 Issue 内容生成。

评估

技术栈
github
领域
ai, desktop
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
冷清
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。