github / github/app

Associate pick-and-polish selections with voice transcription references

未关闭
#2,190 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
没有语言数据
星标
2.1k
派生
153
PR 合并指标
30 天内没有已合并 PR

描述

### What problem are you trying to solve?

Pick and polish works well for attaching a selected UI element to a typed prompt, but it does not support naturally referencing multiple elements while dictating a request.

For example, while speaking I want to select different elements and say:

> Make **[click] this one** larger, remove **[click] this one**, and move **[click] this one** to the left.

Currently, the selections are treated as general attachments rather than being associated with the exact moment or phrase in the voice transcript. The agent therefore cannot reliably determine which instruction belongs to which selected element.

### Proposed solution

Integrate pick-and-polish selections with voice transcription so each selection is linked to its position in the transcript.

When the user clicks an element while speaking, the transcript could insert a visible reference token, such as:

> Make **[Header]** larger, remove **[Secondary button]**, and move **[Image card]** to the left.

The agent should receive both the selected element's context and its relationship to the corresponding spoken instruction. Multiple selections should be supported within one continuous voice request.

### Workflow impact

This would make voice-driven UI editing faster and more natural. Users could describe several changes in one pass without stopping to submit a separate prompt for every element or verbally identifying elements that are already selectable on screen.

It would also create a stronger multimodal workflow by combining visual selection, speech, and agent context.

### Installation context

GitHub Copilot app on macOS, using browser/app preview and voice transcription.

### Additional context

Pick and polish already provides the core selection mechanism. The request is to preserve the timing and semantic relationship between each selection and the corresponding part of a voice-transcribed prompt.

贡献指南

打开贡献指南

调研方向

首先跟踪 GitHub Copilot 应用中现有的选择与润色流程以及语音转录流程;该 issue 未指定任何文件、测试或入口点。定义选择如何成为可见的转录引用,保留每个选择的时间和上下文,支持在一个请求中包含多个选择,并验证每条口述指令都与预期元素相关联。

由索引模型根据 Issue 内容生成。

评估

技术栈
macos
领域
desktop
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
冷清
描述清晰度
需要澄清
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。