github / github/app

Associate pick-and-polish selections with voice transcription references

Ouverte
#2,190 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Aucune donnée de langage
Étoiles
2.1k
Forks
153
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

### What problem are you trying to solve?

Pick and polish works well for attaching a selected UI element to a typed prompt, but it does not support naturally referencing multiple elements while dictating a request.

For example, while speaking I want to select different elements and say:

> Make **[click] this one** larger, remove **[click] this one**, and move **[click] this one** to the left.

Currently, the selections are treated as general attachments rather than being associated with the exact moment or phrase in the voice transcript. The agent therefore cannot reliably determine which instruction belongs to which selected element.

### Proposed solution

Integrate pick-and-polish selections with voice transcription so each selection is linked to its position in the transcript.

When the user clicks an element while speaking, the transcript could insert a visible reference token, such as:

> Make **[Header]** larger, remove **[Secondary button]**, and move **[Image card]** to the left.

The agent should receive both the selected element's context and its relationship to the corresponding spoken instruction. Multiple selections should be supported within one continuous voice request.

### Workflow impact

This would make voice-driven UI editing faster and more natural. Users could describe several changes in one pass without stopping to submit a separate prompt for every element or verbally identifying elements that are already selectable on screen.

It would also create a stronger multimodal workflow by combining visual selection, speech, and agent context.

### Installation context

GitHub Copilot app on macOS, using browser/app preview and voice transcription.

### Additional context

Pick and polish already provides the core selection mechanism. The request is to preserve the timing and semantic relationship between each selection and the corresponding part of a voice-transcribed prompt.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start by tracing the existing pick-and-polish selection flow and voice transcription flow in the GitHub Copilot app; the issue names no files, tests, or entry points. Define how selections become visible transcript references, preserve each selection's timing and context, support multiple selections in one request, and verify that each spoken instruction is associated with the intended element.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
macos
Domaine
desktop
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Calme
Clarté
À clarifier
Accessibilité débutants
30/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.