Add an interactive "human-assist" mode for GUI/emulator actions in Copilot Agent Mode (VS Code)
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
**Summary**
When Copilot's agent mode is performing actions in a GUI context it doesn't have reliable visual/coordinate access to (e.g., an Android emulator, a desktop app, a browser window opened during a task), it currently has to either (a) burn time and tokens inferring element positions and attempting the action itself, or (b) stop and ask the user to perform the action and report back manually. Both are worse than they need to be. This issue proposes a lightweight, opt-in "assist mode" that lets the user perform the physical action while Copilot keeps driving the task.
**Motivation**
The underlying capability already exists, Copilot can render prompts/confirmations in the chat panel and interpret simple user responses (it already does this for tool-approval steps). What's missing is a structured handoff mechanism for GUI actions specifically, so the human can act as Copilot's "hands" for the parts it can't reliably do itself, without derailing the session into manual back-and-forth.
**Example use case**
I asked Copilot agent mode to test an Android app in an emulator launched from VS Code. After starting the emulator, Copilot had no reliable way to identify exact button coordinates, so it spent significant time and tokens guessing, clicking, and re-checking screenshots to find the right UI element.
Instead, the flow could be:
1. After launching the emulator (or entering any GUI-driven step), Copilot asks a single yes/no question in the chat panel: *"Would you like to assist with GUI interactions for this task?"*
2. **If "No"**: Copilot proceeds exactly as it does today, working out coordinates/actions on its own.
3. **If "Yes"**: whenever Copilot needs to take a GUI action, instead of attempting it blindly, it presents a small structured prompt in the chat/inline UI describing the action it wants performed, e.g.:
- "Please tap the 'Login' button in the top-right corner."
- Response options: **Done** (confirms the action was taken) or a free-text field for **"I don't understand / clarify"** if the instruction is unclear.
4. Copilot waits for the response, then continues the task using that confirmation as ground truth, instead of re-verifying via screenshots or guesswork.
**Proposed solution**
- Add a toggleable "assist mode" that can be offered at the start of any GUI-interaction-heavy agent task (emulator testing, desktop app automation, browser-driven tasks, etc.).
- When enabled, replace autonomous GUI action attempts with a simple confirm/clarify prompt per action, using the existing chat panel UI (similar to current tool-call approval prompts).
- When disabled (or declined), fall back to current autonomous agent behavior.
- Should apply within Copilot's agent mode in VS Code, and ideally extend to any GUI/emulator-adjacent tooling Copilot can invoke (e.g., Android emulator, Simulator, browser automation via Playwright/Puppeteer extensions).
**Expected benefit**
- Saves tokens and time on tasks where Copilot has to laboriously infer UI coordinates or state.
- Increases reliability, since a human confirms the action instead of Copilot assuming success.
- Keeps the human in the loop without forcing them to fully take over the task or narrate results back manually.
- Fits naturally alongside existing agent-mode UX patterns (approval prompts, todo lists, inline confirmations).
Contributor guide
Assessment
This issue has not been assessed yet.