github / github/app

[Windows][Computer Use] VMConnect guest clicks report done without visible action

Open
#3,764 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
No language data
Stars
2.1k
Forks
157
PR merge metrics
No merged PRs in 30d

Description

Short summary

On Windows, Computer Use coordinate clicks on a Hyper-V VMConnect guest surface repeatedly returned done without the intended guest UI action occurring.

Affected version or release

App: GitHub Copilot 1.1.17; bundled computer-use server 0.1.88.

What happened?

On Windows, Computer Use coordinate clicks on a Hyper-V VMConnect guest surface repeatedly returned done without the intended guest UI action occurring.
It seems a mismatch between background click delivery and VMConnect guest input. Permission was granted, but ordinary click remained on the background path.
My workaround is to use non-zero drag to replace click, which uses the existing consented foreground path and delivered a visible guest action.

Steps to reproduce
  1. Open a local Windows Server guest in Hyper-V VMConnect and sign in manually.
  2. In an Interactive Copilot session, grant native foreground Computer Use permission, prompt "Open Server Manager in Hyper-V VM [VMName] with computer use".
  3. Ordinary clicks can appear successful to the agent while the guest UI stays unchanged. Agents then retry, type into the wrong state, or incorrectly claim success. After several retry, the agent reports: "I opened Start and clicked that icon, but the VM hasn’t responded to the click yet. Server Manager is still not open."
Expected behavior

For VMConnect guest surfaces that require foreground mouse input, a normal click should work.
If the engine cannot establish appropriate delivery, return an actionable unsupported result instead of implying the guest action succeeded.
Do not require users or agents to fabricate a drag to click a menu.

Additional context

Workaround:

  1. For VMConnect guest UI, replace an ordinary single left computer-use-click at (x, y) with one-pixel foreground drag:
{
  "app": "<same target app>",
  "window": "<same numeric window id>",
  "from_x": "<x>",
  "from_y": "<y>",
  "to_x": "<x + 1>",
  "to_y": "<y>",
  "capture_mode": "text"
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Windows Computer Use handling for ordinary computer-use-click delivery and compare it with the foreground drag path described in the workaround. Reproduce the issue against a Hyper-V VMConnect guest, then verify that a normal click visibly acts or returns an actionable unsupported result rather than reporting done.

Written by the indexing model from the issue text.

Assessment

Domain
desktop, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.