[Windows][Computer Use] VMConnect guest clicks report done without visible action
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 2.1k
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
Short summary
On Windows, Computer Use coordinate clicks on a Hyper-V VMConnect guest surface repeatedly returned done without the intended guest UI action occurring.
Affected version or release
App: GitHub Copilot 1.1.17; bundled computer-use server 0.1.88.
What happened?
On Windows, Computer Use coordinate clicks on a Hyper-V VMConnect guest surface repeatedly returned done without the intended guest UI action occurring.
It seems a mismatch between background click delivery and VMConnect guest input. Permission was granted, but ordinary click remained on the background path.
My workaround is to use non-zero drag to replace click, which uses the existing consented foreground path and delivered a visible guest action.
Steps to reproduce
- Open a local Windows Server guest in Hyper-V VMConnect and sign in manually.
- In an Interactive Copilot session, grant native foreground Computer Use permission, prompt "Open Server Manager in Hyper-V VM [VMName] with computer use".
- Ordinary clicks can appear successful to the agent while the guest UI stays unchanged. Agents then retry, type into the wrong state, or incorrectly claim success. After several retry, the agent reports: "I opened Start and clicked that icon, but the VM hasn’t responded to the click yet. Server Manager is still not open."
Expected behavior
For VMConnect guest surfaces that require foreground mouse input, a normal click should work.
If the engine cannot establish appropriate delivery, return an actionable unsupported result instead of implying the guest action succeeded.
Do not require users or agents to fabricate a drag to click a menu.
Additional context
Workaround:
- For VMConnect guest UI, replace an ordinary single left
computer-use-clickat(x, y)with one-pixel foreground drag:
{
"app": "<same target app>",
"window": "<same numeric window id>",
"from_x": "<x>",
"from_y": "<y>",
"to_x": "<x + 1>",
"to_y": "<y>",
"capture_mode": "text"
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Windows Computer Use handling for ordinary computer-use-click delivery and compare it with the foreground drag path described in the workaround. Reproduce the issue against a Hyper-V VMConnect guest, then verify that a normal click visibly acts or returns an actionable unsupported result rather than reporting done.
Written by the indexing model from the issue text.
Assessment
- Domain
- desktop, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100