Windows Computer Use: app launch/screenshots work, but click and press_key have no visible effect
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
Environment
Windows 11, native Windows agent environment.
Codex MSIX version: 26.901.6511.0.
Installed @oai/sky runtime version: 0.6.26.
Using the machine through ToDesk; a local-console comparison is not currently possible. ToDesk is NOT established as the cause.
Reproduction
- Initialize the installed runtime through the available Node tool: sky = (await import("@oai/sky")).sky.
- Use list_apps, launch_app, get_window and get_window_state to open Windows Calculator.
- click a numeric button using observed window-relative screenshot coordinates (also tested with screenshotId), or press_key({window, key:"2"}).
- Capture fresh state. The display does not change.
- Repeat a harmless File-menu click in Paint: launch and screenshots succeed, but the menu does not open.
Expected / actual
Expected: Calculator accepts input and Paint opens its menu.
Actual: input calls return without a reported error; screenshots show no corresponding UI change.
Manual user input DOES work and is reflected in subsequent screenshots.
include_text:true returned accessibility:null in Calculator.
Diagnostics
Recreating the Node control session did not resolve it.
Control runtime and Calculator are in Windows session 1.
Helper integrity level: medium; Calculator integrity level: low.
Ctrl/Alt/Shift and mouse buttons were not held at the sampled check.
The app's native-pipe startup log reports ready.
Two helper processes were identified as the main helper plus its system-cursor-manager child, not established duplicate controllers.
Earlier distinct setup failures
Old computer-use cache 26.527.31326 lacked tslib.es6.js. One missing file was copied from the installed runtime; module import then passed.
The old cache bootstrap returned "Computer Use native pipe is unavailable". Using the current runtime initialization above enabled app discovery/launch/capture, but input still failed.
A separate browser control attempt failed because its configured browser-service.mjs path points to a nonexistent old bundled/browser cache; the current runtime contains that service file. This has not been changed.
Please advise a supported way to obtain input-delivery diagnostics or whether a compatible runtime/update fixes this. No private screenshots, account identifiers or full logs are attached. This report was prepared by Codex from user-authorized diagnostics.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the @oai/sky 0.6.26 input path used after list_apps, launch_app, get_window, and get_window_state succeed, using the reported Calculator and Paint reproductions. Done means obtaining input-delivery diagnostics or confirming that a compatible runtime/update fixes click and press_key while preserving screenshots and app launch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js
- Domain
- desktop, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100