openai / openai/codex

Windows Computer Use: app launch/screenshots work, but click and press_key have no visible effect

Open
#44,050 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app bug computer-use windows-os
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Environment

Windows 11, native Windows agent environment.
Codex MSIX version: 26.901.6511.0.
Installed @oai/sky runtime version: 0.6.26.
Using the machine through ToDesk; a local-console comparison is not currently possible. ToDesk is NOT established as the cause.

Reproduction
  1. Initialize the installed runtime through the available Node tool: sky = (await import("@oai/sky")).sky.
  2. Use list_apps, launch_app, get_window and get_window_state to open Windows Calculator.
  3. click a numeric button using observed window-relative screenshot coordinates (also tested with screenshotId), or press_key({window, key:"2"}).
  4. Capture fresh state. The display does not change.
  5. Repeat a harmless File-menu click in Paint: launch and screenshots succeed, but the menu does not open.
Expected / actual

Expected: Calculator accepts input and Paint opens its menu.
Actual: input calls return without a reported error; screenshots show no corresponding UI change.
Manual user input DOES work and is reflected in subsequent screenshots.
include_text:true returned accessibility:null in Calculator.

Diagnostics

Recreating the Node control session did not resolve it.
Control runtime and Calculator are in Windows session 1.
Helper integrity level: medium; Calculator integrity level: low.
Ctrl/Alt/Shift and mouse buttons were not held at the sampled check.
The app's native-pipe startup log reports ready.
Two helper processes were identified as the main helper plus its system-cursor-manager child, not established duplicate controllers.

Earlier distinct setup failures

Old computer-use cache 26.527.31326 lacked tslib.es6.js. One missing file was copied from the installed runtime; module import then passed.
The old cache bootstrap returned "Computer Use native pipe is unavailable". Using the current runtime initialization above enabled app discovery/launch/capture, but input still failed.
A separate browser control attempt failed because its configured browser-service.mjs path points to a nonexistent old bundled/browser cache; the current runtime contains that service file. This has not been changed.

Please advise a supported way to obtain input-delivery diagnostics or whether a compatible runtime/update fixes this. No private screenshots, account identifiers or full logs are attached. This report was prepared by Codex from user-authorized diagnostics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the @oai/sky 0.6.26 input path used after list_apps, launch_app, get_window, and get_window_state succeed, using the reported Calculator and Paint reproductions. Done means obtaining input-delivery diagnostics or confirming that a compatible runtime/update fixes click and press_key while preserving screenshots and app launch.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js
Domain
desktop, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.