zai-org / zai-org/feedback

zcode-cua 0.5.12 (Windows): all raw coordinate clicks refused with 'dispatch app/window identity does not match frame'

Open
#380 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority: P2
Dominant language
No language data
Stars
22
Forks
1
PR merge metrics
No merged PRs in 30d

Description

zcode-cua 0.5.12 (Windows): every raw coordinate click refused — "dispatch app/window identity does not match frame"

Summary

On Windows 10, the zcode-cua plugin refuses 100% of raw coordinate pointer actions (left_click and the other pointer tools with a coordinate target), for both window-bound rasters and full-display rasters, regardless of focus, overlays, or window position. Accessibility-element actions, app-scoped keyboard, screenshots and enumeration all work fine, so the plugin remains usable — but pixel-precise clicking (canvas-like surfaces such as Paint) is impossible.

Environment

  • OS: Windows 10 Pro, build 19045 x64, single display 1920x1080
  • Plugin: zcode-cua 0.5.12 (official marketplace zcode-plugins-official)
  • Broker: "ZCode Computer Use" helper, backend electron, api ZCodeComputerUseIPC-1
  • Behavior is identical before and after a full ZCode desktop-app restart (helper pid 24384 → 29748), which rules out client/helper version skew within a session
  • request_access reports: accessibility granted, screen recording granted, raw_mouse_keyboard: true, coordinate_hit_test: true

Symptom

Every pointer action with a coordinate target returns action_sent=false:

  • window-bound frame:
    click: dispatch app/window identity does not match frame <frame_id>
  • full-display frame:
    click: full-display frame <frame_id> is stale; the live pixel owner changed before dispatch

Tried and refused in all combinations:

  • strategy: "auto" and strategy: "event"
  • with and without explicit frame_id (i.e. transport-local current raster too)
  • with and without app_ref (plain pid, and pid + window_id pinned)
  • on freshly captured rasters (get_app_state immediately followed by the click)
  • on multiple targets (classic Win32 app mspaint, desktop background)
  • open_application(activate=true) before clicking (reports active:true)

Isolation performed (common causes ruled out)

  1. Invisible full-screen overlays. The machine had a phantom full-screen LockApp.exe window (Windows Spotlight leftover, 1920x1080) plus two invisible full-screen TextInputHost.exe windows. Killed both processes, verified gone → refusals unchanged.
  2. Foreground/focus. Target window forced to true OS foreground via minimize→restore + SetForegroundWindow; verified GetForegroundWindow() == target HWND immediately before the click → refusal unchanged. (list_windows focused flag also true.)
  3. Child-HWND ambiguity. Clicks aimed at the title bar (the same HWND the frame was captured from, no child-window ambiguity) are refused identically → not a child-window hit-test mismatch.
  4. Frame staleness / timing. Fresh get_app_state raster followed by an immediate click → refused. Same result after the full ZCode restart.
  5. Target app. Reproduced on mspaint (classic Win32) and against the desktop background → not app-specific.
Minimal repro
  1. open_application({"name":"mspaint"}, activate=true)
  2. get_app_state(pid, include_screenshot=true)
  3. left_click({"type":"coordinate","x":X,"y":Y}) using coordinates from the returned raster
  4. click: dispatch app/window identity does not match frame <frame_id> — every time

What works (same session, same apps)

  • get_app_state (incl. screenshots), screenshot, list_apps, list_windows, request_access
  • Element-target actions: left_click (AXPress) on ribbon buttons and other UIA elements
  • type into editable element targets; key/hold_key/type with app_ref — app-scoped keyboard reaches the target app even when it is not the OS-foreground window
  • write_clipboard / PowerShell clipboard interop

Workaround used in practice: drive UI via accessibility elements and app-scoped keyboard; for Paint's canvas, render text to a bitmap, place it on the clipboard and paste (Ctrl+V), then commit via a ribbon element press.

Suspected area

The dispatch-time identity verification for coordinate targets on the Windows Electron backend appears to never accept its own freshly captured frames — possibly a DPI/coordinate-transform mismatch or a frame→HWND identity mapping issue in the helper. The same plugin is reported working on macOS (CGWindowID-based), so this looks Windows-specific.

Happy to run further diagnostics on the affected machine if useful (happy to test dev builds/patched helpers too).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal repro: open_application, get_app_state(include_screenshot=true), then left_click using coordinates from the returned raster. Inspect the Windows Electron backend's dispatch-time identity verification; done means coordinate-target clicks are accepted for freshly captured window-bound and full-display frames on the described Windows cases while existing element actions remain working.

Written by the indexing model from the issue text.

Assessment

Tech stack
electron
Domain
desktop-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.