Computer Use (Windows): a11y element press silently no-ops on native app buttons (1C:Enterprise dialogs) - document click delivery and publish tested fallback matrix
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 22
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Computer Use on Windows: the accessibility element-press route (element-target left_click) silently does nothing on buttons of some native Windows applications — the tool reports success, but the button is never activated. Please document how clicks are actually delivered, and publish a tested matrix of fallback delivery methods for such windows.
Expected behavior
When the Computer Use tool presses a button via an accessibility element target and the UI state does not change, one of the following should hold:
- The press is delivered in a way that native (non-HTML, non-Electron) Windows apps actually react to, or
- The tool returns a warning / no-op indication instead of a silent success, and the documented fallback (e.g. synthetic mouse input at coordinates, keyboard activation) is applied automatically or is clearly documented.
Additionally, the official documentation should state which delivery mechanism is used per platform (accessibility press vs. synthetic input), so agents can reason about which apps it will work on.
Why this matters
- The failure is silent: dispatch is accepted, no error is returned. The agent believes the button was pressed, takes a screenshot, sees nothing happened, and retries in a loop — or, worse, concludes the task is done.
- Such native dialogs are often modal security/confirmation prompts that block the whole application until answered, so the entire session stalls.
- The same no-op happens with direct UIA
InvokePattern.Invoke()from an external script on the same button, which suggests the a11y press route and UIA Invoke are equivalent here and simply unsupported by the target control — only real input events work.
Steps to reproduce
Reproduced with 1C:Enterprise 8.3 (a large Russian ERP/platform; window class V8TopLevelFrameSDIsec, window title 1С:Предприятие — its modal security confirmation dialog "Предупреждение безопасности"):
- Open any external data processor in 1C:Enterprise thin client; the platform shows a modal security dialog with buttons
Да/Нет/Закрыть(note: UIA names are padded with spaces). - From the accessibility tree, take the button element (
[button] " Да ") and press it via Computer Useleft_clickwith an element target (a11y press route). Also try a coordinate click under the defaultautostrategy — it resolves to the same a11y hit-test press. - Take a screenshot before and after.
Observed: both presses are dispatched without any error, but the dialog stays open and the screenshot is unchanged. A standalone PowerShell script doing InvokePattern.Invoke() on the same button via Windows UI Automation also does nothing.
What does work on this exact window:
- Physical synthetic mouse click (
SendInput/mouse_event) afterSetForegroundWindow— button activates, dialog closes. - Keyboard
Enterafter the window is activated and focused. - UIA is fine for reading:
AutomationId, names, and bounding rects are all correct, and a watcher script that polls for the dialog and clicks withSendInputworks reliably (log excerpt:dialog hwnd=0x3D133E ... CLICKED: Да).
So the button is discoverable but not invokable; it only reacts to real input events. 1C draws its own controls, which likely do not implement UIA InvokeProvider — yet the a11y press route still reports success instead of failing loudly.
Requested actions
- Document the click delivery mechanism per platform in the official docs (zcode.z.ai / docs.z.ai). I searched both sites (checked 2026-09-07, including the devpack tool pages): there is no public page explaining whether clicks go through accessibility press (macOS AXPress / Windows UIA InvokePattern) or synthetic input (SendInput), and no guidance on fallbacks when a press silently no-ops. "AXPress" itself is macOS Accessibility terminology; on Windows the equivalent would be UIA Invoke — please use platform-accurate wording in docs.
- Publish a tested delivery-method matrix for native Windows apps, validated with before/after screenshots (not just "dispatch accepted"). Suggested methods to compare on such a window, e.g. with AutoHotkey (freely available, no licensing issues):
- UIA
InvokePattern.Invoke()/LegacyIAccessible.DoDefaultAction() PostMessage/SendMessageWM_LBUTTONDOWN/WM_LBUTTONDOWN+WM_LBUTTONUPdirectly to the button HWNDSendInputmouse move + click at the button's screen rect (withSetForegroundWindow)mouse_event(legacy API)- keyboard activation: focus the button, send
Enter/Space
Document which methods work on owner-drawn controls that lack Invoke providers, and make the Computer Use tool fall back automatically (or at least warn) when the a11y press has no effect.
- UIA
- Return a warning on suspected no-op: if a press is dispatched via a11y and the target element's state/window content is unchanged after a short delay, the tool result should say so, instead of silent success.
Note on testability: 1C:Enterprise is hardly available in China, so reproducing this exact dialog may be difficult internally. If needed, I can provide UIA dumps, window class names, and run any test build against my environment (Windows 11 x64, 1C:Enterprise 8.3.27).
Test scenarios
- Preparation: any machine with 1C:Enterprise 8.3 thin client (or another app with owner-drawn, non-Invokable buttons — WPF/WinForms custom controls qualify), Computer Use enabled, screenshot validation scripted.
- Action: press the dialog button via element target, then via each delivery method above.
- Expectation: methods that actually activate the button change the screenshot (dialog closes); the tool result for the failing a11y route contains a no-op warning rather than plain success.
Environment
Windows 11 Pro x64 (build 26200), ZCode Computer Use plugin 0.5.14 (Windows backend), target app 1C:Enterprise 8.3.27 thin client (Russian locale UI).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Windows Computer Use element-target left_click path and reproduce the issue on 1C:Enterprise or another owner-drawn native app. Compare UIA Invoke, keyboard activation, and synthetic mouse methods using before/after screenshots. Done means the official docs describe platform delivery, a tested fallback matrix is published, and a suspected unchanged-state press produces a warning rather than silent success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- powershell
- Domain
- accessibility, desktop, documentation, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100