feat(windows): harden and productionize the maka-cu Computer Use executor
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Relationship
Part of #2142, Phase 5 (Windows Computer Use).
## Context
The Windows support roadmap currently lists Computer Use as deferred work: define a Windows backend using UI Automation plus an appropriate capture API, design consent/secure-desktop/elevation/multi-monitor/scaling/session-lock behavior, reuse the platform-neutral Computer Use host event contract, and add Windows integration/E2E evidence.
This issue proposes the concrete executor work needed to make that phase implementable and reviewable. It is based on a code review of [`maka-agent/maka-cu`](https://github.com/maka-agent/maka-cu), revision [`4a9787d`](https://github.com/maka-agent/maka-cu/commit/4a9787d2c7f2fbc6a29b33d691916c6b84543661), especially its experimental [`apps/OpenComputerUseWindows`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows) runtime.
The current implementation is useful as a functional prototype: Go owns the CLI/MCP/tool schema and an in-process snapshot cache; an embedded PowerShell bridge uses Windows UI Automation for discovery/tree rendering/semantic actions and falls back to Win32 window messages for some input paths. It has been validated against basic Notepad flows. It is not yet a safe or reliable equivalent of the macOS `maka.cu/2` executor.
## Review Findings
### 1. Observation/action binding is too weak
The Windows runtime caches snapshots by a lower-cased app query/name/bundle-like process name/PID and action calls reuse a numeric `element_index`. The PowerShell bridge then re-enumerates the current process tree. If a UIA runtime id is unavailable, it falls back to the first matching AutomationId/name and control type:
- [`main.go#L438-L486`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/main.go#L438-L486)
- [`runtime.ps1#L676-L700`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L676-L700)
This can target the wrong window/control after an app restart, process recycling, modal-window creation, duplicate controls, or a reflow. The macOS host protocol already has the stronger model: opaque per-snapshot element tokens, snapshot state (`live`, `spent`, `superseded`, `expired`, `evicted`), process identity, window identity, and an element digest:
- [`HostSnapshotRegistry.swift#L33-L80`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift#L33-L80)
- [`HostProtocolServer+Observe.swift#L447-L499`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift#L447-L499)
### 2. The screenshot path can disagree with the UIA tree
`Capture-WindowPngBase64` uses `Graphics.CopyFromScreen` on the window rectangle:
- [`runtime.ps1#L565-L632`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L565-L632)
When the target is covered by another window, the image contains the occluding window while the tree describes the target app. Minimized, layered, hardware-accelerated, and some redirected windows can also produce incomplete or black captures.
The production path should prefer a target-window capture API such as Windows Graphics Capture (`IGraphicsCaptureItemInterop::CreateForWindow(HWND, ...)`) and report an explicit degraded capability when only a screen-rectangle fallback is available. `PrintWindow` can be evaluated as a compatibility fallback, but it is synchronous and application-dependent, so it must not be treated as universally correct.
### 3. Win32 input currently reports success without verification
The bridge ignores `PostMessage` return values and returns `ok=true` after a fixed 120 ms delay:
- [`runtime.ps1#L135-L203`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L135-L203)
- [`runtime.ps1#L912-L999`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L912-L999)
A message can fail because of UIPI/integrity level, an invalid or recycled HWND, a toolkit that ignores the message, or a target that has not processed it yet. Windows documents that `PostMessage` and `SendInput` are subject to UIPI. The executor must distinguish at least:
- refused before dispatch;
- dispatched and verified;
- dispatched but outcome unknown;
- dispatch failed;
- unsupported for this toolkit/window.
It must never silently turn an explicitly background-safe path into foreground `SendInput` or global pointer input.
### 4. The wheel fallback uses the wrong coordinate space
`Send-Scroll` converts the target point to client coordinates before putting it in `WM_MOUSEWHEEL.lParam`:
- [`runtime.ps1#L188-L203`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L188-L203)
`WM_MOUSEWHEEL` expects screen coordinates in `lParam`, unlike ordinary client-area mouse messages. This is incorrect for a window not positioned at the screen origin and is especially visible on multi-monitor layouts with negative coordinates.
### 5. DPI, threading, and per-call process startup need a production decision
The UIA bounding rectangle is in physical screen coordinates, while the runtime does not establish a clear Per-Monitor-V2 DPI contract for the executor and coordinate conversion. Mixed-DPI monitors can therefore make the screenshot, UIA frame, and Win32 input disagree.
Every tool call currently writes a temporary script and starts a new Windows PowerShell process. This is simple for a prototype but expensive and makes long-lived UIA element/cache/event ownership difficult. Microsoft recommends using a dedicated non-UI MTA thread for UI Automation clients and provides cache requests to reduce cross-process property calls.
## Proposed Direction
### A. Reuse the platform-neutral host contract
Windows should implement the same native host protocol used by macOS (`maka.cu/2`) rather than growing a second, Windows-only nine-tool executor contract. The Maka runtime/host remains responsible for model-facing Anthropic Computer Use semantics; the native executor remains responsible for observation, target binding, dispatch, capture, and verification.
The protocol should carry:
- executor version and capabilities/limits;
- session lifecycle and cancellation;
- `observe` results with opaque snapshot IDs and element tokens;
- target identity including PID, process start time, and HWND/window generation;
- element/window digests and explicit stale/unknown/expired/spent errors;
- dispatch result fields for outcome, tier, path, effect, and verification;
- an explicit capability/degraded result for missing capture, locked desktop, secure desktop, elevation/UIPI, or toolkit limitations.
### B. Use stable target identity, not app-name identity
The minimum Windows target identity should be:
```text
session + snapshotId + pid + processStartTime + hwnd + windowGeneration
```
`HWND` must be revalidated at dispatch time (`IsWindow`, owning PID, and current process start time). A recycled PID or HWND must fail closed. Element tokens must be opaque and resolved only inside the quoted snapshot; numeric indexes can remain a display convenience for the model/runtime, never the dispatch authority.
### C. Make capture and input capability-driven
Recommended dispatch tiers:
1. UIA semantic pattern (`Invoke`, `Toggle`, `SelectionItem`, `Value`, `Scroll`, `Text`);
2. target HWND/window-message path, only when the target control and message contract are known;
3. Windows Graphics Capture / target-window coordinate path;
4. foreground `SendInput`, only as explicit opt-in with user-visible policy and verification.
Each result should identify the selected path and whether the operation was verified. Unsupported or unsafe paths should return typed errors, not fallback silently.
### D. Prefer a long-lived native Windows bridge
Keep the Go/Node integration boundary if useful, but replace per-call PowerShell startup with a long-lived C#/.NET or native helper. It should own:
- UIA COM initialization on a dedicated MTA worker;
- cache requests for bulk tree properties;
- event-driven invalidation for window/control changes;
- Windows Graphics Capture sessions;
- Win32/DPI/monitor identity and coordinate conversion;
- structured HRESULT/Win32/UIPI errors.
A C#/.NET helper is likely the lowest-risk first production step; a direct Go COM/WinRT implementation can be evaluated later if packaging and maintenance justify it.
### E. Define the Windows security/session contract
The implementation must explicitly handle and test:
- normal interactive desktop versus service/SSH/session-0 execution;
- locked workstation and unavailable input desktop;
- UAC secure desktop and elevated target processes;
- UIPI/integrity-level mismatches;
- multi-monitor and mixed-DPI layouts;
- minimized, occluded, redirected, and hardware-accelerated windows;
- sensitive apps and credential/password fields;
- screenshot and input consent/approval;
- no automatic app launch, focus stealing, or foreground fallback by default.
The product must surface unsupported/deferred Computer Use capability in the Windows preview instead of claiming parity.
## Proposed Deliverables
- [ ] Add a Windows-side `maka.cu/2` protocol adapter and capability report.
- [ ] Define and implement Windows snapshot lifecycle and target identity (`PID + process start + HWND/window generation`).
- [ ] Replace numeric-index dispatch with opaque snapshot-bound element tokens and digest validation.
- [ ] Implement target-window capture with Windows Graphics Capture; retain a typed degraded fallback if necessary.
- [ ] Fix coordinate-space handling for wheel, mouse, DPI, virtual-screen, and negative-monitor coordinates.
- [ ] Return structured dispatch outcomes and verify semantic mutations where possible.
- [ ] Move UIA work to a long-lived dedicated MTA bridge and add property/pattern caching.
- [ ] Add consent/locked-desktop/UIPI/elevation/sensitive-app policy and diagnostics.
- [ ] Add a deterministic Windows fixture and interactive desktop E2E suite.
- [ ] Add CI evidence for Windows 11 x64 at minimum, including multi-monitor/mixed-DPI where the runner permits it.
- [ ] Update #2142 and Windows support documentation only after the above capability boundaries are explicit.
## Acceptance Criteria
### Protocol and safety
- An action planned against snapshot A cannot execute against snapshot B, a recycled PID, or a recycled HWND.
- A spent, expired, superseded, evicted, unknown, or mismatched snapshot returns a distinct typed result.
- No unsupported path silently falls back to foreground/global input.
- Every action result states the attempted path, outcome, effect, and verification status.
- Locked/secure desktop/UIPI/elevated-target conditions are reported explicitly.
### Observation and capture
- UIA tree and screenshot refer to the same PID/HWND/window generation.
- Target-window capture is correct when another window covers the target, or the result explicitly declares the degraded capture mode.
- Coordinates remain correct under Per-Monitor-V2 scaling, mixed-DPI monitors, and negative virtual-screen coordinates.
- Tree rendering has bounded node/depth/time budgets and does not hang on a provider that stops responding.
### Actions
- Semantic actions work without foreground activation for supported Win32, WPF/WinUI, Electron/WebView2, and browser fixtures where the toolkit exposes the required pattern.
- Message/input fallbacks have return-value and post-action verification; failures are not reported as success.
- `type_text`, key combinations, click, drag, horizontal/vertical scroll, and set-value have explicit capability coverage per toolkit.
### Release evidence
- A Windows 11 x64 CI lane runs protocol, fixture, and packaged smoke tests.
- Interactive desktop tests cover Notepad, a Chromium/Electron/WebView2 fixture, modal windows, app restart, occlusion, locked session, elevation/UIPI, two monitors, and mixed DPI.
- The Windows preview documentation states exactly what is supported, what is foreground-only, and what remains deferred.
## Non-goals
- This issue does not make every Windows GUI toolkit background-controllable.
- This issue does not require global physical mouse movement as the default path.
- This issue does not make the current PowerShell prototype production-ready by adding more heuristics alone.
- This issue does not claim that Windows Computer Use is supported before the acceptance evidence exists.
## References
- Parent roadmap: [#2142](https://github.com/apache/maka/issues/2142)
- Executor: [`maka-agent/maka-cu`](https://github.com/maka-agent/maka-cu)
- Windows prototype: [`apps/OpenComputerUseWindows`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows)
- macOS host protocol: [`OpenComputerUseKit/HostProtocol`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol)
- Microsoft UI Automation threading guidance:
- Microsoft UI Automation caching guidance:
- Microsoft UI Automation scaling guidance:
- Microsoft Windows Graphics Capture window interop:
- Microsoft `WM_MOUSEWHEEL` coordinate contract:
- Microsoft `PostMessage`/UIPI behavior:
Contributor guide
Research direction
Start with apps/OpenComputerUseWindows/main.go and runtime.ps1, especially the cited snapshot, capture, input, and scroll paths; compare their contract with HostSnapshotRegistry.swift and HostProtocolServer+Observe.swift. Run the existing Windows Notepad flows first, then use the acceptance criteria to define protocol safety, capability reporting, capture/input verification, and Windows fixture and E2E evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, go, powershell, typescript
- Domain
- desktop, devtools, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100