apache / apache/maka

feat(windows): harden and productionize the maka-cu Computer Use executor

Open
#3,785 6 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

## Relationship

Part of #2142, Phase 5 (Windows Computer Use).

## Context

The Windows support roadmap currently lists Computer Use as deferred work: define a Windows backend using UI Automation plus an appropriate capture API, design consent/secure-desktop/elevation/multi-monitor/scaling/session-lock behavior, reuse the platform-neutral Computer Use host event contract, and add Windows integration/E2E evidence.

This issue proposes the concrete executor work needed to make that phase implementable and reviewable. It is based on a code review of [`maka-agent/maka-cu`](https://github.com/maka-agent/maka-cu), revision [`4a9787d`](https://github.com/maka-agent/maka-cu/commit/4a9787d2c7f2fbc6a29b33d691916c6b84543661), especially its experimental [`apps/OpenComputerUseWindows`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows) runtime.

The current implementation is useful as a functional prototype: Go owns the CLI/MCP/tool schema and an in-process snapshot cache; an embedded PowerShell bridge uses Windows UI Automation for discovery/tree rendering/semantic actions and falls back to Win32 window messages for some input paths. It has been validated against basic Notepad flows. It is not yet a safe or reliable equivalent of the macOS `maka.cu/2` executor.

## Review Findings

### 1. Observation/action binding is too weak

The Windows runtime caches snapshots by a lower-cased app query/name/bundle-like process name/PID and action calls reuse a numeric `element_index`. The PowerShell bridge then re-enumerates the current process tree. If a UIA runtime id is unavailable, it falls back to the first matching AutomationId/name and control type:

- [`main.go#L438-L486`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/main.go#L438-L486)
- [`runtime.ps1#L676-L700`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L676-L700)

This can target the wrong window/control after an app restart, process recycling, modal-window creation, duplicate controls, or a reflow. The macOS host protocol already has the stronger model: opaque per-snapshot element tokens, snapshot state (`live`, `spent`, `superseded`, `expired`, `evicted`), process identity, window identity, and an element digest:

- [`HostSnapshotRegistry.swift#L33-L80`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift#L33-L80)
- [`HostProtocolServer+Observe.swift#L447-L499`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift#L447-L499)

### 2. The screenshot path can disagree with the UIA tree

`Capture-WindowPngBase64` uses `Graphics.CopyFromScreen` on the window rectangle:

- [`runtime.ps1#L565-L632`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L565-L632)

When the target is covered by another window, the image contains the occluding window while the tree describes the target app. Minimized, layered, hardware-accelerated, and some redirected windows can also produce incomplete or black captures.

The production path should prefer a target-window capture API such as Windows Graphics Capture (`IGraphicsCaptureItemInterop::CreateForWindow(HWND, ...)`) and report an explicit degraded capability when only a screen-rectangle fallback is available. `PrintWindow` can be evaluated as a compatibility fallback, but it is synchronous and application-dependent, so it must not be treated as universally correct.

### 3. Win32 input currently reports success without verification

The bridge ignores `PostMessage` return values and returns `ok=true` after a fixed 120 ms delay:

- [`runtime.ps1#L135-L203`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L135-L203)
- [`runtime.ps1#L912-L999`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L912-L999)

A message can fail because of UIPI/integrity level, an invalid or recycled HWND, a toolkit that ignores the message, or a target that has not processed it yet. Windows documents that `PostMessage` and `SendInput` are subject to UIPI. The executor must distinguish at least:

- refused before dispatch;
- dispatched and verified;
- dispatched but outcome unknown;
- dispatch failed;
- unsupported for this toolkit/window.

It must never silently turn an explicitly background-safe path into foreground `SendInput` or global pointer input.

### 4. The wheel fallback uses the wrong coordinate space

`Send-Scroll` converts the target point to client coordinates before putting it in `WM_MOUSEWHEEL.lParam`:

- [`runtime.ps1#L188-L203`](https://github.com/maka-agent/maka-cu/blob/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows/runtime.ps1#L188-L203)

`WM_MOUSEWHEEL` expects screen coordinates in `lParam`, unlike ordinary client-area mouse messages. This is incorrect for a window not positioned at the screen origin and is especially visible on multi-monitor layouts with negative coordinates.

### 5. DPI, threading, and per-call process startup need a production decision

The UIA bounding rectangle is in physical screen coordinates, while the runtime does not establish a clear Per-Monitor-V2 DPI contract for the executor and coordinate conversion. Mixed-DPI monitors can therefore make the screenshot, UIA frame, and Win32 input disagree.

Every tool call currently writes a temporary script and starts a new Windows PowerShell process. This is simple for a prototype but expensive and makes long-lived UIA element/cache/event ownership difficult. Microsoft recommends using a dedicated non-UI MTA thread for UI Automation clients and provides cache requests to reduce cross-process property calls.

## Proposed Direction

### A. Reuse the platform-neutral host contract

Windows should implement the same native host protocol used by macOS (`maka.cu/2`) rather than growing a second, Windows-only nine-tool executor contract. The Maka runtime/host remains responsible for model-facing Anthropic Computer Use semantics; the native executor remains responsible for observation, target binding, dispatch, capture, and verification.

The protocol should carry:

- executor version and capabilities/limits;
- session lifecycle and cancellation;
- `observe` results with opaque snapshot IDs and element tokens;
- target identity including PID, process start time, and HWND/window generation;
- element/window digests and explicit stale/unknown/expired/spent errors;
- dispatch result fields for outcome, tier, path, effect, and verification;
- an explicit capability/degraded result for missing capture, locked desktop, secure desktop, elevation/UIPI, or toolkit limitations.

### B. Use stable target identity, not app-name identity

The minimum Windows target identity should be:

```text
session + snapshotId + pid + processStartTime + hwnd + windowGeneration
```

`HWND` must be revalidated at dispatch time (`IsWindow`, owning PID, and current process start time). A recycled PID or HWND must fail closed. Element tokens must be opaque and resolved only inside the quoted snapshot; numeric indexes can remain a display convenience for the model/runtime, never the dispatch authority.

### C. Make capture and input capability-driven

Recommended dispatch tiers:

1. UIA semantic pattern (`Invoke`, `Toggle`, `SelectionItem`, `Value`, `Scroll`, `Text`);
2. target HWND/window-message path, only when the target control and message contract are known;
3. Windows Graphics Capture / target-window coordinate path;
4. foreground `SendInput`, only as explicit opt-in with user-visible policy and verification.

Each result should identify the selected path and whether the operation was verified. Unsupported or unsafe paths should return typed errors, not fallback silently.

### D. Prefer a long-lived native Windows bridge

Keep the Go/Node integration boundary if useful, but replace per-call PowerShell startup with a long-lived C#/.NET or native helper. It should own:

- UIA COM initialization on a dedicated MTA worker;
- cache requests for bulk tree properties;
- event-driven invalidation for window/control changes;
- Windows Graphics Capture sessions;
- Win32/DPI/monitor identity and coordinate conversion;
- structured HRESULT/Win32/UIPI errors.

A C#/.NET helper is likely the lowest-risk first production step; a direct Go COM/WinRT implementation can be evaluated later if packaging and maintenance justify it.

### E. Define the Windows security/session contract

The implementation must explicitly handle and test:

- normal interactive desktop versus service/SSH/session-0 execution;
- locked workstation and unavailable input desktop;
- UAC secure desktop and elevated target processes;
- UIPI/integrity-level mismatches;
- multi-monitor and mixed-DPI layouts;
- minimized, occluded, redirected, and hardware-accelerated windows;
- sensitive apps and credential/password fields;
- screenshot and input consent/approval;
- no automatic app launch, focus stealing, or foreground fallback by default.

The product must surface unsupported/deferred Computer Use capability in the Windows preview instead of claiming parity.

## Proposed Deliverables

- [ ] Add a Windows-side `maka.cu/2` protocol adapter and capability report.
- [ ] Define and implement Windows snapshot lifecycle and target identity (`PID + process start + HWND/window generation`).
- [ ] Replace numeric-index dispatch with opaque snapshot-bound element tokens and digest validation.
- [ ] Implement target-window capture with Windows Graphics Capture; retain a typed degraded fallback if necessary.
- [ ] Fix coordinate-space handling for wheel, mouse, DPI, virtual-screen, and negative-monitor coordinates.
- [ ] Return structured dispatch outcomes and verify semantic mutations where possible.
- [ ] Move UIA work to a long-lived dedicated MTA bridge and add property/pattern caching.
- [ ] Add consent/locked-desktop/UIPI/elevation/sensitive-app policy and diagnostics.
- [ ] Add a deterministic Windows fixture and interactive desktop E2E suite.
- [ ] Add CI evidence for Windows 11 x64 at minimum, including multi-monitor/mixed-DPI where the runner permits it.
- [ ] Update #2142 and Windows support documentation only after the above capability boundaries are explicit.

## Acceptance Criteria

### Protocol and safety

- An action planned against snapshot A cannot execute against snapshot B, a recycled PID, or a recycled HWND.
- A spent, expired, superseded, evicted, unknown, or mismatched snapshot returns a distinct typed result.
- No unsupported path silently falls back to foreground/global input.
- Every action result states the attempted path, outcome, effect, and verification status.
- Locked/secure desktop/UIPI/elevated-target conditions are reported explicitly.

### Observation and capture

- UIA tree and screenshot refer to the same PID/HWND/window generation.
- Target-window capture is correct when another window covers the target, or the result explicitly declares the degraded capture mode.
- Coordinates remain correct under Per-Monitor-V2 scaling, mixed-DPI monitors, and negative virtual-screen coordinates.
- Tree rendering has bounded node/depth/time budgets and does not hang on a provider that stops responding.

### Actions

- Semantic actions work without foreground activation for supported Win32, WPF/WinUI, Electron/WebView2, and browser fixtures where the toolkit exposes the required pattern.
- Message/input fallbacks have return-value and post-action verification; failures are not reported as success.
- `type_text`, key combinations, click, drag, horizontal/vertical scroll, and set-value have explicit capability coverage per toolkit.

### Release evidence

- A Windows 11 x64 CI lane runs protocol, fixture, and packaged smoke tests.
- Interactive desktop tests cover Notepad, a Chromium/Electron/WebView2 fixture, modal windows, app restart, occlusion, locked session, elevation/UIPI, two monitors, and mixed DPI.
- The Windows preview documentation states exactly what is supported, what is foreground-only, and what remains deferred.

## Non-goals

- This issue does not make every Windows GUI toolkit background-controllable.
- This issue does not require global physical mouse movement as the default path.
- This issue does not make the current PowerShell prototype production-ready by adding more heuristics alone.
- This issue does not claim that Windows Computer Use is supported before the acceptance evidence exists.

## References

- Parent roadmap: [#2142](https://github.com/apache/maka/issues/2142)
- Executor: [`maka-agent/maka-cu`](https://github.com/maka-agent/maka-cu)
- Windows prototype: [`apps/OpenComputerUseWindows`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/apps/OpenComputerUseWindows)
- macOS host protocol: [`OpenComputerUseKit/HostProtocol`](https://github.com/maka-agent/maka-cu/tree/4a9787d2c7f2fbc6a29b33d691916c6b84543661/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol)
- Microsoft UI Automation threading guidance:
- Microsoft UI Automation caching guidance:
- Microsoft UI Automation scaling guidance:
- Microsoft Windows Graphics Capture window interop:
- Microsoft `WM_MOUSEWHEEL` coordinate contract:
- Microsoft `PostMessage`/UIPI behavior:

Contributor guide

Open the contributing guide

Research direction

Start with apps/OpenComputerUseWindows/main.go and runtime.ps1, especially the cited snapshot, capture, input, and scroll paths; compare their contract with HostSnapshotRegistry.swift and HostProtocolServer+Observe.swift. Run the existing Windows Notepad flows first, then use the acceptance criteria to define protocol safety, capability reporting, capture/input verification, and Windows fixture and E2E evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, go, powershell, typescript
Domain
desktop, devtools, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.