apache / apache/maka

set_value and press_key do not move the Computer Use overlay cursor

Open
#5,049 1 comment 0 reactions 1 assignee Claimed by @wutongyuonce View on GitHub
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
706

Description

### Problem

Clicks, select_text, secondary_action and scroll already fly the overlay cursor to the bound element's centre. `set_value` and an element-targeted `press_key` go through the same binding path and already have a `presentationScreenPoint`, but overlay `kindOf` does not recognise those action types, so the hook only `ensure`s the session cursor and never moves. A person watching Computer Use type into a field sees no cursor travel.

### Desired outcome

`set_value` and `press_key` with a presentation point move and land the click cursor like `click_element`. A `press_key` without an element (no point) still only ensures. Coordinate-dialect `type` / `key` / `screenshot` / `wait` stay unmoved. `window_action` is out of scope (often no element frame).

### Alternatives or workarounds

Relying on the post-action observation text. That tells the model the field changed; it does not show the operator where the agent is typing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.