set_value and press_key do not move the Computer Use overlay cursor
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 706
Description
### Problem
Clicks, select_text, secondary_action and scroll already fly the overlay cursor to the bound element's centre. `set_value` and an element-targeted `press_key` go through the same binding path and already have a `presentationScreenPoint`, but overlay `kindOf` does not recognise those action types, so the hook only `ensure`s the session cursor and never moves. A person watching Computer Use type into a field sees no cursor travel.
### Desired outcome
`set_value` and `press_key` with a presentation point move and land the click cursor like `click_element`. A `press_key` without an element (no point) still only ensures. Coordinate-dialect `type` / `key` / `screenshot` / `wait` stay unmoved. `window_action` is out of scope (often no element frame).
### Alternatives or workarounds
Relying on the post-action observation text. That tells the model the field changed; it does not show the operator where the agent is typing.
Contributor guide
Assessment
This issue has not been assessed yet.