apache / apache/maka

Computer Use cannot show the model what select_text actually selected

Open
#5,047 1 comment 0 reactions 1 assignee Claimed by @wutongyuonce View on GitHub
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
706

Description

### Problem

`maka_computer` already accepts `select_text`. The maka-cu executor reports the selected string on the snapshot as `selectedText: { text, truncated }`, and the host backend copies it onto the observation object. The shared `CuObservation` type does not declare that field, and `renderObservationForModel` never writes it. After a successful select, the model only sees `ok` plus the element tree. `element.selected` is AX control selection (checkbox/row), not the window text range. `select_text` also uses screenshot policy `never`, so there is no picture of the highlight either. The next turn cannot confirm what was selected.

### Desired outcome

A model that just called `select_text` can read the selected string from the fresh observation header, truncated the same way element `value` is truncated. If the focused control is a secure field, the selected string is withheld rather than printed. Observations with no selection do not mention the field.

### Alternatives or workarounds

Asking the model to re-observe and guess from element values. That is slower and still does not report a range that spans multiple elements.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.