computer tool: truthful observability animation for background AX actions (highlight when visible, notice when occluded)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 30
Description
Summary
The macos_computer_use tool (added in #345, gated to macOS) can act on the
desktop two ways:
- Background AX / scripting (
press,set_value,select_menu,
run_applescript, ...): acts on a UI element by reference. No cursor
moves, the target app need not be focused or even frontmost. This is the
preferred, non-disruptive path. - Visible CGEvent input (
move,click,type,drag,scroll):
drives the one real system cursor on the shared HID stream, so the user
sees it.
The background path is great for staying out of the user's way, but it has a
trust cost: silent actions on the user's live machine with no signal that
anything happened. We should add a lightweight, truthful observability
indicator so the user knows what the agent is doing.
This is observability/trust work, not decoration. It does not replace the
policy text or any future confirmation gate; a flash is a notice, not consent.
Proposed behavior
Decide what to show based on visibility/occlusion, not focus (the two are
orthogonal on macOS: a window can be unfocused but fully visible).
- Target's window is actually visible on screen (focused or not, and
topmost at the element's rect): draw a brief, click-through,
non-activating highlight/flash on the element's on-screen bounds. We
already resolve the element's@posinfind_element, so we have the
rectangle. ~300ms fade. - Target is occluded / minimized / hidden / on another Space or display:
do not draw a positional highlight, it would float over an unrelated
window the user is actually looking at, which is misleading. Fall back to
a non-positional notice (toast / activity-log line), e.g. "agent set
value in TextEdit (background)". - Visible CGEvent actions already move the real cursor, so the user sees
them. A small click ripple/trail is optional and is about legibility (the
cursor teleports instantly, which is jarring), not about revealing hidden
activity.
Deciding signal = occlusion / on-screen hit-test (element AX frame vs. the
window list to confirm the target isn't covered), not whether the window
has focus.
Explicitly out of scope / anti-goals
- No fake "ghost cursor." Animating a pointer gliding around for
background actions implies a cursor moved when nothing did. Don't fake
motion that didn't happen. - No focus stealing, no fullscreen overlay, no added latency to the
action itself. Any overlay must be click-through and non-activating
("stay out of the user's way", consistent with the restraint guidance
already in the tool description /discoverpolicy). - Don't make the animation load-bearing for safety.
Open questions
- Surface: this likely needs a small always-on overlay helper (the TUI
can't draw a macOS overlay). Standalone borderlessNSWindow/
NSPanel(non-activating, ignoresMouseEvents) vs. reusing existing
desktop-app surface? - Occlusion hit-test: cheap/robust way to confirm the element's rect is the
topmost window there (CGWindowList ordering + frame intersection?). - Multi-display / Spaces handling for the "is it visible" check.
- Toast/activity-log channel for the non-visible case, new surface or reuse
an existing notification path?
Refs
- Parent feature: #340
- Implementation PR: #345
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the macos_computer_use implementation from PR #345, especially find_element and the existing discover policy guidance. Define the overlay, occlusion check, and non-visible notice surfaces there; done means visible targets receive a truthful click-through highlight, while occluded or hidden targets receive only a non-positional notice without focus stealing or action latency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, rust
- Domain
- desktop-dev, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100