1jehuang / 1jehuang/jcode

computer tool: truthful observability animation for background AX actions (highlight when visible, notice when occluded)

Open
#348 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement triage: needs-decision
Dominant language
Rust
Stars
19.9k
Forks
2.3k
Avg merge
2d 7h
Merged PRs (30d)
30

Description

Summary

The macos_computer_use tool (added in #345, gated to macOS) can act on the
desktop two ways:

  • Background AX / scripting (press, set_value, select_menu,
    run_applescript, ...): acts on a UI element by reference. No cursor
    moves, the target app need not be focused or even frontmost. This is the
    preferred, non-disruptive path.
  • Visible CGEvent input (move, click, type, drag, scroll):
    drives the one real system cursor on the shared HID stream, so the user
    sees it.

The background path is great for staying out of the user's way, but it has a
trust cost: silent actions on the user's live machine with no signal that
anything happened.
We should add a lightweight, truthful observability
indicator so the user knows what the agent is doing.

This is observability/trust work, not decoration. It does not replace the
policy text or any future confirmation gate; a flash is a notice, not consent.

Proposed behavior

Decide what to show based on visibility/occlusion, not focus (the two are
orthogonal on macOS: a window can be unfocused but fully visible).

  1. Target's window is actually visible on screen (focused or not, and
    topmost at the element's rect): draw a brief, click-through,
    non-activating highlight/flash on the element's on-screen bounds. We
    already resolve the element's @pos in find_element, so we have the
    rectangle. ~300ms fade.
  2. Target is occluded / minimized / hidden / on another Space or display:
    do not draw a positional highlight, it would float over an unrelated
    window the user is actually looking at, which is misleading. Fall back to
    a non-positional notice (toast / activity-log line), e.g. "agent set
    value in TextEdit (background)".
  3. Visible CGEvent actions already move the real cursor, so the user sees
    them. A small click ripple/trail is optional and is about legibility (the
    cursor teleports instantly, which is jarring), not about revealing hidden
    activity.

Deciding signal = occlusion / on-screen hit-test (element AX frame vs. the
window list to confirm the target isn't covered), not whether the window
has focus.

Explicitly out of scope / anti-goals

  • No fake "ghost cursor." Animating a pointer gliding around for
    background actions implies a cursor moved when nothing did. Don't fake
    motion that didn't happen.
  • No focus stealing, no fullscreen overlay, no added latency to the
    action itself. Any overlay must be click-through and non-activating
    ("stay out of the user's way", consistent with the restraint guidance
    already in the tool description / discover policy).
  • Don't make the animation load-bearing for safety.

Open questions

  • Surface: this likely needs a small always-on overlay helper (the TUI
    can't draw a macOS overlay). Standalone borderless NSWindow /
    NSPanel (non-activating, ignoresMouseEvents) vs. reusing existing
    desktop-app surface?
  • Occlusion hit-test: cheap/robust way to confirm the element's rect is the
    topmost window there (CGWindowList ordering + frame intersection?).
  • Multi-display / Spaces handling for the "is it visible" check.
  • Toast/activity-log channel for the non-visible case, new surface or reuse
    an existing notification path?

Refs

  • Parent feature: #340
  • Implementation PR: #345

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the macos_computer_use implementation from PR #345, especially find_element and the existing discover policy guidance. Define the overlay, occlusion check, and non-visible notice surfaces there; done means visible targets receive a truthful click-through highlight, while occluded or hidden targets receive only a non-positional notice without focus stealing or action latency.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, rust
Domain
desktop-dev, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.