openai / openai/codex

Let Codey accompany Computer Use and act out clicks, typing, scrolling, and drag-and-drop

Open
#42,697 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app computer-use enhancement pets
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What variant of Codex are you using?

Codex desktop app on macOS, using an awake pet and Computer Use.

What feature would you like to see?

I noticed that when Computer Use takes over, I can see its little cursor moving around the screen. I already keep Codey awake in the corner, and I had a simple thought: what if Codey became the thing I saw doing the work?

Instead of only watching a cursor move, I’d see Codey travel to the target and act out what Computer Use is doing:

  • Run or glide toward whatever Computer Use is selecting.
  • Tap, punch, or playfully kick a button when clicking it.
  • Open a tiny laptop or keyboard while text is being entered.
  • Pull a page or turn a little wheel while scrolling.
  • Carry a folder, parcel, or generic object while dragging and dropping something.
  • Wave, jump, or otherwise get my attention when a Codex task needs me.
  • Celebrate when the intended result has actually been verified.
  • Stop and show the appropriate waiting, blocked, failed, or cancelled state when necessary.

Codey could be the main visual representation of Computer Use, while a small target dot, hotspot, or click ripple remains visible so the exact action is still precise and Codey never covers a small control.

I think it would be both cute and genuinely useful. It would make Computer Use feel less like an invisible system moving a pointer and more like the little Codex companion is actually helping with things on my computer.

Additional information

One particularly useful case would be background tasks.

If I am working on something else and another Codex task reaches a point where it is about to use Computer Use, Codey could wave, jump, or display a small notice such as:

“A Codex task is ready to use Computer Use.”

The notice could offer simple choices such as:

  • Continue
  • Pause Computer Use
  • Open the task

This should respect all existing app permissions and approval requirements. It should not silently authorize anything, steal focus, or claim an action succeeded before Codex verifies the result.

A deliberately small first version could support only built-in Codey animations driven by sanitized first-party Computer Use events such as move, click, type, scroll, drag/drop, waiting, completion, failure, and cancellation.

Custom pets without action-specific artwork could continue using their existing status behavior. Custom action mappings could be considered later or alongside the broader animation-manifest proposals.

Important behavior:

  • The animation must never delay or alter the actual Computer Use action.
  • An action animation represents what Computer Use is attempting or dispatching—not proof that it succeeded.
  • Celebration should happen only after the intended result is verified.
  • Approval, needs-input, blocked, failure, and cancellation states should take priority over decorative animations.
  • Codey must not obscure controls, text fields, permission prompts, or drag destinations.
  • The feature should respect Reduce Motion.
  • It should behave correctly across displays, display scaling, Spaces, and fullscreen applications.
  • Cursor tracking and animation work should suspend when Codey or Computer Use is inactive.
  • Ideally, it should require no broader macOS access than official Computer Use already uses.

A community report in #33224 says that a V2 pet’s gaze can already respond to the Computer Use virtual cursor. That is not a documented public API or confirmed implementation contract, but it suggests that a pet-facing cursor-position signal may already exist.

If full first-party choreography is out of scope, would the team consider exposing a supported, local, read-only stream of sanitized Computer Use events—such as cursor position, action type, and lifecycle state—or point users to an existing supported extension surface? That could allow this visualization without patching Codex, intercepting input, or accessing screen contents.

Related but distinct:

  • #20863 requests configurable pet animation sequences and richer activity events.
  • #21657 requests pet interaction hooks and supported APIs.
  • #36255 requests broad Codex-state-to-animation mapping.
  • #33224 concerns pet gaze responding to the Computer Use virtual cursor.

This request is specifically about optional, spatial, action-by-action visualization of Computer Use through the active pet.

Disclosure: The original idea and examples came from me during a short, casual exploration—probably about ten minutes. I used Codex to research related requests, examine whether anything similar already existed, and organize the idea into this post. I did not personally write most of the wording or conduct the technical analysis. I’m sharing it because it seemed fun and useful, and I wanted to see whether other people might enjoy it too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are identified. Start by reviewing the related requests #20863, #21657, #36255, and #33224, then determine whether a supported Computer Use event or extension surface exists; done would require optional action visualization that preserves approval, verification, accessibility, display, and motion-reduction behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, rust
Domain
ai, desktop
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.