Stream: option to composite the remote cursor into the framebuffer (spectator-visible pointer)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 184
- Avg merge
- 12h 6m
- Merged PRs (30d)
- 7
Description
Problem
When streaming a desktop sandbox to spectators (users watching an AI agent work), the agent's mouse pointer is invisible on the stream.
Cause: getVNCCommand() starts x11vnc with default cursor handling, so the cursor is delivered via the cursor-shape pseudo-encoding. VNC clients render that shape at the viewer's own pointer position. A remote pointer moved programmatically (e.g. sandbox.moveMouse() / clicks from a computer-use agent) is therefore never drawn for a viewer who isn't moving their own mouse — which is exactly the spectator case that stream.start() + view-only embedding is used for.
For computer-use products this matters a lot: the whole point of the live stream is to watch the agent operate, and the pointer is the main thing to follow.
Proposal
Expose a cursor mode on the stream options, e.g.:
await sandbox.stream.start({ requireAuth: true, cursor: 'composite' })
which appends -nocursorshape to the x11vnc command in getVNCCommand(). With that flag, x11vnc composites the cursor into the framebuffer updates, so every viewer sees the remote pointer at its true position, moving and clicking. Default stays 'shape' (current behavior).
It's a small change — the command is built in one place:
return `x11vnc -bg -display ${this.desktop.display} -forever -wait 50 -shared -rfbport ${this.vncPort} ${pwdFlag} ...`
Current workaround
After stream.start(), we kill the SDK's x11vnc and re-run the same command with -nocursorshape appended. It works, but it duplicates SDK internals (display, port, auth flag) and will drift if those change — hence this request.
Happy to send a PR if you'd take one.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in getVNCCommand(), where the x11vnc command is assembled, and trace the stream options passed by stream.start(). Add the cursor mode while preserving the current default behavior; done means the composite mode adds -nocursorshape and spectators see the remote pointer at its actual position.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- desktop
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100