e2b-dev / e2b-dev/desktop

Stream: option to composite the remote cursor into the framebuffer (spectator-visible pointer)

Open Beginner friendly
#249 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.5k
Forks
184
Avg merge
12h 6m
Merged PRs (30d)
7

Description

Problem

When streaming a desktop sandbox to spectators (users watching an AI agent work), the agent's mouse pointer is invisible on the stream.

Cause: getVNCCommand() starts x11vnc with default cursor handling, so the cursor is delivered via the cursor-shape pseudo-encoding. VNC clients render that shape at the viewer's own pointer position. A remote pointer moved programmatically (e.g. sandbox.moveMouse() / clicks from a computer-use agent) is therefore never drawn for a viewer who isn't moving their own mouse — which is exactly the spectator case that stream.start() + view-only embedding is used for.

For computer-use products this matters a lot: the whole point of the live stream is to watch the agent operate, and the pointer is the main thing to follow.

Proposal

Expose a cursor mode on the stream options, e.g.:

await sandbox.stream.start({ requireAuth: true, cursor: 'composite' })

which appends -nocursorshape to the x11vnc command in getVNCCommand(). With that flag, x11vnc composites the cursor into the framebuffer updates, so every viewer sees the remote pointer at its true position, moving and clicking. Default stays 'shape' (current behavior).

It's a small change — the command is built in one place:

return `x11vnc -bg -display ${this.desktop.display} -forever -wait 50 -shared -rfbport ${this.vncPort} ${pwdFlag} ...`

Current workaround

After stream.start(), we kill the SDK's x11vnc and re-run the same command with -nocursorshape appended. It works, but it duplicates SDK internals (display, port, auth flag) and will drift if those change — hence this request.

Happy to send a PR if you'd take one.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in getVNCCommand(), where the x11vnc command is assembled, and trace the stream options passed by stream.start(). Add the cursor mode while preserving the current default behavior; done means the composite mode adds -nocursorshape and spectators see the remote pointer at its actual position.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
desktop
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.