NVIDIA / NVIDIA/IsaacTeleop

[Bug Report] isaacsim.kit.xr.teleop.bridge extension not found - CloudXR teleoperation cannot fully function

Open
#383 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
385
Forks
88
Avg merge
3d 23h
Merged PRs (30d)
82

Description

Describe the bug

Following the official CloudXR teleoperation documentation:
https://isaac-sim.github.io/IsaacLab/develop/source/how-to/cloudxr_teleoperation.html

The isaacsim.kit.xr.teleop.bridge extension cannot be resolved from any extension registry when using Isaac Sim 6.0 (pip install) with Isaac Teleop 1.0.191. This prevents CloudXR teleoperation from functioning - no controller input reaches Isaac Teleop from the VR headset.

Failed to resolve extension dependencies. Failure hints:
  * No versions of isaacsim.kit.xr.teleop.bridge that satisfies: [enable list]
    - Available packages for isaacsim.kit.xr.teleop.bridge version *: (none found)
  Synced registries:
     - kit/default         : found 0 packages (couldn't connect or empty)
     - kit/sdk             : found 0 packages (couldn't connect or empty)
     - kit/prod/default    : found 931 packages 
     - kit/prod/sdk        : found 785 packages 

The kit/default registry (pdx.s8k.io) is unreachable (TLS handshake fails). The extension is also not included in the Docker image (nvcr.io/nvidia/isaac-sim:6.0.0-dev2).

Additionally, omni.kit.xr.system.openxr does not expose get_instance_proc_addr(), which the bridge extension is supposed to polyfill (session_lifecycle.py line 68-82). Without the polyfill, _acquire_kit_oxr_handles() crashes with AttributeError.

What works
  • G1 simulation rendering streamed to PICO 4 Ultra Enterprise via CloudXR
  • Camera view follows head movement in the headset
What does not work
  • Controller input from PICO does not reach Isaac Teleop at all (no pose tracking, no button input from headset controllers)
  • ControllerTracker update failed repeats continuously
  • XR_ERROR_FORM_FACTOR_UNSUPPORTED: xrGetSystem(getInfo->formFactor) no matching system (given: 2, first: 1) - Kit XR requests form factor 2 (handheld/AR) but CloudXR Monado only supports form factor 1 (HMD)
Steps to reproduce
  1. Install Isaac Sim 6.0 via pip: pip install "isaacsim[all,extscache]==6.0.0"
  2. Install Isaac Lab (develop branch)
  3. Install Isaac Teleop: pip install "isaacteleop[retargeters,ui,cloudxr]~=1.0.0" --extra-index-url https://pypi.nvidia.com
  4. Start CloudXR: python -m isaacteleop.cloudxr --accept-eula (with NV_CXR_ENABLE_PUSH_DEVICES=0)
  5. Launch teleop: ./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py --task Isaac-PickPlace-Locomanipulation-G1-Abs-v0 --visualizer kit --xr
  6. Click Start XR in the XR tab, connect PICO via CloudXR.js client
Questions
  1. Is the isaacsim.kit.xr.teleop.bridge extension supposed to be included in the Isaac Sim 6.0 pip package or downloaded from a registry?
  2. Is there a workaround to get controller input working without the bridge extension?
  3. Is this expected to work with Isaac Sim 6.0 Early Developer Release, or should we wait for the GA release?

Related issue: https://github.com/isaac-sim/IsaacLab/issues/5221

System Info
  • OS: Ubuntu 22.04
  • GPU: NVIDIA RTX PRO 6000 Blackwell (98GB VRAM)
  • Driver: 580.126.09
  • Isaac Sim: 6.0.0 (pip)
  • Isaac Lab: develop branch
  • Isaac Teleop: 1.0.191
  • CloudXR: 6.1.0
  • VR Headset: PICO 4 Ultra Enterprise
  • Python: 3.12.13

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the pip installation and CloudXR launch sequence, then inspect session_lifecycle.py lines 68-82 and the teleop_se3_agent.py entry point. Check whether isaacsim.kit.xr.teleop.bridge is packaged or available through the listed registries, and verify that the extension resolves without the get_instance_proc_addr() failure. Done means controller input reaches Isaac Teleop without repeated tracker errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ar-vr-xr
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.