NVIDIA / NVIDIA/IsaacTeleop

[Feature Request] Supported Unity/Android client path for a native Meta Quest robot teleoperation app

Open
#784 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
385
Forks
88
Avg merge
3d 23h
Merged PRs (30d)
82

Description

Is your feature request related to a problem? Please describe.

I would like to build a native Unity/OpenXR application for the Meta Quest Store that allows users to:

  • view low-latency live video from a real robot;
  • control the robot using Quest Touch controllers;
  • use clutch, gripper, stop/hold, and disconnect controls inside the application.

I have already successfully tested the following real-hardware workflow:

Quest 3S Touch Plus controllers
→ CloudXR Runtime 6.2.0
→ NVIDIA Isaac Teleop 1.3.131
→ LeRobot v0.6.0 official examples/isaac_teleop_to_so101 example
→ SO-101 Follower robot

Test system:

  • Ubuntu 24.04
  • NVIDIA RTX 4080
  • Quest 3S
  • SO-101 robot arm

The Quest controller poses and buttons reached the Ubuntu host correctly. Grip/Squeeze worked as the clutch, Trigger controlled the gripper, and the SO-101 followed the controller successfully.

The existing browser client is excellent for validating the workflow. However, for an application that can be distributed through the Meta Quest Store, I need a native Unity/Android client with its own UI, connection flow, camera layout, robot selection, and safe-disconnection behavior.

My understanding is that the current CloudXR 6.

https://github.com/user-attachments/assets/e5cf9dcf-1734-42d9-aee0-9f998278e5ad

x documentation supports Quest through CloudXR.js in the browser, but I could not find a supported, redistributable native Unity/Android client path compatible with CloudXR Runtime 6.x and Isaac Teleop.

Please correct me if an existing supported option already covers this use case.

Describe the solution you'd like

Ideally, NVIDIA would provide one of the following:

  1. A supported CloudXR Unity package or Android client SDK compatible with CloudXR Runtime 6.x; or
  2. A stable and documented client API, together with a minimal Unity/OpenXR sample.

The Quest client should ideally support:

  • receiving the low-latency streamed XR view;
  • displaying one or two real robot camera views;
  • sending HMD and Touch-controller poses and actions upstream;
  • Unity OpenXR integration;
  • connection, reconnection, latency, and streaming-quality status;
  • Grip-to-clutch and Trigger-to-gripper mappings;
  • a clear stop/hold state after clutch release or connection loss;
  • redistribution as part of a Meta Quest Store application.

A first MVP could be LAN-only:

Robot camera(s)
    → camera_viz / Televiz
    → CloudXR Runtime
    → Native Unity app on Quest

Quest controller poses and buttons
    → CloudXR
    → Isaac Teleop
    → LeRobot
    → SO-101

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing browser client and the examples/isaac_teleop_to_so101 workflow, along with the referenced CloudXR.js documentation. Determine whether a supported Unity/Android client path exists for CloudXR Runtime 6.x and Isaac Teleop. Done would require a documented or sample-supported native Quest path covering streaming, controller actions, robot camera views, and safe disconnection.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, unity
Domain
mobile, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.