[Feature Request] Supported Unity/Android client path for a native Meta Quest robot teleoperation app
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 385
- Forks
- 88
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 82
Description
Is your feature request related to a problem? Please describe.
I would like to build a native Unity/OpenXR application for the Meta Quest Store that allows users to:
- view low-latency live video from a real robot;
- control the robot using Quest Touch controllers;
- use clutch, gripper, stop/hold, and disconnect controls inside the application.
I have already successfully tested the following real-hardware workflow:
Quest 3S Touch Plus controllers
→ CloudXR Runtime 6.2.0
→ NVIDIA Isaac Teleop 1.3.131
→ LeRobot v0.6.0 official examples/isaac_teleop_to_so101 example
→ SO-101 Follower robot
Test system:
- Ubuntu 24.04
- NVIDIA RTX 4080
- Quest 3S
- SO-101 robot arm
The Quest controller poses and buttons reached the Ubuntu host correctly. Grip/Squeeze worked as the clutch, Trigger controlled the gripper, and the SO-101 followed the controller successfully.
The existing browser client is excellent for validating the workflow. However, for an application that can be distributed through the Meta Quest Store, I need a native Unity/Android client with its own UI, connection flow, camera layout, robot selection, and safe-disconnection behavior.
My understanding is that the current CloudXR 6.
https://github.com/user-attachments/assets/e5cf9dcf-1734-42d9-aee0-9f998278e5ad
x documentation supports Quest through CloudXR.js in the browser, but I could not find a supported, redistributable native Unity/Android client path compatible with CloudXR Runtime 6.x and Isaac Teleop.
Please correct me if an existing supported option already covers this use case.
Describe the solution you'd like
Ideally, NVIDIA would provide one of the following:
- A supported CloudXR Unity package or Android client SDK compatible with CloudXR Runtime 6.x; or
- A stable and documented client API, together with a minimal Unity/OpenXR sample.
The Quest client should ideally support:
- receiving the low-latency streamed XR view;
- displaying one or two real robot camera views;
- sending HMD and Touch-controller poses and actions upstream;
- Unity OpenXR integration;
- connection, reconnection, latency, and streaming-quality status;
- Grip-to-clutch and Trigger-to-gripper mappings;
- a clear stop/hold state after clutch release or connection loss;
- redistribution as part of a Meta Quest Store application.
A first MVP could be LAN-only:
Robot camera(s)
→ camera_viz / Televiz
→ CloudXR Runtime
→ Native Unity app on Quest
Quest controller poses and buttons
→ CloudXR
→ Isaac Teleop
→ LeRobot
→ SO-101
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing browser client and the examples/isaac_teleop_to_so101 workflow, along with the referenced CloudXR.js documentation. Determine whether a supported Unity/Android client path exists for CloudXR Runtime 6.x and Isaac Teleop. Done would require a documented or sample-supported native Quest path covering streaming, controller actions, robot camera views, and safe disconnection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, unity
- Domain
- mobile, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100