dimensionalOS / dimensionalOS/dimos
VisionPro Teleop
Open
@ruthwikdasyam is already working on this.
Since Feb 12, 2026.
manipulation
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 808
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 233
Description
Summary
Add a VisionPro teleop module for dimOS, following the same architecture as the existing MetaQuest teleop integration.
Context
dimOS already has a full MetaQuest teleop pipeline in dimos/teleop/quest/. The architecture is:
- Teleop module — Runs a FastAPI WebSocket server that receives controller poses from a WebXR app in the browser. Converts WebXR coordinates to robot frame, computes deltas on press-and-hold engagement, and publishes
PoseStamped+Buttonsover LCM. - ControlCoordinator — Receives the pose deltas and runs a
TeleopIKTaskper arm to solve IK and output joint commands. - Subclasses —
TwistTeleopModule(for mobile base) andArmTeleopModule(for arm control with per-hand task routing and analog gripper triggers).
The VisionPro equivalent should follow the same pattern but adapt to the VisionPro's input model (hand tracking, eye tracking, pinch gestures vs. physical controllers).
Key pointers
| What | Where |
|---|---|
| Quest teleop module (reference) | dimos/teleop/quest/quest_teleop_module.py |
| Quest types (controller state, buttons) | dimos/teleop/quest/quest_types.py |
| Quest extensions (Twist, Arm subclasses) | dimos/teleop/quest/quest_extensions.py |
| Quest blueprints | dimos/teleop/quest/blueprints.py |
| VR coordinate transforms | dimos/teleop/utils/teleop_transforms.py |
| WebXR client (browser app) | dimos/teleop/quest/web/static/index.html |
| TeleopIK task (coordinator side) | dimos/control/tasks/teleop_task.py |
| Coordinator teleop blueprints | dimos/control/blueprints/teleop.py |
Deliverables
-
VisionProTeleopModuleindimos/teleop/visionpro/— WebSocket server receiving hand/pose data from a VisionPro WebXR client - Coordinate transform from VisionPro frame to robot frame (similar to
webxr_to_robot) - Engagement model adapted for VisionPro input (pinch gestures instead of physical buttons)
-
ArmTeleopModule-equivalent for per-hand arm routing with gripper control - WebXR client app that runs in VisionPro Safari (
dimos/teleop/visionpro/web/) - At least one blueprint (e.g.
teleop_visionpro_xarm7) registered and runnable - Analog hand openness mapped to gripper position (pinch = close, open hand = open)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.