dimensionalOS / dimensionalOS/dimos

Tooling to send arm urdf to rerun

Open
#845 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good trial issue manipulation
Dominant language
Python
Stars
4.5k
Forks
808
Avg merge
3d 5h
Merged PRs (30d)
233

Description

Summary

Add URDF-based arm visualization to Rerun so manipulation can be visualized the same way navigation uses Rerun for maps — without depending on Drake/Meshcat.

Context

Currently, all manipulation visualization goes through Drake's Meshcat server (ManipulationModule with enable_viz=True). This is a separate browser window, disconnected from the rest of the dimOS observability stack. Meanwhile, navigation already visualizes maps, poses, and sensor data in Rerun — the standard dimOS visualization tool.

The goal is to load a robot arm URDF into Rerun's 3D view and update joint states in real time, so keyboard teleop, planning, and general manipulation can be visualized alongside camera feeds, point clouds, and other sensor data in a single Rerun session.

Key pointers

What Where
Rerun SDK docs for URDF Rerun supports rr.log("robot", rr.Asset3D(...)) and rr.Transform3D per link
Current Drake visualization dimos/manipulation/manipulation_module.pyenable_viz flag
Keyboard teleop blueprints (use Meshcat) dimos/robot/manipulators/xarm/blueprints.py
Joint state message dimos/msgs/sensor_msgs/JointState.py
URDF locations data/xarm_description/, data/openarm_description/, etc.
Rerun integration examples dimos/navigation/ — see how nav logs to Rerun
Coordinator publishes joint state ControlCoordinator with publish_joint_state=True

Deliverables

  • Utility/module that loads a URDF and logs the kinematic tree to Rerun (links as meshes, joints as rr.Transform3D)
  • Real-time joint state updates — subscribe to JointState from the coordinator and update link transforms each tick
  • Works as a drop-in replacement for ManipulationModule's Drake viz in keyboard teleop blueprints
  • At least one blueprint (e.g. keyboard_teleop_xarm7) running with Rerun visualization instead of Meshcat
  • Verify it works with at least two different URDFs (e.g. xArm7 and OpenArm)

Notes

  • Rerun has native support for mesh assets and hierarchical transforms — the approach is to log each URDF link as a mesh under a transform hierarchy and update joint transforms from FK.
  • You'll need forward kinematics to map JointState → per-link transforms. Pinocchio is already a dependency and can do this, or you can parse the URDF joint tree directly.
  • The coordinator already publishes JointState over LCM when publish_joint_state=True — the visualizer just needs to subscribe.
  • This should not remove Drake/Meshcat support — just provide a Rerun alternative that can be toggled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading dimos/manipulation/manipulation_module.py, dimos/robot/manipulators/xarm/blueprints.py, and the Rerun integration in dimos/navigation/. Trace JointState from ControlCoordinator and inspect the URDFs under data/xarm_description/ and data/openarm_description/. Done means a Rerun-based alternative supports live joint updates, works in a keyboard teleop blueprint, and is verified with two URDFs without removing Meshcat support.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.