Visual Servoing with hand in eye

Summary

Implement visual servoing with eye-in-hand camera for dynamic marker tracking and manipulation. Build on the work in the ruthwik_dynamic_tracking branch, which has ArUco tracking and Pinocchio IK but needs to be synced with current main, fully calibrated, and integrated with the RealSense module and manipulation coordinator.

Context

The ruthwik_dynamic_tracking branch has an initial implementation:

• ArUco tracker (dimos/manipulation/dynamic_tracking/aruco_tracker.py) — detects ArUco markers, estimates camera-to-marker pose, publishes transforms, visualizes in Rerun
• Pinocchio IK (dimos/manipulation/pinocchio/pin_kinematics.py) — forward/inverse kinematics for servo-rate control
• Joint stream task (dimos/control/tasks/joint_stream_task.py) — joint-level servo streaming to the coordinator
• Static calibration (dimos/hardware/sensors/camera/realsense/handeyeout_xarm6/calibration.json) — a single EEF→RealSense extrinsic transform

The branch is out of sync with main and the pieces aren't fully wired together through the coordinator and TF tree.

Key pointers

Deliverables

• Rebase the ruthwik_dynamic_tracking work onto current main
• Complete eye-in-hand calibration pipeline (EEF → camera transform) with a calibration tool/script
• Full TF tree: base → joints → EEF → camera → marker, so any frame can be looked up relative to any other
• ArUco marker detection and pose estimation integrated with the RealSense module (color + depth)
• Visual servoing control loop: marker pose error → Cartesian velocity → IK → joint commands, running through the ControlCoordinator
• Blueprint that wires RealSense + ArUco tracker + coordinator for at least one arm (e.g. xArm6)
• One line run command to test