NVIDIA-ISAAC-ROS / NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation
isaac_ros_foundationpose on zed camera
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 501
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am trying to run foundationpose on zed camera.
In the documentation, I don't see a column for zed camera, but there is for rosbag, realsense camera, and hawk camera. I managed to look at centerpose and compared the launch arguments.
So I ran this launch command
ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=zed_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_centerpose/zed2_quickstart_interface_specs.json
to run zed mono rect depth.
To visualize, I ran rviz2 -d $(ros2 pkg prefix isaac_ros_foundationpose --share)/rviz/foundationpose_zed.rviz, to visualize
The correct feed shows up but somehow when it detects my tumblr (even though I am running Mac and cheese?) the program shuts down by saying below
What would be the solution to this problem?
Also, do we have to only use DETR object detection? What if we want to use yolo model or other object detection models, how would I go about this?
Thank you so much for your help!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the FoundationPose documentation and reproduce the supplied isaac_ros_examples launch command using the zed_mono_rect_depth and foundationpose launch fragments. Inspect the reported shutdown output and the foundationpose_zed.rviz configuration, then compare the ZED setup with the documented rosbag, RealSense, and Hawk camera paths. Done means identifying the failure and clarifying ZED and alternative detector support.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, computer-vision, robotics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100