[foundationpose] Severe frame drops in RViz camera view when publishing simulated camera image from IsaacLab (RTX 3090)

Description

I'm currently running isaac_ros_foundationpose to estimate object poses using camera images published from an IsaacLab simulation environment. While the model itself seems to be working, RViz2 exhibits severe frame drops in the camera image view (sensor_msgs/Image topic). The display becomes choppy and refreshes at an inconsistent and very low frame rate (~1–2 FPS or less), despite a capable GPU (RTX 3090).

Setup Details

• GPU: NVIDIA RTX 3090
• Simulation: IsaacLab (Isaac Sim via omni.isaac.lab)
• Camera publisher: Python-based ROS2 publisher sending /image_rect, /depth, and /camera_info from IsaacLab to ROS2
• FoundationPose launch: Using official isaac_ros_foundationpose container
• RViz2 Settings:
  • Fixed Frame: tf_camera
  • Topics subscribed: /detections, /image_rect, /camera_info
  • QoS: Best Effort
• Camera resolution: 600×800, 15 FPS (manually throttled)

Problem Summary

• When FoundationPose is active, RViz2's camera image view (/image_rect) becomes extremely laggy.
• Without FoundationPose running, RViz2 displays the camera image smoothly at the expected frame rate.
• RViz2 warning: No tf data occasionally appears before catching up.
• CPU/GPU utilization appears normal (~20–30% GPU).

Expected Behavior

• Camera image in RViz should display smoothly while FoundationPose is running, assuming reasonable frame rates and hardware (RTX 3090).

What I’ve Tried

• Lowering resolution and FPS of the published image stream
• Using CompressedImage instead of raw image
• Reducing RViz update rate
• Confirming correct QoS profile on both publisher and subscriber
• Changing from Docker GUI forwarding to native host RViz2 (same result)

Possible Hypothesis

• FoundationPose pipeline might be blocking or slowing down image subscribers in the same ROS graph?
• TF tree broadcast from IsaacLab might not be synchronized tightly with detection callbacks
• GPU memory bandwidth contention?

Request

Please advise whether:

• There are known issues regarding FoundationPose interfering with sensor_msgs/Image topics in RViz2
• There are recommended practices for handling simulated camera input with FoundationPose
• Any performance profiling tools or pipeline separation methods (e.g., using separate nodes or callbacks) are recommended

Thanks in advance for your help!

Start by reproducing the issue with the IsaacLab Python publisher, the official FoundationPose container, and native or containerized RViz2 using the stated 600×800, 15 FPS setup. Profile the /image_rect, /depth, /camera_info, /detections, and TF flow to determine where the frame-rate loss occurs. Done means identifying a reproducible cause and documenting a validated mitigation or separation approach.