Isaac Ros Foundation pose not getting a detection with real sense camera

Hi, we tested Isaac ros foundation pose with different objects with an Intel realsense d435i camera. We retrieved the usdz file from the AR Code app to get the cad model and Most of the time it was not getting a detection at all. So we had to move the camera around to get one detection and the detection seems off from rviz

We also tested it with the Foundation pose repo and it was working properly. The difference was that we manually made the segmantation mask for that model but not when we are using the realsense camera. My guess is that isaac ros foundation pose is not segmenting the object properly for us.
The model performance also seems well if we use the rosbag example from the website. However, once we use the realsense camera, the output seen in rviz is lagging and not updating the detection most of the time.
when we passed in the model of an orange stick of glue, it was tracking my elbow instead of the glue.

We would want to use isaac ros foundation pose to get the initial pose of the object as the perception system for the project.
I wonder if you guys have some insights on how to solve the problem I mentioned above? Thanks!

Start by reproducing the Isaac ROS FoundationPose pipeline with the Intel RealSense D435i input, then compare it with the working rosbag example and the FoundationPose run using a manually created segmentation mask. Check whether detections are reliable and whether the RViz pose updates without lag or tracking the wrong object.