Issue Persist - libtritonserver.so missing error , loadExtensions Error: GXF_FAILURE

agx-orin@agx-orin:~$ ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py

Run the above command and get the following error

[INFO] [launch]: All log files can be found below /home/agx-orin/.ros/log/2024-09-25-11-02-51-783634-agx-orin-72726
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [72739]
[component_container_mt-1] [INFO] [1727233372.330269030] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/lib/libresize_node.so
[component_container_mt-1] [INFO] [1727233372.379226314] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1727233372.379363181] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1727233372.384657529] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1727233372.384977375] [image_resize]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.386433629] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1727233372.394073112] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1727233372.398385072] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1727233372.403409622] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1727233372.408950055] [NitrosContext]: [NitrosContext] Loading application: '/home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1727233372.410260290] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233372.414354709] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2024-09-25 11:02:52.414 WARN  gxf/std/program.cpp@514: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1727233372.416194170] [image_resize]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.416252508] [image_resize]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.419034356] [image_resize]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.419152343] [image_resize]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.424359489] [image_resize]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.424474883] [image_resize]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.424957933] [image_resize]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.425791422] [image_resize]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1727233372.428377970] [image_resize]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1727233372.430388443] [image_resize]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1727233372.438524065] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1727233372.440148578] [image_resize]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.443976848] [image_resize]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.493788197] [image_resize]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.496638303] [image_resize]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.512293822] [image_resize]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/image_resize' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233372.519357837] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/lib/libdnn_image_encoder_node.so
[component_container_mt-1] [INFO] [1727233372.524754427] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1727233372.524862845] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1727233372.530387950] [dope_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.532631835] [dope_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.532706237] [dope_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.534896425] [dope_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.534977419] [dope_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.537223257] [dope_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.537330619] [dope_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.537810341] [dope_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.539977521] [dope_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1727233372.547792400] [dope_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.553611430] [dope_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.772957781] [dope_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.779016592] [dope_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.796975933] [dope_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dope_encoder' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233372.798867172] [image_resize]: Negotiating
[component_container_mt-1] [INFO] [1727233372.801147026] [dope_encoder]: Negotiating
[component_container_mt-1] [INFO] [1727233372.801253172] [dope_encoder]: Could not negotiate
[component_container_mt-1] [INFO] [1727233372.802279849] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_triton/lib/libtriton_node.so
[component_container_mt-1] [INFO] [1727233372.805071298] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1727233372.805131715] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1727233372.813943222] [dope_inference]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.817003157] [dope_inference]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233372.817070742] [dope_inference]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1727233372.817202073] [dope_inference]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.817225817] [dope_inference]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.822337857] [dope_inference]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.822447907] [dope_inference]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.827121122] [dope_inference]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.827204676] [dope_inference]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.827795760] [dope_inference]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.832051815] [dope_inference]: [NitrosContext] Loading extension: gxf/triton/libgxf_triton_ext.so
[component_container_mt-1] [ERROR] [1727233372.834654684] [dope_inference]: [NitrosContext] dlopen failed when opening "/home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_triton/share/isaac_ros_triton/gxf/triton/libgxf_triton_ext.so": libtritonserver.so: cannot open shared object file: No such file or directory
[component_container_mt-1] [ERROR] [1727233372.834745054] [dope_inference]: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1727233372.834981954] [dope_inference]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1727233372.835018435] [dope_inference]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] 2024-09-25 11:02:52.835 ERROR gxf/std/program.cpp@533: Attempted interrupting when not running (state=0).
[component_container_mt-1] 2024-09-25 11:02:52.835 ERROR gxf/core/runtime.cpp@1400: Graph interrupt failed with error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [ERROR] [1727233372.835097797] [dope_inference]: [NitrosContext] GxfGraphInterrupt Error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [INFO] [1727233372.835118213] [dope_inference]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1727233372.835150214] [dope_inference]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1727233372.835184231] [dope_inference]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1727233372.835225895] [dope_inference]: [NitrosNode] Application termination done
[component_container_mt-1] [ERROR] [1727233372.842207317] [dope_container]: Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'dope_inference' of type 'nvidia::isaac_ros::dnn_inference::TritonNode' in container '/dope_container': Component constructor threw an exception: [NitrosNode] loadExtensions Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1727233372.847150170] [dope_container]: Load Library: /home/agx-orin/workspaces/isaac_ros-dev/install/isaac_ros_dope/lib/libdope_decoder_node.so
[component_container_mt-1] [INFO] [1727233372.854090055] [dope_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dope::DopeDecoderNode>
[component_container_mt-1] [INFO] [1727233372.854179241] [dope_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dope::DopeDecoderNode>
[component_container_mt-1] [INFO] [1727233372.863758924] [dope_decoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1727233372.868125733] [dope_decoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1727233372.868309608] [dope_decoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1727233372.873930971] [dope_decoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1727233372.874028733] [dope_decoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1727233372.874800076] [dope_decoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1727233372.874841933] [dope_decoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1727233372.874869390] [dope_decoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1727233372.880137177] [dope_decoder]: [NitrosContext] Loading extension: gxf/lib/dope/libgxf_dope.so
[component_container_mt-1] [INFO] [1727233372.890965365] [dope_decoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1727233372.892724281] [dope_decoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1727233372.896013852] [dope_decoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1727233372.896348963] [dope_decoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1727233372.900684539] [dope_decoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dope_decoder' in container '/dope_container'
[component_container_mt-1] [INFO] [1727233373.515063117] [image_resize]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.515239856] [image_resize]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.515301585] [image_resize]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.515341874] [image_resize]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515365875] [image_resize]: [NitrosPublisher] Use only the compatible publisher: topic_name="/resize/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1727233373.515390931] [image_resize]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515412308] [image_resize]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1727233373.515434068] [image_resize]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.515485621] [image_resize]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1727233373.515593079] [image_resize]: [NitrosPublisherSubscriberGroup] Adjusted the compatible format of the component "input_compositor/image_in" (type="nvidia::gxf::DoubleBufferReceiver") from "nitros_image_bgr8" to "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.516048961] [image_resize]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.532879383] [image_resize]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/VIQFKZGMPC/VIQFKZGMPC.yaml"
[component_container_mt-1] [INFO] [1727233373.533028058] [image_resize]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.533128764] [image_resize]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.533163965] [image_resize]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/VIQFKZGMPC/VIQFKZGMPC.yaml'
[component_container_mt-1] [INFO] [1727233373.542136243] [image_resize]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.542612765] [image_resize]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.542671614] [image_resize]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.542866114] [image_resize]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.797827684] [dope_encoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.797980999] [dope_encoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.798019816] [dope_encoder]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.798045001] [dope_encoder]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_pub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233373.798073609] [dope_encoder]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1727233373.799566088] [dope_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.868288414] [image_resize]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.868732007] [image_resize]: [NitrosNode] Starting a heartbeat timer (eid=17)
[component_container_mt-1] [INFO] [1727233373.901425792] [dope_decoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1727233373.901550850] [dope_decoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1727233373.901570723] [dope_decoder]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.901588835] [dope_decoder]: [NitrosPublisher] Use only the compatible publisher: topic_name="/poses", data_format="nitros_pose_array"
[component_container_mt-1] [INFO] [1727233373.901600419] [dope_decoder]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1727233373.901609827] [dope_decoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1727233373.901848424] [dope_decoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1727233373.904317242] [dope_decoder]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/QAOLSFIFQD/QAOLSFIFQD.yaml"
[component_container_mt-1] [INFO] [1727233373.904385756] [dope_decoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.904398428] [dope_decoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.904412412] [dope_decoder]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/QAOLSFIFQD/QAOLSFIFQD.yaml'
[component_container_mt-1] [INFO] [1727233373.906357892] [dope_decoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.906511079] [dope_decoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.906601385] [dope_decoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.908811798] [dope_decoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.909311424] [dope_decoder]: [NitrosNode] Starting a heartbeat timer (eid=67)
[component_container_mt-1] [INFO] [1727233373.911403179] [dope_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/WFJKNUFLZT/WFJKNUFLZT.yaml"
[component_container_mt-1] [INFO] [1727233373.911514989] [dope_encoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1727233373.911545134] [dope_encoder]: In DNN Image Encoder Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.911627119] [dope_encoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1727233373.911648016] [dope_encoder]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/WFJKNUFLZT/WFJKNUFLZT.yaml'
[component_container_mt-1] [INFO] [1727233373.917195840] [dope_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1727233373.917432837] [dope_encoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1727233373.917458534] [dope_encoder]: In DNN Image Encoder Node postLoadGraphCallback().
[component_container_mt-1] [INFO] [1727233373.917694859] [dope_encoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1727233373.944590862] [dope_encoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1727233373.944967477] [dope_encoder]: [NitrosNode] Starting a heartbeat timer (eid=98)

Have tried the solution suggested in https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation/issues/19 , but the issue still persist. Any solution for this ?

Best
Samuel

Start by reproducing ros2 launch isaac_ros_dope isaac_ros_dope_triton.launch.py and inspect the Triton node loading path shown in the log, especially gxf/triton/libgxf_triton_ext.so. Confirm why libtritonserver.so is unavailable in the launch environment; done means the Triton node loads without GXF_FAILURE and the launch completes successfully.