NVIDIA-ISAAC-ROS / NVIDIA-ISAAC-ROS/isaac_ros_visual_slam

VisualSlamNode intermittently stops publishing odometry/status after 60-360 s while inputs keep flowing (3.2.6, AGX Orin, JetPack 6.2.3)

Open
#229 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.5k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Environment

  • isaac-ros-visual-slam 3.2.6-0jammy (apt), ROS 2 Humble, CycloneDDS
  • Jetson AGX Orin 64 GB, JetPack 6.2.3 (L4T 36.5.2), docker
  • VisualSlamNode composed in a component container; stereo gray/rect pairs at 30 fps from a ZED X, fed via plain rclcpp publishers on the remapped input topics (type-adapted sensor_msgs/Image); tested with enable_imu_fusion both true (IMU on visual_slam/imu at 200 Hz) and false

Symptom
In 4 of 5 independent launches on the same bench, the node silently stops publishing odometry and status after 60–360 s of clean 30 Hz tracking:

  • container stays alive, no error/exception in the log
  • log shows escalating inter-frame delta warnings right before the stall (66 ms → 533 ms → silence)
  • track_execution_time stays ~2.7 ms up to the stall (not compute-bound)
  • after the stall, CLI probes inside the same container still receive the input images and IMU normally — so the inputs and DDS transport are fine; the wedge is inside the node's input pipeline

Occurrences at ~280 s (fusion on), ~357 s, ~209 s, ~60 s (fusion off). One clean instance ran 780+ s including a deliberate exposure-blackout window, so it is intermittent, not deterministic.

Ruled out

  • Compute saturation (track_execution_time flat at ~2.7 ms)
  • Transport/DDS (post-stall probes still receive inputs in-container)
  • An early theory that CLI subscribers on the image topics triggered renegotiation does not survive the fusion-off occurrences, which had no image-topic probes at all

Impact
Bounds unattended session length for any externally-fed cuVSLAM deployment on Orin. Since Isaac ROS 4.x is Thor-only, Orin deployments have no upgrade path past 3.2.x, so a 3.2.x fix (or a diagnostic to narrow this) would be very valuable.

Questions

  1. Is this a known issue on the 3.2 branch?
  2. Are there internal watchdog/verbose flags that would expose where the input pipeline stops consuming?
  3. Any recommended recovery short of restarting the container?

Happy to provide full logs (launch logs incl. the frame-delta warning escalation, odometry CSVs cut at the stall, node-info dumps) and run instrumented builds/repro steps on our rig.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the VisualSlamNode input pipeline and reproduce the intermittent stall with the supplied stereo, IMU, and deployment setup. Compare the escalating frame-delta warnings with odometry/status output and the post-stall input probes; done means isolating the failure or identifying a diagnostic or recovery path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, linux
Domain
computer-vision, robotics
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.