Memory leak in exploration_node causes severe slowdown during full HM3D v2 benchmark (~14 GB RSS, 30+ min/episode after ~400 eps)

While running the full HM3D v2 val benchmark (1000 episodes) with habitat_evaluation.py, I observed severe slowdown over time caused by what appears to be a memory leak in the C++ exploration_node. After ~420 episodes the planner's RSS grew to ~14 GB and per-episode time went from ~37s (episode 0) to 30+ minutes (episode ~420). Restarting only the ROS 2 planner (while keeping VLM servers and habitat_evaluation.py's continue.txt checkpoint) brought per-episode time back to ~60s and RSS down to ~500 MB.

Environment

• OS: Ubuntu 22.04 (jammy) — note: ROS 2 Jazzy is Tier 3 here, source-built per official docs
• ROS 2: Jazzy (built from source against the conda env's Python)
• conda env: apexnav (Python 3.9, per apexnav_environment.yaml)
• Branch: ros2-jazzy
• GPU: NVIDIA RTX 4090, CUDA 12.8, PyTorch 2.8.0+cu128
• VLM servers (default ports 12181-12184): all running fresh, never restarted during the run
• Command:

  python habitat_evaluation.py --dataset hm3dv2 \
      habitat.dataset.split=val \
      need_video=true \
      test_epi_num=-1
  

Symptoms

After episode ~420 the system began swapping (60 GB total RAM, planner alone holding ~14 GB), turning the 60-second exploration loops into 30+ minutes due to OS-level page thrashing.

Reproduction

1. Run the full HM3D v2 val split with test_epi_num=-1 (1000 episodes) and need_video=true.
2. Periodically observe RSS of exploration_node via:

   ps -o pid,rss,etime,cmd -p $(pgrep -f exploration_node)
   

3. Check the cumulative seconds spend in this task field in videos/test_hm3dv2_val/record.txt — the per-episode delta grows roughly monotonically from ~40s to thousands of seconds.

Workaround (works, but obviously not ideal)

I wrote a tmux watchdog that restarts only the ROS 2 planner every 200 episodes (or whenever its RSS exceeds 5 GB), while keeping the 4 VLM servers and habitat_evaluation.py running. Because continue.txt checkpoints per-episode, evaluation resumes seamlessly from where it left off, and per-episode time returns to ~30-60s after each restart.

Possible Causes

In exploration_fsm.cpp lines 14-20 the FSM code does fp_.reset(...), fd_.reset(...), expl_manager_.reset(...) etc. on each episode reset, which suggests inter-episode reset was intended. But something downstream — possibly:

• a shared_ptr to point clouds / occupancy grids / SDF map is held by another component (visualization? mapping? planner cache?) and never released
• PCL / Octomap-style data structures that grow but never shrink
• accumulated frontiers / TF buffer / parameter listeners

—keeps the previous episode's allocations alive across resets.

Thanks for the great work.

Start with exploration_fsm.cpp lines 14-20 and trace the episode-reset calls from exploration_node. Reproduce the HM3D v2 val run with habitat_evaluation.py and monitor RSS plus the per-episode times in videos/test_hm3dv2_val/record.txt. Done means memory remains bounded across episodes without restarting the planner and slowdown no longer accumulates.