InternRobotics / InternRobotics/NavDP

[BUG] Memory Overflow and Thread Error in `MWF4WLIKTIFZIAABAAAAADY8_usd` Scene

Open
#58 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
793
Forks
69
PR merge metrics
No merged PRs in 30d

Description

Description:
I encountered an issue when testing the NavDP model with the MWF4WLIKTIFZIAABAAAAADY8_usd scene in scenes_commercial. The memory usage spikes to 80GB during the test, whereas other scenes only use around 10GB. After this, the following errors appear:

2025-11-20 08:11:29 [1,286,639ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for Storage status: 1
http://localhost:8002/navigator_reset
No trajectory available, using zero action
No trajectory available, using zero action
No trajectory available, using zero action   [ERROR:0@1278.827] global parallel_impl.cpp:244 WorkerThread 54: Can't spawn new thread: res[0/1840]
LLVM ERROR: out of memory
[ERROR:0@1278.827] global parallel_impl.cpp:244 WorkerThread 55: Can't spawn new thread: res = 11
[ERROR:0@1278.829] global parallel_impl.cpp:244 WorkerThread 56: Can't spawn new thread: res = 11
[ERROR:0@1278.829] global parallel_impl.cpp:244 WorkerThread 57: Can't spawn new thread: res = 11
[ERROR:0@1278.829] global parallel_impl.cpp:244 WorkerThread 58: Can't spawn new thread: res = 11
[ERROR:0@1278.829] global parallel_impl.cpp:244 WorkerThread 59: Can't spawn new thread: res = 11
LLVM ERROR: out of memory
Aborted (core dumped)

Steps to Reproduce:

  1. Clone the NavDP repository and set up the environment.

  2. Download and set up the scenes_commercial/MWF4WLIKTIFZIAABAAAAADY8_usd scene

  3. Run the model with the scenes_commercial/MWF4WLIKTIFZIAABAAAAADY8_usd scene using the following command:

    python eval_pointgoal_wheeled.py --port 8000 --scene_dir assets/scenes/internscenes_commercial/scenes_commercial --scene_index 17 --scene_scale 0.01 
    

The modefied part of eval_pointgoal_wheeled.py is as follows:

all_scenes = sorted(os.listdir(args_cli.scene_dir))
scene_path = os.path.join(args_cli.scene_dir,all_scenes[args_cli.scene_index]) + "/"
  1. Observe the error and the program aborts.

Additional Information:

  • This issue only occurs in the MWF4WLIKTIFZIAABAAAAADY8_usd scene. Other scenes, like those in the cluttered_easy or internscenes_commercial directories, do not show this behavior.
  • It appears to be related to the memory management or threading during the model's operation.

Environment:

  • Operating System: Ubuntu 22.04.5 LTS
  • GPU: NVIDIA GeForce RTX 4090
  • Memory: 1 TB

Request:
Could you please advise on possible solutions for this issue? Any suggestions for addressing this issue would be appreciated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with eval_pointgoal_wheeled.py and reproduce the command using scene_index 17 and scene_scale 0.01. Compare memory and thread behavior with the unaffected scenes, then trace the scene-specific failure shown in the logs; done means identifying the cause and providing a verified fix or workaround for MWF4WLIKTIFZIAABAAAAADY8_usd.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, robotics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.