benchmark execution issues
Open
@jmatejcz is already working on this.
Since Jul 25, 2025.
bug
question
- Dominant language
- Python
- Stars
- 589
- Forks
- 76
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 3
Description
Describe the bug
Investigate issues that sometimes occurs when running benchmarks:
- Model (qwen3:14b for example) sometimes gets stuck on thinking till timeout
- When running long manipulation benchmark with large models that take a lot of RAM, the python process after couple hours, gets killed because of RAM usage, which suggests that there is some memory leakage.
- When a large model takes up a lot of VRAM, there is no space for openset models and program thrors CUDA OOM, should we adress that issue?
To Reproduce
- Run manipulation benchmark with qwen3:14b, no thinking
- Run manipulation benchmark with large model like 32GB on CPU
- Run manipulation benchmark with large model like 32GB on GPU
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Platform
- OS: Ubuntu 22.04
- ROS 2 Humble
Version
Release number or commit hash.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.