Running into free(): double free detected in tcache 2 when using trtllm-bench in a multi-node scenario
Open
@Shixiaowei02 is already working on this.
Since Apr 9, 2025.
Investigating
triaged
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
I am trying to run distributed inference benchmarks for a large model (running across 4 nodes / 32 GPUs), and using trtllm-bench for the same. However, I run into this error, and I am not sure what fixes it
INFO - flashinfer.jit: Finished loading JIT ops: norm
free(): double free detected in tcache 2
*** Process received signal ***
Signal: Aborted (6)
Signal code: (-6)
[4] init.cc:720 NCCL WARN Duplicate GPU detected : rank 28 and rank 4 both on CUDA device *****
[ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x15555520c320]
[ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x155555265b1c]
[ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x15555520c26e]
[ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x1555551ef8ff]
[ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x1555551f07b6]
[ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8fe5)[0x15555526ffe5]
[ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab54f)[0x15555527254f]
[ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x155555274d9e]
8] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x1552ff26337e]
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.