NVIDIA / NVIDIA/TensorRT-LLM

Running into free(): double free detected in tcache 2 when using trtllm-bench in a multi-node scenario

Open
#2,953 32 comments 0 reactions 2 assignees View on GitHub

@Shixiaowei02 is already working on this.

Since Apr 9, 2025.

Investigating triaged
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

I am trying to run distributed inference benchmarks for a large model (running across 4 nodes / 32 GPUs), and using trtllm-bench for the same. However, I run into this error, and I am not sure what fixes it

INFO - flashinfer.jit: Finished loading JIT ops: norm
free(): double free detected in tcache 2
*** Process received signal ***
Signal: Aborted (6)
Signal code:  (-6)

[4] init.cc:720 NCCL WARN Duplicate GPU detected : rank 28 and rank 4 both on CUDA device *****
[ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x15555520c320]
[ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x155555265b1c]
[ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x15555520c26e]
[ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x1555551ef8ff]
[ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x1555551f07b6]
[ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8fe5)[0x15555526ffe5]
[ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab54f)[0x15555527254f]
[ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x155555274d9e]
 8] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x1552ff26337e]

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.