Could not run on a machine with dual RTX 5090s, using WSL2 and Docker

System Info

Greetings, everyone!

I encountered an issue while attempting to run TensorRT-LLM with two NVIDIA GeForce RTX 5090 GPUs, utilizing a tensor parallelism size of 2.

My system configuration and software versions are as follows:

AMD Ryzen Threadripper 7960X 24-Cores
Two NVIDIA GeForce RTX 5090  
The main machine is Windows 11 with WSL2
Docker image: tritonserver:25.02-trtllm-python-py3
TensorRT-LLM version: 0.17.0.post1

The command used to reproduce the error was:

trtllm-serve ./Qwen2.5-7B-Instruct-engine \
--tokenizer ./Qwen2.5-7B-Instruct  \
--tp_size 2 \
--host 0.0.0.0 \
--port 5001		

And got these error logs:

[TensorRT-LLM][INFO] Engine version 0.17.0.post1 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.17.0.post1 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] MPI size: 2, MPI local size: 2, rank: 1
[TensorRT-LLM][INFO] MPI size: 2, MPI local size: 2, rank: 0
[TensorRT-LLM][INFO] Rank 1 is using GPU 1
[TensorRT-LLM][INFO] Rank 0 is using GPU 0
[TensorRT-LLM][WARNING] Fix optionalParams : KV cache reuse disabled because model was not built with paged context FMHA support
[TensorRT-LLM][WARNING] Fix optionalParams : KV cache reuse disabled because model was not built with paged context FMHA support
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 2048
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 2048
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 32768
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 2048
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 2048
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 32768
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: (32768) * 28
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: (32768) * 28
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 0
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 8192
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 8192 = min(maxSequenceLen - 1, maxNumTokens) since context FMHA and usePackedInput are enabled
[TensorRT-LLM][INFO] TRTGptModel If model type is encoder, maxInputLen would be reset in trtEncoderModel to maxInputLen: min(maxSequenceLen, maxNumTokens).
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 0
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 8192
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 8192 = min(maxSequenceLen - 1, maxNumTokens) since context FMHA and usePackedInput are enabled
[TensorRT-LLM][INFO] TRTGptModel If model type is encoder, maxInputLen would be reset in trtEncoderModel to maxInputLen: min(maxSequenceLen, maxNumTokens).
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][INFO] Loaded engine size: 7804 MiB
[TensorRT-LLM][INFO] Loaded engine size: 7804 MiB
free(): double free detected in tcache 2
[DESKTOP-5MQV55Q:19090] *** Process received signal ***
[DESKTOP-5MQV55Q:19090] Signal: Aborted (6)
[DESKTOP-5MQV55Q:19090] Signal code:  (-6)
[DESKTOP-5MQV55Q:19090] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7f9912877330]
[DESKTOP-5MQV55Q:19090] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7f99128d0b2c]
[DESKTOP-5MQV55Q:19090] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7f991287727e]
[DESKTOP-5MQV55Q:19090] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7f991285a8ff]
[DESKTOP-5MQV55Q:19090] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x7f991285b7b6]
[DESKTOP-5MQV55Q:19090] [ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8ff5)[0x7f99128daff5]
[DESKTOP-5MQV55Q:19090] [ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab55f)[0x7f99128dd55f]
[DESKTOP-5MQV55Q:19090] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x7f99128dfdae]
[DESKTOP-5MQV55Q:19090] [ 8] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x7f9605c6337e]
[DESKTOP-5MQV55Q:19090] [ 9] /usr/lib/x86_64-linux-gnu/libnccl.so.2(pncclCommInitRank+0x153)[0x7f9605c65513]
[DESKTOP-5MQV55Q:19090] [10] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_Z7getCommRKSt3setIiSt4lessIiESaIiEE+0x5f9)[0x7f95816065e9]
[DESKTOP-5MQV55Q:19090] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZN12tensorrt_llm7plugins15AllreducePlugin10initializeEv+0xc5)[0x7f948e01fc05]
[DESKTOP-5MQV55Q:19090] [12] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1eda698)[0x7f96c7b0c698]
[DESKTOP-5MQV55Q:19090] [13] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e0c2d1)[0x7f96c7a3e2d1]
[DESKTOP-5MQV55Q:19090] [14] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e12cb8)[0x7f96c7a44cb8]
[DESKTOP-5MQV55Q:19090] [15] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e13acc)[0x7f96c7a45acc]
[DESKTOP-5MQV55Q:19090] [16] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e14260)[0x7f96c7a46260]
[DESKTOP-5MQV55Q:19090] [17] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e4e9c8)[0x7f96c7a809c8]
[DESKTOP-5MQV55Q:19090] [18] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e4f70a)[0x7f96c7a8170a]
[DESKTOP-5MQV55Q:19090] [19] /usr/local/tensorrt/lib/libnvinfer.so.10(+0x1e4f87c)[0x7f96c7a8187c]
[DESKTOP-5MQV55Q:19090] [20] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm7runtime11TllmRuntimeC1ERKNS0_9RawEngineEPN8nvinfer17ILoggerEfb+0x585)[0x7f95818751f5]
[DESKTOP-5MQV55Q:19090] [21] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager27TrtGptModelInflightBatchingC1ESt10shared_ptrIN8nvinfer17ILoggerEERKNS_7runtime11ModelConfigERKNS6_11WorldConfigERKNS6_9RawEngineEbRKNS0_25TrtGptModelOptionalParamsE+0x56a)[0x7f95822357ca]
[DESKTOP-5MQV55Q:19090] [22] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager18TrtGptModelFactory6createERKNS_7runtime9RawEngineERKNS2_11ModelConfigERKNS2_11WorldConfigENS0_15TrtGptModelTypeERKNS0_25TrtGptModelOptionalParamsE+0x1e9)[0x7f95821aedf9]
[DESKTOP-5MQV55Q:19090] [23] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4Impl11createModelERKNS_7runtime9RawEngineERKNS3_11ModelConfigERKNS3_11WorldConfigERKNS0_14ExecutorConfigE+0xb9)[0x7f95822cf369]
[DESKTOP-5MQV55Q:19090] [24] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4Impl9loadModelERKSt8optionalINSt10filesystem7__cxx114pathEERKS3_ISt17basic_string_viewIhSt11char_traitsIhEEERKNS_7runtime13GptJsonConfigERKNS0_14ExecutorConfigEbRKS3_ISt3mapINSt7__cxx1112basic_stringIcSB_IcESaIcEEENS0_6TensorESt4lessIST_ESaISt4pairIKST_SU_EEEE+0x4cd)[0x7f95822cf9fd]
[DESKTOP-5MQV55Q:19090] [25] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8Executor4ImplC1ERKNSt10filesystem7__cxx114pathERKSt8optionalIS5_ENS0_9ModelTypeERKNS0_14ExecutorConfigE+0x9aa)[0x7f95822d0c4a]
[DESKTOP-5MQV55Q:19090] [26] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm8executor8ExecutorC2ERKNSt10filesystem7__cxx114pathENS0_9ModelTypeERKNS0_14ExecutorConfigE+0x57)[0x7f95822b66d7]
[DESKTOP-5MQV55Q:19090] [27] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x1565d0)[0x7f95b9e905d0]
[DESKTOP-5MQV55Q:19090] [28] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0xa43af)[0x7f95b9dde3af]
[DESKTOP-5MQV55Q:19090] [29] /usr/bin/python3[0x58208f]
[DESKTOP-5MQV55Q:19090] *** End of error message ***
--------------------------------------------------------------------------
Child job 2 terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

python3 /workspace/TensorRT-LLM/examples/llama/convert_checkpoint.py
--model_dir ./Llama-3.1-8B-Instruct
--output_dir ./Llama-3.1-8B-Instruct-convert
--tp_size 2
--dtype bfloat16

trtllm-build
--checkpoint_dir ./Llama-3.1-8B-Instruct-convert
--gpt_attention_plugin bfloat16
--gemm_plugin bfloat16
--output_dir ./Llama-3.1-8B-Instruct-engine

trtllm-serve ./Qwen2.5-7B-Instruct-engine
--tokenizer ./Qwen2.5-7B-Instruct
--tp_size 2
--host 0.0.0.0
--port 5001

Expected behavior

N/A

actual behavior

N/A

additional notes

N/A

Start with the listed examples/llama/convert_checkpoint.py, trtllm-build, and trtllm-serve commands, resolving the mismatch between the Qwen paths and Llama reproduction. Reproduce on the dual-GPU WSL2 Docker setup and trace the abort at NCCL pncclCommInitRank during model loading. Done means the dual-GPU server starts without the reported double-free abort.