Llama4 scout is not starting up on H100

System Info

x86_64
DGX H100 system
TensorRT-LLM
Driver 550.90.07
Ubuntu 24.04

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the example here but with meta-llama/Llama-4-Scout-17B-16E-Instruct with tp 4.

Expected behavior

Model should startup successfully.

actual behavior

The 4 executor processes keep running at 100 cpu for a while but then crash with the following error:

free(): double free detected in tcache 2
[m-0:02554] *** Process received signal ***
[m-0:02554] Signal: Aborted (6)
[m-0:02554] Signal code:  (-6)
free(): double free detected in tcache 2
[m-0:02553] *** Process received signal ***
[m-0:02553] Signal: Aborted (6)
[m-0:02553] Signal code:  (-6)
[m-0:02553] [ 0] free(): double free detected in tcache 2
[m-0:02555] *** Process received signal ***
[m-0:02555] Signal: Aborted (6)
[m-0:02555] Signal code:  (-6)
[m-0:02555] [ 0] [m-0:02554] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7edf9822a330]
[m-0:02554] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7f48f636e330]
[m-0:02555] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x79c6836ae330]
[m-0:02553] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x79c683707b2c]
[m-0:02553] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7f48f63c7b2c]
[m-0:02555] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7edf98283b2c]
[m-0:02554] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7edf9822a27e]
[m-0:02554] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x79c6836ae27e]
[m-0:02553] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7f48f636e27e]
[m-0:02555] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7f48f63518ff]
[m-0:02555] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7edf9820d8ff]
[m-0:02554] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x79c6836918ff]
[m-0:02553] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x79c6836927b6]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x7f48f63527b6]
[m-0:02555] [ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x7edf9820e7b6]
[m-0:02554] [ 5] [m-0:02553] [ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8ff5)[0x7f48f63d1ff5]
[m-0:02555] [ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8ff5)[0x7edf9828dff5]
[m-0:02554] [ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8ff5)[0x79c683711ff5]
[m-0:02553] [ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab55f)[0x7f48f63d455f]
[m-0:02555] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x7f48f63d6dae]
[m-0:02555] [ 8] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab55f)[0x79c68371455f]
[m-0:02553] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x79c683716dae]
[m-0:02553] [ 8] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab55f)[0x7edf9829055f]
[m-0:02554] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x7edf98292dae]
[m-0:02554] [ 8] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x7f46b926337e]
[m-0:02555] [ 9] /usr/lib/x86_64-linux-gnu/libnccl.so.2(pncclCommInitRank+0x153)[0x7f46b9265513]
[m-0:02555] [10] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x7edd5d26337e]
[m-0:02554] [ 9] /usr/lib/x86_64-linux-gnu/libnccl.so.2(pncclCommInitRank+0x153)[0x7edd5d265513]
[m-0:02554] [10] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x79c44866337e]
[m-0:02553] [ 9] /usr/lib/x86_64-linux-gnu/libnccl.so.2(pncclCommInitRank+0x153)[0x79c448665513]
[m-0:02553] [10] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_Z7getCommRKSt3setIiSt4lessIiESaIiEE+0x5f9)[0x7edb68ed0089]
[m-0:02554] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_Z7getCommRKSt3setIiSt4lessIiESaIiEE+0x5f9)[0x79c2543d0089]
[m-0:02553] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_Z7getCommRKSt3setIiSt4lessIiESaIiEE+0x5f9)[0x7f44c6fd0089]
[m-0:02555] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN9torch_ext9allreduceEN2at6TensorESt8optionalIS1_EN3c108ArrayRefIS1_EENS4_4ListIlEEllldbbb+0x404)[0x7ed9df8ed874]
[m-0:02554] [12] free(): double free detected in tcache 2
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFSt6vectorIN2at6TensorESaIS6_EES6_St8optionalIS6_ENS_8ArrayRefIS6_EENS_4ListIlEEllldbbbES8_NS_4guts8typelist8typelistIJS6_SA_SC_SE_llldbbbEEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPS4_INS_6IValueESaIST_EE+0x79c)[0x7ed9df8f1c4c]
[m-0:02554] [13] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN9torch_ext9allreduceEN2at6TensorESt8optionalIS1_EN3c108ArrayRefIS1_EENS4_4ListIlEEllldbbb+0x404)[0x7f433d9ed874]
[m-0:02555] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN9torch_ext9allreduceEN2at6TensorESt8optionalIS1_EN3c108ArrayRefIS1_EENS4_4ListIlEEllldbbb+0x404)[0x79c0caded874]
[m-0:02553] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFSt6vectorIN2at6TensorESaIS6_EES6_St8optionalIS6_ENS_8ArrayRefIS6_EENS_4ListIlEEllldbbbES8_NS_4guts8typelist8typelistIJS6_SA_SC_SE_llldbbbEEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPS4_INS_6IValueESaIST_EE+0x79c)[0x7f433d9f1c4c]
[m-0:02555] [13] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFSt6vectorIN2at6TensorESaIS6_EES6_St8optionalIS6_ENS_8ArrayRefIS6_EENS_4ListIlEEllldbbbES8_NS_4guts8typelist8typelistIJS6_SA_SC_SE_llldbbbEEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPS4_INS_6IValueESaIST_EE+0x79c)[0x79c0cadf1c4c]
[m-0:02553] [13] [m-0:02552] *** Process received signal ***
[m-0:02552] Signal: Aborted (6)
[m-0:02552] Signal code:  (-6)
[m-0:02552] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7d4b6eb11330]
[m-0:02552] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7d4b6eb6ab2c]
[m-0:02552] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7d4b6eb1127e]
[m-0:02552] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7d4b6eaf48ff]
[m-0:02552] [ 4] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x297b6)[0x7d4b6eaf57b6]
[m-0:02552] [ 5] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xa8ff5)[0x7d4b6eb74ff5]
[m-0:02552] [ 6] /usr/lib/x86_64-linux-gnu/libc.so.6(+0xab55f)[0x7d4b6eb7755f]
[m-0:02552] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x7e)[0x7d4b6eb79dae]
[m-0:02552] [ 8] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x6437e)[0x7d492926337e]
[m-0:02552] [ 9] /usr/lib/x86_64-linux-gnu/libnccl.so.2(pncclCommInitRank+0x153)[0x7d4929265513]
[m-0:02552] [10] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x55a6277)[0x79c356e7a277]
[m-0:02553] [14] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x55a6277)[0x7f45c9a7a277]
[m-0:02555] [14] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x55a6277)[0x7edc6b87a277]
[m-0:02554] [14] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_Z7getCommRKSt3setIiSt4lessIiESaIiEE+0x5f9)[0x7d47377d0089]
[m-0:02552] [11] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit24invokeOperatorFromPythonERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EERKN8pybind114argsERKNS9_6kwargsESt8optionalIN3c1011DispatchKeyEE+0xef)[0x79c35f14741f]
[m-0:02553] [15] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit24invokeOperatorFromPythonERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EERKN8pybind114argsERKNS9_6kwargsESt8optionalIN3c1011DispatchKeyEE+0xef)[0x7edc73b4741f]
[m-0:02554] [15] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit24invokeOperatorFromPythonERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EERKN8pybind114argsERKNS9_6kwargsESt8optionalIN3c1011DispatchKeyEE+0xef)[0x7f45d1d4741f]
[m-0:02555] [15] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN9torch_ext9allreduceEN2at6TensorESt8optionalIS1_EN3c108ArrayRefIS1_EENS4_4ListIlEEllldbbb+0x404)[0x7d45ae1ed874]
[m-0:02552] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFSt6vectorIN2at6TensorESaIS6_EES6_St8optionalIS6_ENS_8ArrayRefIS6_EENS_4ListIlEEllldbbbES8_NS_4guts8typelist8typelistIJS6_SA_SC_SE_llldbbbEEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPS4_INS_6IValueESaIST_EE+0x79c)[0x7d45ae1f1c4c]
[m-0:02552] [13] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit37_get_operation_for_overload_or_packetERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EEN3c106SymbolERKN8pybind114argsERKNSB_6kwargsEbSt8optionalINS9_11DispatchKeyEE+0x229)[0x79c35f1476c9]
[m-0:02553] [16] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit37_get_operation_for_overload_or_packetERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EEN3c106SymbolERKN8pybind114argsERKNSB_6kwargsEbSt8optionalINS9_11DispatchKeyEE+0x229)[0x7edc73b476c9]
[m-0:02554] [16] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit37_get_operation_for_overload_or_packetERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EEN3c106SymbolERKN8pybind114argsERKNSB_6kwargsEbSt8optionalINS9_11DispatchKeyEE+0x229)[0x7f45d1d476c9]
[m-0:02555] [16] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x8b5fd1)[0x79c35f057fd1]
[m-0:02553] [17] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x8b5fd1)[0x7f45d1c57fd1]
[m-0:02555] [17] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x8b5fd1)[0x7edc73a57fd1]
[m-0:02554] [17] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x42238d)[0x79c35ebc438d]
[m-0:02553] [18] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x42238d)[0x7edc735c438d]
[m-0:02554] [18] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x42238d)[0x7f45d17c438d]
[m-0:02555] [18] /usr/bin/python[0x5820ff]
[m-0:02554] [19] /usr/bin/python[0x5820ff]
[m-0:02553] [19] /usr/bin/python[0x5820ff]
[m-0:02555] [19] /usr/bin/python(PyObject_Call+0x9c)[0x54b07c]
[m-0:02553] [20] /usr/bin/python(PyObject_Call+0x9c)[0x54b07c]
[m-0:02555] [20] /usr/bin/python(PyObject_Call+0x9c)[0x54b07c]
[m-0:02554] [20] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02553] [21] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02554] [21] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02555] [21] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02554] [22] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02553] [22] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02555] [22] /usr/bin/python[0x5a3698]
[m-0:02554] [23] /usr/bin/python[0x5a3698]
[m-0:02555] [23] /usr/bin/python(_PyObject_MakeTpCall+0x75)[0x548ec5]
[m-0:02555] [24] /usr/bin/python[0x5a3698]
[m-0:02553] [23] /usr/bin/python(_PyObject_MakeTpCall+0x75)[0x548ec5]
[m-0:02553] [24] /usr/bin/python(_PyObject_MakeTpCall+0x75)[0x548ec5]
[m-0:02554] [24] /usr/bin/python(_PyEval_EvalFrameDefault+0xa89)[0x5d74d9]
[m-0:02554] [25] /usr/bin/python[0x54ca1d]
[m-0:02554] [26] /usr/bin/python(_PyEval_EvalFrameDefault+0xa89)[0x5d74d9]
[m-0:02555] [25] /usr/bin/python[0x54ca1d]
[m-0:02555] [26] /usr/bin/python(_PyEval_EvalFrameDefault+0xa89)[0x5d74d9]
[m-0:02553] [25] /usr/bin/python[0x54ca1d]
[m-0:02553] [26] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02554] [27] /usr/bin/python[0x54ca1d]
[m-0:02554] [28] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
/usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02555] [27] /usr/bin/python[0x54ca1d]
[m-0:02555] [28] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02555] [29] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02553] [27] /usr/bin/python[0x54ca1d]
[m-0:02553] [28] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02553] [29] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02554] [29] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02554] *** End of error message ***
[m-0:02553] *** End of error message ***
/usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02555] *** End of error message ***
/usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x55a6277)[0x7d483a27a277]
[m-0:02552] [14] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit24invokeOperatorFromPythonERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EERKN8pybind114argsERKNS9_6kwargsESt8optionalIN3c1011DispatchKeyEE+0xef)[0x7d484254741f]
[m-0:02552] [15] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(_ZN5torch3jit37_get_operation_for_overload_or_packetERKSt6vectorISt10shared_ptrINS0_8OperatorEESaIS4_EEN3c106SymbolERKN8pybind114argsERKNSB_6kwargsEbSt8optionalINS9_11DispatchKeyEE+0x229)[0x7d48425476c9]
[m-0:02552] [16] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x8b5fd1)[0x7d4842457fd1]
[m-0:02552] [17] /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so(+0x42238d)[0x7d4841fc438d]
[m-0:02552] [18] /usr/bin/python[0x5820ff]
[m-0:02552] [19] /usr/bin/python(PyObject_Call+0x9c)[0x54b07c]
[m-0:02552] [20] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02552] [21] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02552] [22] /usr/bin/python[0x5a3698]
[m-0:02552] [23] /usr/bin/python(_PyObject_MakeTpCall+0x75)[0x548ec5]
[m-0:02552] [24] /usr/bin/python(_PyEval_EvalFrameDefault+0xa89)[0x5d74d9]
[m-0:02552] [25] /usr/bin/python[0x54ca1d]
[m-0:02552] [26] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02552] [27] /usr/bin/python[0x54ca1d]
[m-0:02552] [28] /usr/bin/python(_PyEval_EvalFrameDefault+0x4c3a)[0x5db68a]
[m-0:02552] [29] /usr/bin/python(_PyObject_Call_Prepend+0xc2)[0x54a712]
[m-0:02552] *** End of error message ***
--------------------------------------------------------------------------
Child job 2 terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

additional notes

A colleague pointed out that this happens with Llama4 on H100 even directly with trtllm-serve, I haven't tried it. Seems like a recent break. This is on the current code on main branch, sha fffb403125.

Start by running the examples/tensorrt_llm example with meta-llama/Llama-4-Scout-17B-16E-Instruct and tp 4 on the reported H100 setup. Inspect the executor startup path around the NCCL pncclCommInitRank failure and compare behavior with other supported models; done means all four executor processes start without the double-free crash.