[disaggr + wide ep] GB200 R1, context node startup successfully, but generation node always reports NCCL errors

System Info

GB200

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

trtllm 1.0.0rc2, nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc2
reference to https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/disaggregated/slurm/submit.sh
sbatch --nodelist=xxx
--nodes=3
--ntasks=12
--ntasks-per-node=4
--gres=gpu:4
--segment=12
disaggr_torch.slurm 1 4 4 4480 true 1 8 1024 1024 true "0.7" 0 0 "$concurrency"

Expected behavior

Normal benchmark

actual behavior

Unable to startup service.
From work's log:
RuntimeError: Failed, NCCL error /src/tensorrt_llm/cpp/tensorrt_llm/common/opUtils.cpp:125 'unhandled system error (run with NCCL_DEBUG=INFO for details)'
8: (/src/tensorrt_llm/cpp/tensorrt_llm/common/opUtils.cpp:125)
8: 1 0x4003e1d3f628 getComm(std::set<int, std::less, std::allocator > const&) + 2888
8: 2 0x400452dc3bb0 torch_ext::allgather_list(c10::ArrayRefat::Tensor, std::optional<c10::List >, c10::List) + 316
8: 3 0x400452dc5e64 c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<std::vector<at::Tensor, std::allocatorat::Tensor > ()(c10::ArrayRefat::Tensor, std::optional<c10::List >, c10::List), std::vector<at::Tensor, std::allocatorat::Tensor >, c10::guts::typelist::typelist<c10::ArrayRefat::Tensor, std::optional<c10::List >, c10::List > >, false>::call(c10::OperatorKernel, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocatorc10::IValue >*) + 580
4: llm = PyTorchLLM(**llm_args)
8: 4 0x4000077154b8 /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x59254b8) [0x4000077154b8]
4: ^^^^^^^^^^^^^^^^^^^^^^
4: File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1017, in init
9: 73 0x567848 _PyEval_EvalFrameDefault + 16948
9: 74 0x4c6e98 /usr/bin/python() [0x4c6e98]
9: 75 0x4c5228 PyObject_Call + 280
9: 76 0x567308 _PyEval_EvalFrameDefault + 15604
9: 77 0x4c6e34 /usr/bin/python() [0x4c6e34]
9: 78 0x567308 _PyEval_EvalFrameDefault + 15604
9: 79 0x4c4774 _PyObject_Call_Prepend + 196
9: 80 0x528ba0 /usr/bin/python() [0x528ba0]
9: 81 0x4c2c98 _PyObject_MakeTpCall + 120
9: 82 0x563eb4 _PyEval_EvalFrameDefault + 2208
9: 83 0x562204 PyEval_EvalCode + 304
9: 84 0x59c3c4 /usr/bin/python() [0x59c3c4]
9: 85 0x680e94 /usr/bin/python() [0x680e94]
9: 86 0x680a68 _PyRun_SimpleFileObject + 404
9: 87 0x680834 _PyRun_AnyFileObject + 84
9: 88 0x68b83c Py_RunMain + 732
9: 89 0x68b3f8 Py_BytesMain + 40
9: 90 0x4000001f84c4 /usr/lib/aarch64-linux-gnu/libc.so.6(+0x284c4) [0x4000001f84c4]
9: 91 0x4000001f8598 __libc_start_main + 152
9: 92 0x5f6df0 _start + 48

additional notes

No

Start with examples/disaggregated/slurm/submit.sh and the reported disaggr_torch.slurm invocation, then compare the three-node setup with the failing work log. Run with NCCL_DEBUG=INFO as suggested by the error and inspect the NCCL initialization path at common/opUtils.cpp:125 and llmapi/llm.py:1017. Done means identifying the startup incompatibility and documenting or implementing a reproducible resolution for GB200 disaggregated generation nodes.