How to use all_to_all_kernel in Megatron-LM

I want to use all_to_all_kernel in Megatron-LM , but when I init nvshmem in Megatron-LM ,some error occur. I have already run this test success [04-deepseek-infer-all2all.py]。

==== backtrace (tid: 4505) ====
0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7fe86245d654]
1 /opt/hpcx/ucx/lib/libucs.so.0(+0x3684c) [0x7fe86245d84c]
2 /opt/hpcx/ucx/lib/libucs.so.0(+0x36a88) [0x7fe86245da88]
3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7fe8d2bfc330]
4 /lib64/ld-linux-x86-64.so.2(+0xa5ad) [0x7fe8d2f145ad]
5 /lib64/ld-linux-x86-64.so.2(+0xb1fc) [0x7fe8d2f151fc]
6 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x185124) [0x7fe8d2d3c124]
7 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x982b0) [0x7fe8d2c4f2b0]
8 /lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0x9c) [0x7fe8d2f0b51c]
9 /lib64/ld-linux-x86-64.so.2(+0x1669) [0x7fe8d2f0b669]
10 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x97c83) [0x7fe8d2c4ec83]
11 /usr/lib/x86_64-linux-gnu/libc.so.6(dlsym+0x84) [0x7fe8d2c4f344]
12 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3(+0x3c0e6) [0x7fe26143c0e6]
13 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3(+0x3b524) [0x7fe26143b524]
14 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3(+0x1126b8) [0x7fe2615126b8]
15 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3(nvshmemid_hostlib_init_attr+0x2b8) [0x7fe261517818]
16 /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3(nvshmemx_hostlib_init_attr+0x2c) [0x7fe261517dcc]
17 /usr/local/lib/python3.12/dist-packages/nvshmem/bindings/_internal/nvshmem.cpython-312-x86_64-linux-gnu.so(+0x1a066) [0x7fe4f5675066]
18 /usr/local/lib/python3.12/dist-packages/nvshmem/bindings/cynvshmem.cpython-312-x86_64-linux-gnu.so(+0x789e) [0x7fe4f56bd89e]
19 /usr/local/lib/python3.12/dist-packages/nvshmem/bindings/nvshmem.cpython-312-x86_64-linux-gnu.so(+0x2b996) [0x7fe4f56fc996]
20 /usr/local/lib/python3.12/dist-packages/nvshmem/bindings/nvshmem.cpython-312-x86_64-linux-gnu.so(+0x48426) [0x7fe4f5719426]
21 /usr/bin/python(PyObject_Vectorcall+0x35) [0x549b85]
22 /usr/bin/python(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
23 /usr/bin/python(_PyObject_Call_Prepend+0x18a) [0x54aa9a]
24 /usr/bin/python() [0x59e09f]
25 /usr/bin/python() [0x599b63]
26 /usr/bin/python(_PyObject_MakeTpCall+0x13e) [0x54924e]
27 /usr/bin/python(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
28 /usr/bin/python(_PyObject_Call_Prepend+0x18a) [0x54aa9a]
29 /usr/bin/python() [0x59e09f]
30 /usr/bin/python() [0x599b63]
31 /usr/bin/python(PyObject_Call+0x6c) [0x54b30c]
32 /usr/bin/python(_PyEval_EvalFrameDefault+0x4c1b) [0x5db55b]
33 /usr/bin/python(_PyObject_Call_Prepend+0x18a) [0x54aa9a]
34 /usr/bin/python() [0x59e09f]
35 /usr/bin/python() [0x599b63]
36 /usr/bin/python(PyObject_Call+0x6c) [0x54b30c]
37 /usr/bin/python(_PyEval_EvalFrameDefault+0x4c1b) [0x5db55b]
38 /usr/bin/python(_PyObject_Call_Prepend+0x18a) [0x54aa9a]
39 /usr/bin/python() [0x59e09f]
40 /usr/bin/python() [0x599b63]
41 /usr/bin/python(_PyObject_MakeTpCall+0x13e) [0x54924e]
42 /usr/bin/python(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
43 /usr/bin/python(_PyObject_Call_Prepend+0x18a) [0x54aa9a]
44 /usr/bin/python() [0x59e09f]
45 /usr/bin/python() [0x599b63]
46 /usr/bin/python(_PyObject_MakeTpCall+0x13e) [0x54924e]
47 /usr/bin/python(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
48 /usr/bin/python(PyEval_EvalCode+0x15b) [0x5d58eb]
49 /usr/bin/python() [0x608b42]
50 /usr/bin/python() [0x6b4e93]
51 /usr/bin/python(_PyRun_SimpleFileObject+0x1aa) [0x6b4bfa]
52 /usr/bin/python(_PyRun_AnyFileObject+0x4f) [0x6b4a2f]
53 /usr/bin/python(Py_RunMain+0x3b5) [0x6bca95]
54 /usr/bin/python(Py_BytesMain+0x2d) [0x6bc57d]
55 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7fe8d2be11ca]
56 /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7fe8d2be128b]
57 /usr/bin/python(_start+0x25) [0x657ce5]
=================================
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:ncclSocketInit:727: ncclSocketInit: connecting to address with family 0 is neither AF_INET(2) nor AF_INET6(10)
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:720: non-zero status: -3 /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/bootstrap/bootstrap_loader.cpp:103: non-zero status: -3 Bootstrap plugin init failed for 'nvshmem_bootstrap_uid.so.3'

[VM-17-82-centos:4503 :0:4503] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7feb69e0de60)

Start by comparing the successful 04-deepseek-infer-all2all.py test with the Megatron-LM path that initializes nvshmem. Examine the reported nvshmem bootstrap UID and ncclSocketInit errors and determine the missing setup or configuration. Done means the all_to_all_kernel path initializes without the reported failure, with the required steps documented.

Written by the indexing model from the issue text.