nvshmem [GPU 0] Peer GPU 8 is not accessible

Hi team,
I'm using import torch.distributed._symmetric_memory to implement EP with PyTorch 2.9.0. My EP implementation passed 8-GPU unit test.
```
class ExpertParallel(ParallelStyle):
def __init__(self, device_mesh):
super().__init__()
self.input_splits = None
self.output_splits = None

# Set NVSHMEM as SymmMem backend
symm_mem.set_backend("NVSHMEM")
# Enable symm mem for groups
symm_mem.enable_symm_mem_for_group(torch.distributed.group.WORLD.group_name)
self.group_name = device_mesh.get_group().group_name
symm_mem.enable_symm_mem_for_group(self.group_name)
```
But when I scale the training to 2 nodes. I got error below.
```
2025-12-15 14:39:19.047571115 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 3] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.047577196 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.047956103 [s6]:
2025-12-15 14:39:19.057911428 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 4] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.057915939 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.057916639 [s6]:
2025-12-15 14:39:19.065598520 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 5] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.065604440 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.065605300 [s6]:
2025-12-15 14:39:19.084143291 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 6] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.084149981 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.084151391 [s6]:
2025-12-15 14:39:19.088895622 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 0] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.088900643 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.088901403 [s6]:
2025-12-15 14:39:19.093209112 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 7] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.093211972 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.093212722 [s6]:
2025-12-15 14:39:19.097185897 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.097188057 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.097336277 [s6]:
2025-12-15 14:39:19.100848469 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 2] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.101046654 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.101047624 [s6]:
2025-12-15 14:39:19.157089306 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 2] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.157091516 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.157091976 [s6]:
2025-12-15 14:39:19.157264289 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.157265079 [s6]:
2025-12-15 14:39:19.159627848 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.159628728 [s6]:
2025-12-15 14:39:19.167340521 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 4] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.167342152 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.167342621 [s6]:
2025-12-15 14:39:19.167343592 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.167344202 [s6]:
2025-12-15 14:39:19.170030725 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.170031585 [s6]:
2025-12-15 14:39:19.180239928 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 5] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.180244488 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.180245148 [s6]:
2025-12-15 14:39:19.180245959 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.180246499 [s6]:
2025-12-15 14:39:19.183328170 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.183330290 [s6]:
2025-12-15 14:39:19.185915715 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 6] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.185919375 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.185920226 [s6]:
2025-12-15 14:39:19.185921206 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.185921656 [s6]:
2025-12-15 14:39:19.188326728 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.188328028 [s6]:
2025-12-15 14:39:19.195200631 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 7] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.195206881 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.195208082 [s6]:
2025-12-15 14:39:19.195209132 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.195209792 [s6]:
2025-12-15 14:39:19.197721142 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.197724062 [s6]:
2025-12-15 14:39:19.199558884 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 3] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.199564404 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.199565174 [s6]:
2025-12-15 14:39:19.199912699 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.199914650 [s6]:
2025-12-15 14:39:19.203090478 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.203092518 [s6]:
2025-12-15 14:39:19.220204426 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 0] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.220210367 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.220211107 [s6]:
2025-12-15 14:39:19.220212147 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.220212856 [s6]:
2025-12-15 14:39:19.223185560 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
2025-12-15 14:39:19.223189450 [s6]:
2025-12-15 14:39:19.227672692 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 8 is not accessible, exiting ...
2025-12-15 14:39:19.227677612 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:1035: non-zero status: 3 building transport map failed
2025-12-15 14:39:19.227678392 [s6]:
2025-12-15 14:39:19.227679522 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1117: nvshmem initialization failed, exiting
2025-12-15 14:39:19.227680002 [s6]:
2025-12-15 14:39:19.230502525 [s6]: /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
```

I feel it's because of `symm_mem.enable_symm_mem_for_group(torch.distributed.group.WORLD.group_name)`.
But without that line, my unit test will fail at the first `symm_mem.empty(self.max_in_len, *self.dim, dtype=self.dtype, device=input.device)` in EP callback `def _token_dispatch(self, mod, inputs, device_mesh):` saying that
```
[rank0]: File "/mnt/localssd/.../expert_parallel.py", line 98, in _token_dispatch
[rank0]: self.symm_in = symm_mem.empty(self.max_in_len, *self.dim, dtype=self.dtype, device=self.device)
[rank0]: File "/mnt/localssd/.../lib/python3.10/site-packages/torch/distributed/_symmetric_memory/__init__.py", line 1706, in empty
[rank0]: return _SymmetricMemory.empty_strided_p2p(
[rank0]: RuntimeError: get_group_info: no group info associated with the group name 0
```

What's the expected way to `enable_symm_mem_for_group` for EP? I cannot find any reference or information anywhere including PyTorch docs and unit tests.

Thanks!

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @weifengpy @H-Huang