aliyun / aliyun/SimAI

Segmentation Error when running SimCCL

Open
#146 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

Steps

1.1 In the SimCCL folder, I first need to chmod +x src/device/generate.py otherwise there will be an error saying that permission denied.
1.2 I ran make -j src.build which succeeded.
1.3 cd test, make
1.4 bash run.sh

==> Error!

Log

NCCL version 2.20.5-MockNCCL+cuda12.6

inspur-gpu-server-15:2563676:2563678 [0] misc/cudawrap.cc:34 NCCL WARN Cuda failure 'no CUDA-capable device is detected'

inspur-gpu-server-15:2563676:2563678 [0] init.cc:270 NCCL WARN Cuda failure 3 'initialization error'
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Mocking start: mock_nNodes(8), mock_nRanks(64), mock_nRanksPerNode(8).
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO info_index(0): 0x7f9f08096a30, rank: 0, cudaDev: 0, nvmlDev: 0, gdrSupport: 1,hostHash: 2935421449793392623, pidHash: 12347059415262710392, busId: ffffffffffffffff:ff:ff.f, cudaCompCap: -1.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm_index(0): 0x55c7f6f30c00, node: 0, nNodes: 0, localRank: 0, localRanks: 0, maxLocalRanks: 0, intraRank: 0, intraRanks: 0, ncclCollNet: 0.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO nNodes: 64.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm_index(0): 0x55c7f6f30c00, node: 0, nNodes: 0, localRank: 0, localRanks: 0, maxLocalRanks: 0, intraRank: 0, intraRanks: 1, ncclCollNet: 0.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO After ncclTopoGetSystem, system_info{nodes[NET].count: 5, nodes[GPU].count: 8, maxBW: 0.000000, totalBW: 0.000000}.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[0].id: 0000:13:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[1].id: 0000:19:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[2].id: 0000:48:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[3].id: 0000:4d:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[4].id: 0000:89:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[5].id: 0000:8e:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[6].id: 0000:ad:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[7].id: 0000:b3:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO After ncclTopoGetSystem, system_info{nodes[NET].count: 5, nodes[GPU].count: 8, maxBW: 0.000000, totalBW: 0.000000}.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[0].id: 0000:13:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[1].id: 0000:19:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[2].id: 0000:48:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[3].id: 0000:4d:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[4].id: 0000:89:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[5].id: 0000:8e:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[6].id: 0000:ad:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO (*system)->nodes[GPU].nodes[7].id: 0000:b3:00.0, gdr = 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO After ncclTopoSearchInit, system_info{nodes[NET].count: 5, nodes[GPU].count: 8, maxBW: 24.000000, totalBW: 240.000000}.
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/13000 :GPU/13000 (0/5000.000000/LOC) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (4/24.000000/PXB) NET/2 (6/24.000000/PXN) NET/3 (6/24.000000/PXN) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/19000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (0/5000.000000/LOC) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (4/24.000000/PXB) NET/2 (6/24.000000/PXN) NET/3 (6/24.000000/PXN) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/48000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (0/5000.000000/LOC) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PXN) NET/2 (4/24.000000/PXB) NET/3 (6/24.000000/PXN) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/4D000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (0/5000.000000/LOC) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PXN) NET/2 (4/24.000000/PXB) NET/3 (6/24.000000/PXN) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/89000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (0/5000.000000/LOC) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (6/24.000000/PXN) NET/2 (6/24.000000/PXN) NET/3 (4/24.000000/PXB) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/8E000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (0/5000.000000/LOC) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (6/24.000000/PXN) NET/2 (6/24.000000/PXN) NET/3 (4/24.000000/PXB) NET/4 (6/24.000000/PXN)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/AD000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (0/5000.000000/LOC) GPU/B3000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (6/24.000000/PXN) NET/2 (6/24.000000/PXN) NET/3 (6/24.000000/PXN) NET/4 (4/24.000000/PXB)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO GPU/B3000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) GPU/89000 (2/240.000000/NVL) GPU/8E000 (2/240.000000/NVL) GPU/AD000 (2/240.000000/NVL) GPU/B3000 (0/5000.000000/LOC) NVS/0 (1/240.000000/NVL) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (6/24.000000/PXN) NET/2 (6/24.000000/PXN) NET/3 (6/24.000000/PXN) NET/4 (4/24.000000/PXB)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO NET/0 :GPU/13000 (5/24.000000/PHB) GPU/19000 (5/24.000000/PHB) GPU/48000 (5/24.000000/PHB) GPU/4D000 (5/24.000000/PHB) GPU/89000 (6/10.000000/SYS) GPU/8E000 (6/10.000000/SYS) GPU/AD000 (6/10.000000/SYS) GPU/B3000 (6/10.000000/SYS) CPU/0 (2/24.000000/PHB) CPU/1 (3/10.000000/SYS) NET/0 (0/5000.000000/LOC) NET/1 (5/24.000000/PHB) NET/2 (5/24.000000/PHB) NET/3 (6/10.000000/SYS) NET/4 (6/10.000000/SYS)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO NET/1 :GPU/13000 (4/24.000000/PXB) GPU/19000 (4/24.000000/PXB) GPU/48000 (6/24.000000/PHB) GPU/4D000 (6/24.000000/PHB) GPU/89000 (7/10.000000/SYS) GPU/8E000 (7/10.000000/SYS) GPU/AD000 (7/10.000000/SYS) GPU/B3000 (7/10.000000/SYS) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (0/5000.000000/LOC) NET/2 (6/24.000000/PHB) NET/3 (7/10.000000/SYS) NET/4 (7/10.000000/SYS)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO NET/2 :GPU/13000 (6/24.000000/PHB) GPU/19000 (6/24.000000/PHB) GPU/48000 (4/24.000000/PXB) GPU/4D000 (4/24.000000/PXB) GPU/89000 (7/10.000000/SYS) GPU/8E000 (7/10.000000/SYS) GPU/AD000 (7/10.000000/SYS) GPU/B3000 (7/10.000000/SYS) CPU/0 (3/24.000000/PHB) CPU/1 (4/10.000000/SYS) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PHB) NET/2 (0/5000.000000/LOC) NET/3 (7/10.000000/SYS) NET/4 (7/10.000000/SYS)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO NET/3 :GPU/13000 (7/10.000000/SYS) GPU/19000 (7/10.000000/SYS) GPU/48000 (7/10.000000/SYS) GPU/4D000 (7/10.000000/SYS) GPU/89000 (4/24.000000/PXB) GPU/8E000 (4/24.000000/PXB) GPU/AD000 (6/24.000000/PHB) GPU/B3000 (6/24.000000/PHB) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (7/10.000000/SYS) NET/2 (7/10.000000/SYS) NET/3 (0/5000.000000/LOC) NET/4 (6/24.000000/PHB)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO NET/4 :GPU/13000 (7/10.000000/SYS) GPU/19000 (7/10.000000/SYS) GPU/48000 (7/10.000000/SYS) GPU/4D000 (7/10.000000/SYS) GPU/89000 (6/24.000000/PHB) GPU/8E000 (6/24.000000/PHB) GPU/AD000 (4/24.000000/PXB) GPU/B3000 (4/24.000000/PXB) CPU/0 (4/10.000000/SYS) CPU/1 (3/24.000000/PHB) NET/0 (6/10.000000/SYS) NET/1 (7/10.000000/SYS) NET/2 (7/10.000000/SYS) NET/3 (6/24.000000/PHB) NET/4 (0/5000.000000/LOC)
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[0].gpu.rank: 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[1].gpu.rank: 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[2].gpu.rank: 2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[3].gpu.rank: 3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[4].gpu.rank: 4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[5].gpu.rank: 5
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[6].gpu.rank: 6
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO s->nodes[GPU].nodes[7].gpu.rank: 7
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->compCap: 80, comm->minCompCap: 80, comm->maxCompCap: 80
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO mock_comm[0].compCap: 80, mock_comm[0].minCompCap: 80, mock_comm[0].maxCompCap: 80
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/7 GPU/6 GPU/5 GPU/4 GPU/3 GPU/1 GPU/2 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/1 GPU/7 GPU/6 GPU/5 GPU/4 GPU/3 GPU/0 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/4 GPU/3 GPU/2 GPU/1 GPU/0 GPU/7 GPU/5 GPU/6 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/6 GPU/5 GPU/2 GPU/1 GPU/0 GPU/7 GPU/3 GPU/4 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7 GPU/0 GPU/1 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/4 GPU/5 GPU/6 GPU/7 GPU/0 GPU/1 GPU/2 GPU/3 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/6 GPU/7 GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 0]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/7 GPU/6 GPU/5 GPU/4 GPU/3 GPU/1 GPU/2 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/1 GPU/7 GPU/6 GPU/5 GPU/4 GPU/3 GPU/0 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/4 GPU/3 GPU/2 GPU/1 GPU/0 GPU/7 GPU/5 GPU/6 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/6 GPU/5 GPU/2 GPU/1 GPU/0 GPU/7 GPU/3 GPU/4 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 0]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/3 GPU/4 GPU/5 GPU/6 GPU/7 GPU/0 GPU/1 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/4 GPU/5 GPU/6 GPU/7 GPU/0 GPU/1 GPU/2 GPU/3 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/6 GPU/7 GPU/0 GPU/1 GPU/2 GPU/3 GPU/4 GPU/5 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 1]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/8 GPU/15 GPU/14 GPU/13 GPU/12 GPU/11 GPU/9 GPU/10 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/10 GPU/9 GPU/15 GPU/14 GPU/13 GPU/12 GPU/11 GPU/8 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/12 GPU/11 GPU/10 GPU/9 GPU/8 GPU/15 GPU/13 GPU/14 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/14 GPU/13 GPU/10 GPU/9 GPU/8 GPU/15 GPU/11 GPU/12 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 1]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/8 GPU/9 GPU/10 GPU/11 GPU/12 GPU/13 GPU/14 GPU/15 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/10 GPU/11 GPU/12 GPU/13 GPU/14 GPU/15 GPU/8 GPU/9 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/12 GPU/13 GPU/14 GPU/15 GPU/8 GPU/9 GPU/10 GPU/11 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/14 GPU/15 GPU/8 GPU/9 GPU/10 GPU/11 GPU/12 GPU/13 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 2]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/16 GPU/23 GPU/22 GPU/21 GPU/20 GPU/19 GPU/17 GPU/18 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/18 GPU/17 GPU/23 GPU/22 GPU/21 GPU/20 GPU/19 GPU/16 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/20 GPU/19 GPU/18 GPU/17 GPU/16 GPU/23 GPU/21 GPU/22 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/22 GPU/21 GPU/18 GPU/17 GPU/16 GPU/23 GPU/19 GPU/20 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 2]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/16 GPU/17 GPU/18 GPU/19 GPU/20 GPU/21 GPU/22 GPU/23 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/18 GPU/19 GPU/20 GPU/21 GPU/22 GPU/23 GPU/16 GPU/17 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/20 GPU/21 GPU/22 GPU/23 GPU/16 GPU/17 GPU/18 GPU/19 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/22 GPU/23 GPU/16 GPU/17 GPU/18 GPU/19 GPU/20 GPU/21 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 3]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/24 GPU/31 GPU/30 GPU/29 GPU/28 GPU/27 GPU/25 GPU/26 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/26 GPU/25 GPU/31 GPU/30 GPU/29 GPU/28 GPU/27 GPU/24 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/28 GPU/27 GPU/26 GPU/25 GPU/24 GPU/31 GPU/29 GPU/30 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/30 GPU/29 GPU/26 GPU/25 GPU/24 GPU/31 GPU/27 GPU/28 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 3]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/24 GPU/25 GPU/26 GPU/27 GPU/28 GPU/29 GPU/30 GPU/31 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/26 GPU/27 GPU/28 GPU/29 GPU/30 GPU/31 GPU/24 GPU/25 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/28 GPU/29 GPU/30 GPU/31 GPU/24 GPU/25 GPU/26 GPU/27 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/30 GPU/31 GPU/24 GPU/25 GPU/26 GPU/27 GPU/28 GPU/29 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 4]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/32 GPU/39 GPU/38 GPU/37 GPU/36 GPU/35 GPU/33 GPU/34 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/34 GPU/33 GPU/39 GPU/38 GPU/37 GPU/36 GPU/35 GPU/32 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/36 GPU/35 GPU/34 GPU/33 GPU/32 GPU/39 GPU/37 GPU/38 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/38 GPU/37 GPU/34 GPU/33 GPU/32 GPU/39 GPU/35 GPU/36 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 4]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/32 GPU/33 GPU/34 GPU/35 GPU/36 GPU/37 GPU/38 GPU/39 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/34 GPU/35 GPU/36 GPU/37 GPU/38 GPU/39 GPU/32 GPU/33 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/36 GPU/37 GPU/38 GPU/39 GPU/32 GPU/33 GPU/34 GPU/35 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/38 GPU/39 GPU/32 GPU/33 GPU/34 GPU/35 GPU/36 GPU/37 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 5]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/40 GPU/47 GPU/46 GPU/45 GPU/44 GPU/43 GPU/41 GPU/42 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/42 GPU/41 GPU/47 GPU/46 GPU/45 GPU/44 GPU/43 GPU/40 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/44 GPU/43 GPU/42 GPU/41 GPU/40 GPU/47 GPU/45 GPU/46 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/46 GPU/45 GPU/42 GPU/41 GPU/40 GPU/47 GPU/43 GPU/44 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 5]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/40 GPU/41 GPU/42 GPU/43 GPU/44 GPU/45 GPU/46 GPU/47 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/42 GPU/43 GPU/44 GPU/45 GPU/46 GPU/47 GPU/40 GPU/41 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/44 GPU/45 GPU/46 GPU/47 GPU/40 GPU/41 GPU/42 GPU/43 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/46 GPU/47 GPU/40 GPU/41 GPU/42 GPU/43 GPU/44 GPU/45 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 6]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/48 GPU/55 GPU/54 GPU/53 GPU/52 GPU/51 GPU/49 GPU/50 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/50 GPU/49 GPU/55 GPU/54 GPU/53 GPU/52 GPU/51 GPU/48 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/52 GPU/51 GPU/50 GPU/49 GPU/48 GPU/55 GPU/53 GPU/54 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/54 GPU/53 GPU/50 GPU/49 GPU/48 GPU/55 GPU/51 GPU/52 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 6]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/48 GPU/49 GPU/50 GPU/51 GPU/52 GPU/53 GPU/54 GPU/55 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/50 GPU/51 GPU/52 GPU/53 GPU/54 GPU/55 GPU/48 GPU/49 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/52 GPU/53 GPU/54 GPU/55 GPU/48 GPU/49 GPU/50 GPU/51 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/54 GPU/55 GPU/48 GPU/49 GPU/50 GPU/51 GPU/52 GPU/53 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock ring Graph[node: 7]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 4, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/56 GPU/63 GPU/62 GPU/61 GPU/60 GPU/59 GPU/57 GPU/58 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/58 GPU/57 GPU/63 GPU/62 GPU/61 GPU/60 GPU/59 GPU/56 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/60 GPU/59 GPU/58 GPU/57 GPU/56 GPU/63 GPU/61 GPU/62 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/62 GPU/61 GPU/58 GPU/57 GPU/56 GPU/63 GPU/59 GPU/60 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO ------ Mock tree Graph[node: 7]: ------
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 4, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  0 : NET/1 GPU/56 GPU/57 GPU/58 GPU/59 GPU/60 GPU/61 GPU/62 GPU/63 NET/2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  1 : NET/2 GPU/58 GPU/59 GPU/60 GPU/61 GPU/62 GPU/63 GPU/56 GPU/57 NET/1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  2 : NET/3 GPU/60 GPU/61 GPU/62 GPU/63 GPU/56 GPU/57 GPU/58 GPU/59 NET/4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO  3 : NET/4 GPU/62 GPU/63 GPU/56 GPU/57 GPU/58 GPU/59 GPU/60 GPU/61 NET/3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->nNodes: 8
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO treeGraph.nChannels: 4, ringGraph.nChannels: 4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 00/08 :     0    7    6    5    4    3    1    2   10    9   15   14   13   12   11    8   16   23   22   21   20   19   17   18   26   25   31   30   29   28   27   24   32   39   38   37   36   35   33   34   42   41   47   46   45   44   43   40   48   55   54   53   52   51   49   50   58   57   63   62   61   60   59   56
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 01/08 :     0    8   15   14   13   12   11    9   10   18   17   23   22   21   20   19   16   24   31   30   29   28   27   25   26   34   33   39   38   37   36   35   32   40   47   46   45   44   43   41   42   50   49   55   54   53   52   51   48   56   63   62   61   60   59   57   58    2    1    7    6    5    4    3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 02/08 :     0    7    5    6   14   13   10    9    8   15   11   12   20   19   18   17   16   23   21   22   30   29   26   25   24   31   27   28   36   35   34   33   32   39   37   38   46   45   42   41   40   47   43   44   52   51   50   49   48   55   53   54   62   61   58   57   56   63   59   60    4    3    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 03/08 :     0    7    3    4   12   11   10    9    8   15   13   14   22   21   18   17   16   23   19   20   28   27   26   25   24   31   29   30   38   37   34   33   32   39   35   36   44   43   42   41   40   47   45   46   54   53   50   49   48   55   51   52   60   59   58   57   56   63   61   62    6    5    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 04/08 :     0    7    6    5    4    3    1    2   10    9   15   14   13   12   11    8   16   23   22   21   20   19   17   18   26   25   31   30   29   28   27   24   32   39   38   37   36   35   33   34   42   41   47   46   45   44   43   40   48   55   54   53   52   51   49   50   58   57   63   62   61   60   59   56
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 05/08 :     0    8   15   14   13   12   11    9   10   18   17   23   22   21   20   19   16   24   31   30   29   28   27   25   26   34   33   39   38   37   36   35   32   40   47   46   45   44   43   41   42   50   49   55   54   53   52   51   48   56   63   62   61   60   59   57   58    2    1    7    6    5    4    3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 06/08 :     0    7    5    6   14   13   10    9    8   15   11   12   20   19   18   17   16   23   21   22   30   29   26   25   24   31   27   28   36   35   34   33   32   39   37   38   46   45   42   41   40   47   43   44   52   51   50   49   48   55   53   54   62   61   58   57   56   63   59   60    4    3    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 07/08 :     0    7    3    4   12   11   10    9    8   15   13   14   22   21   18   17   16   23   19   20   28   27   26   25   24   31   29   30   38   37   34   33   32   39   35   36   44   43   42   41   40   47   45   46   54   53   50   49   48   55   51   52   60   59   58   57   56   63   61   62    6    5    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO mock_comm->config.maxCTAs: 32, mock_comm->config.minCTAs: 1, mock_comm->sharedRes->tpNChannels: 0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 00/08 :     0    7    6    5    4    3    1    2   10    9   15   14   13   12   11    8   16   23   22   21   20   19   17   18   26   25   31   30   29   28   27   24   32   39   38   37   36   35   33   34   42   41   47   46   45   44   43   40   48   55   54   53   52   51   49   50   58   57   63   62   61   60   59   56
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 01/08 :     0    8   15   14   13   12   11    9   10   18   17   23   22   21   20   19   16   24   31   30   29   28   27   25   26   34   33   39   38   37   36   35   32   40   47   46   45   44   43   41   42   50   49   55   54   53   52   51   48   56   63   62   61   60   59   57   58    2    1    7    6    5    4    3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 02/08 :     0    7    5    6   14   13   10    9    8   15   11   12   20   19   18   17   16   23   21   22   30   29   26   25   24   31   27   28   36   35   34   33   32   39   37   38   46   45   42   41   40   47   43   44   52   51   50   49   48   55   53   54   62   61   58   57   56   63   59   60    4    3    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 03/08 :     0    7    3    4   12   11   10    9    8   15   13   14   22   21   18   17   16   23   19   20   28   27   26   25   24   31   29   30   38   37   34   33   32   39   35   36   44   43   42   41   40   47   45   46   54   53   50   49   48   55   51   52   60   59   58   57   56   63   61   62    6    5    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 04/08 :     0    7    6    5    4    3    1    2   10    9   15   14   13   12   11    8   16   23   22   21   20   19   17   18   26   25   31   30   29   28   27   24   32   39   38   37   36   35   33   34   42   41   47   46   45   44   43   40   48   55   54   53   52   51   49   50   58   57   63   62   61   60   59   56
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 05/08 :     0    8   15   14   13   12   11    9   10   18   17   23   22   21   20   19   16   24   31   30   29   28   27   25   26   34   33   39   38   37   36   35   32   40   47   46   45   44   43   41   42   50   49   55   54   53   52   51   48   56   63   62   61   60   59   57   58    2    1    7    6    5    4    3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 06/08 :     0    7    5    6   14   13   10    9    8   15   11   12   20   19   18   17   16   23   21   22   30   29   26   25   24   31   27   28   36   35   34   33   32   39   37   38   46   45   42   41   40   47   43   44   52   51   50   49   48   55   53   54   62   61   58   57   56   63   59   60    4    3    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Channel 07/08 :     0    7    3    4   12   11   10    9    8   15   13   14   22   21   18   17   16   23   19   20   28   27   26   25   24   31   29   30   38   37   34   33   32   39   35   36   44   43   42   41   40   47   45   46   54   53   50   49   48   55   51   52   60   59   58   57   56   63   61   62    6    5    2    1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_0 Trees [0] 1/32/-1->0->-1 [1] 1/-1/-1->0->7 [2] 1/-1/-1->0->7 [3] 1/-1/-1->0->7 [4] 1/-1/-1->0->8 [5] 1/-1/-1->0->7 [6] 1/-1/-1->0->7 [7] 1/-1/-1->0->7
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_1 Trees [0] 2/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_2 Trees [0] 3/-1/-1->2->1 [1] 3/34/-1->2->-1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->10 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_3 Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] -1/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] -1/-1/-1->3->2 [7] 4/-1/-1->3->2
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_4 Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/36/-1->4->-1 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->12 [7] 5/-1/-1->4->3
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_5 Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] -1/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] -1/-1/-1->5->4
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_6 Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/38/-1->6->-1 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->14
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_7 Trees [0] -1/-1/-1->7->6 [1] 0/-1/-1->7->6 [2] 0/-1/-1->7->6 [3] 0/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] 0/-1/-1->7->6 [6] 0/-1/-1->7->6 [7] 0/-1/-1->7->6
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_8 Trees [0] 9/-1/-1->8->17 [1] 9/-1/-1->8->15 [2] 9/-1/-1->8->15 [3] 9/-1/-1->8->15 [4] 9/0/-1->8->24 [5] 9/-1/-1->8->15 [6] 9/-1/-1->8->15 [7] 9/-1/-1->8->15
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_9 Trees [0] 10/-1/-1->9->8 [1] -1/-1/-1->9->8 [2] 10/-1/-1->9->8 [3] 10/-1/-1->9->8 [4] 10/16/-1->9->8 [5] -1/-1/-1->9->8 [6] 10/-1/-1->9->8 [7] 10/-1/-1->9->8
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_10 Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->19 [2] 11/-1/-1->10->9 [3] 11/-1/-1->10->9 [4] 11/-1/-1->10->9 [5] 11/2/-1->10->26 [6] 11/-1/-1->10->9 [7] 11/-1/-1->10->9
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_11 Trees [0] 12/-1/-1->11->10 [1] 12/-1/-1->11->10 [2] -1/-1/-1->11->10 [3] 12/-1/-1->11->10 [4] 12/-1/-1->11->10 [5] 12/18/-1->11->10 [6] -1/-1/-1->11->10 [7] 12/-1/-1->11->10
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_12 Trees [0] 13/-1/-1->12->11 [1] 13/-1/-1->12->11 [2] 13/-1/-1->12->21 [3] 13/-1/-1->12->11 [4] 13/-1/-1->12->11 [5] 13/-1/-1->12->11 [6] 13/4/-1->12->28 [7] 13/-1/-1->12->11
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_13 Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->12 [2] 14/-1/-1->13->12 [3] -1/-1/-1->13->12 [4] 14/-1/-1->13->12 [5] 14/-1/-1->13->12 [6] 14/20/-1->13->12 [7] -1/-1/-1->13->12
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_14 Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 15/-1/-1->14->13 [3] 15/-1/-1->14->23 [4] 15/-1/-1->14->13 [5] 15/-1/-1->14->13 [6] 15/-1/-1->14->13 [7] 15/6/-1->14->30
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_15 Trees [0] -1/-1/-1->15->14 [1] 8/-1/-1->15->14 [2] 8/-1/-1->15->14 [3] 8/-1/-1->15->14 [4] -1/-1/-1->15->14 [5] 8/-1/-1->15->14 [6] 8/-1/-1->15->14 [7] 8/22/-1->15->14
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_16 Trees [0] 17/24/-1->16->33 [1] 17/-1/-1->16->23 [2] 17/-1/-1->16->23 [3] 17/-1/-1->16->23 [4] 17/-1/-1->16->9 [5] 17/-1/-1->16->23 [6] 17/-1/-1->16->23 [7] 17/-1/-1->16->23
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_17 Trees [0] 18/8/-1->17->16 [1] -1/-1/-1->17->16 [2] 18/-1/-1->17->16 [3] 18/-1/-1->17->16 [4] 18/-1/-1->17->16 [5] -1/-1/-1->17->16 [6] 18/-1/-1->17->16 [7] 18/-1/-1->17->16
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_18 Trees [0] 19/-1/-1->18->17 [1] 19/26/-1->18->35 [2] 19/-1/-1->18->17 [3] 19/-1/-1->18->17 [4] 19/-1/-1->18->17 [5] 19/-1/-1->18->11 [6] 19/-1/-1->18->17 [7] 19/-1/-1->18->17
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_19 Trees [0] 20/-1/-1->19->18 [1] 20/10/-1->19->18 [2] -1/-1/-1->19->18 [3] 20/-1/-1->19->18 [4] 20/-1/-1->19->18 [5] 20/-1/-1->19->18 [6] -1/-1/-1->19->18 [7] 20/-1/-1->19->18
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_20 Trees [0] 21/-1/-1->20->19 [1] 21/-1/-1->20->19 [2] 21/28/-1->20->37 [3] 21/-1/-1->20->19 [4] 21/-1/-1->20->19 [5] 21/-1/-1->20->19 [6] 21/-1/-1->20->13 [7] 21/-1/-1->20->19
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_21 Trees [0] 22/-1/-1->21->20 [1] 22/-1/-1->21->20 [2] 22/12/-1->21->20 [3] -1/-1/-1->21->20 [4] 22/-1/-1->21->20 [5] 22/-1/-1->21->20 [6] 22/-1/-1->21->20 [7] -1/-1/-1->21->20
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_22 Trees [0] 23/-1/-1->22->21 [1] 23/-1/-1->22->21 [2] 23/-1/-1->22->21 [3] 23/30/-1->22->39 [4] 23/-1/-1->22->21 [5] 23/-1/-1->22->21 [6] 23/-1/-1->22->21 [7] 23/-1/-1->22->15
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_23 Trees [0] -1/-1/-1->23->22 [1] 16/-1/-1->23->22 [2] 16/-1/-1->23->22 [3] 16/14/-1->23->22 [4] -1/-1/-1->23->22 [5] 16/-1/-1->23->22 [6] 16/-1/-1->23->22 [7] 16/-1/-1->23->22
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_24 Trees [0] 25/-1/-1->24->16 [1] 25/-1/-1->24->31 [2] 25/-1/-1->24->31 [3] 25/-1/-1->24->31 [4] 25/8/-1->24->56 [5] 25/-1/-1->24->31 [6] 25/-1/-1->24->31 [7] 25/-1/-1->24->31
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_25 Trees [0] 26/-1/-1->25->24 [1] -1/-1/-1->25->24 [2] 26/-1/-1->25->24 [3] 26/-1/-1->25->24 [4] 26/40/-1->25->24 [5] -1/-1/-1->25->24 [6] 26/-1/-1->25->24 [7] 26/-1/-1->25->24
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_26 Trees [0] 27/-1/-1->26->25 [1] 27/-1/-1->26->18 [2] 27/-1/-1->26->25 [3] 27/-1/-1->26->25 [4] 27/-1/-1->26->25 [5] 27/10/-1->26->58 [6] 27/-1/-1->26->25 [7] 27/-1/-1->26->25
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_27 Trees [0] 28/-1/-1->27->26 [1] 28/-1/-1->27->26 [2] -1/-1/-1->27->26 [3] 28/-1/-1->27->26 [4] 28/-1/-1->27->26 [5] 28/42/-1->27->26 [6] -1/-1/-1->27->26 [7] 28/-1/-1->27->26
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_28 Trees [0] 29/-1/-1->28->27 [1] 29/-1/-1->28->27 [2] 29/-1/-1->28->20 [3] 29/-1/-1->28->27 [4] 29/-1/-1->28->27 [5] 29/-1/-1->28->27 [6] 29/12/-1->28->60 [7] 29/-1/-1->28->27
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_29 Trees [0] 30/-1/-1->29->28 [1] 30/-1/-1->29->28 [2] 30/-1/-1->29->28 [3] -1/-1/-1->29->28 [4] 30/-1/-1->29->28 [5] 30/-1/-1->29->28 [6] 30/44/-1->29->28 [7] -1/-1/-1->29->28
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_30 Trees [0] 31/-1/-1->30->29 [1] 31/-1/-1->30->29 [2] 31/-1/-1->30->29 [3] 31/-1/-1->30->22 [4] 31/-1/-1->30->29 [5] 31/-1/-1->30->29 [6] 31/-1/-1->30->29 [7] 31/14/-1->30->62
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_31 Trees [0] -1/-1/-1->31->30 [1] 24/-1/-1->31->30 [2] 24/-1/-1->31->30 [3] 24/-1/-1->31->30 [4] -1/-1/-1->31->30 [5] 24/-1/-1->31->30 [6] 24/-1/-1->31->30 [7] 24/46/-1->31->30
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_32 Trees [0] 33/48/-1->32->0 [1] 33/-1/-1->32->39 [2] 33/-1/-1->32->39 [3] 33/-1/-1->32->39 [4] 33/-1/-1->32->40 [5] 33/-1/-1->32->39 [6] 33/-1/-1->32->39 [7] 33/-1/-1->32->39
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_33 Trees [0] 34/16/-1->33->32 [1] -1/-1/-1->33->32 [2] 34/-1/-1->33->32 [3] 34/-1/-1->33->32 [4] 34/-1/-1->33->32 [5] -1/-1/-1->33->32 [6] 34/-1/-1->33->32 [7] 34/-1/-1->33->32
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_34 Trees [0] 35/-1/-1->34->33 [1] 35/50/-1->34->2 [2] 35/-1/-1->34->33 [3] 35/-1/-1->34->33 [4] 35/-1/-1->34->33 [5] 35/-1/-1->34->42 [6] 35/-1/-1->34->33 [7] 35/-1/-1->34->33
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_35 Trees [0] 36/-1/-1->35->34 [1] 36/18/-1->35->34 [2] -1/-1/-1->35->34 [3] 36/-1/-1->35->34 [4] 36/-1/-1->35->34 [5] 36/-1/-1->35->34 [6] -1/-1/-1->35->34 [7] 36/-1/-1->35->34
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_36 Trees [0] 37/-1/-1->36->35 [1] 37/-1/-1->36->35 [2] 37/52/-1->36->4 [3] 37/-1/-1->36->35 [4] 37/-1/-1->36->35 [5] 37/-1/-1->36->35 [6] 37/-1/-1->36->44 [7] 37/-1/-1->36->35
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_37 Trees [0] 38/-1/-1->37->36 [1] 38/-1/-1->37->36 [2] 38/20/-1->37->36 [3] -1/-1/-1->37->36 [4] 38/-1/-1->37->36 [5] 38/-1/-1->37->36 [6] 38/-1/-1->37->36 [7] -1/-1/-1->37->36
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_38 Trees [0] 39/-1/-1->38->37 [1] 39/-1/-1->38->37 [2] 39/-1/-1->38->37 [3] 39/54/-1->38->6 [4] 39/-1/-1->38->37 [5] 39/-1/-1->38->37 [6] 39/-1/-1->38->37 [7] 39/-1/-1->38->46
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_39 Trees [0] -1/-1/-1->39->38 [1] 32/-1/-1->39->38 [2] 32/-1/-1->39->38 [3] 32/22/-1->39->38 [4] -1/-1/-1->39->38 [5] 32/-1/-1->39->38 [6] 32/-1/-1->39->38 [7] 32/-1/-1->39->38
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_40 Trees [0] 41/-1/-1->40->49 [1] 41/-1/-1->40->47 [2] 41/-1/-1->40->47 [3] 41/-1/-1->40->47 [4] 41/32/-1->40->25 [5] 41/-1/-1->40->47 [6] 41/-1/-1->40->47 [7] 41/-1/-1->40->47
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_41 Trees [0] 42/-1/-1->41->40 [1] -1/-1/-1->41->40 [2] 42/-1/-1->41->40 [3] 42/-1/-1->41->40 [4] 42/48/-1->41->40 [5] -1/-1/-1->41->40 [6] 42/-1/-1->41->40 [7] 42/-1/-1->41->40
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_42 Trees [0] 43/-1/-1->42->41 [1] 43/-1/-1->42->51 [2] 43/-1/-1->42->41 [3] 43/-1/-1->42->41 [4] 43/-1/-1->42->41 [5] 43/34/-1->42->27 [6] 43/-1/-1->42->41 [7] 43/-1/-1->42->41
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_43 Trees [0] 44/-1/-1->43->42 [1] 44/-1/-1->43->42 [2] -1/-1/-1->43->42 [3] 44/-1/-1->43->42 [4] 44/-1/-1->43->42 [5] 44/50/-1->43->42 [6] -1/-1/-1->43->42 [7] 44/-1/-1->43->42
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_44 Trees [0] 45/-1/-1->44->43 [1] 45/-1/-1->44->43 [2] 45/-1/-1->44->53 [3] 45/-1/-1->44->43 [4] 45/-1/-1->44->43 [5] 45/-1/-1->44->43 [6] 45/36/-1->44->29 [7] 45/-1/-1->44->43
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_45 Trees [0] 46/-1/-1->45->44 [1] 46/-1/-1->45->44 [2] 46/-1/-1->45->44 [3] -1/-1/-1->45->44 [4] 46/-1/-1->45->44 [5] 46/-1/-1->45->44 [6] 46/52/-1->45->44 [7] -1/-1/-1->45->44
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_46 Trees [0] 47/-1/-1->46->45 [1] 47/-1/-1->46->45 [2] 47/-1/-1->46->45 [3] 47/-1/-1->46->55 [4] 47/-1/-1->46->45 [5] 47/-1/-1->46->45 [6] 47/-1/-1->46->45 [7] 47/38/-1->46->31
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_47 Trees [0] -1/-1/-1->47->46 [1] 40/-1/-1->47->46 [2] 40/-1/-1->47->46 [3] 40/-1/-1->47->46 [4] -1/-1/-1->47->46 [5] 40/-1/-1->47->46 [6] 40/-1/-1->47->46 [7] 40/54/-1->47->46
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_48 Trees [0] 49/56/-1->48->32 [1] 49/-1/-1->48->55 [2] 49/-1/-1->48->55 [3] 49/-1/-1->48->55 [4] 49/-1/-1->48->41 [5] 49/-1/-1->48->55 [6] 49/-1/-1->48->55 [7] 49/-1/-1->48->55
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_49 Trees [0] 50/40/-1->49->48 [1] -1/-1/-1->49->48 [2] 50/-1/-1->49->48 [3] 50/-1/-1->49->48 [4] 50/-1/-1->49->48 [5] -1/-1/-1->49->48 [6] 50/-1/-1->49->48 [7] 50/-1/-1->49->48
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_50 Trees [0] 51/-1/-1->50->49 [1] 51/58/-1->50->34 [2] 51/-1/-1->50->49 [3] 51/-1/-1->50->49 [4] 51/-1/-1->50->49 [5] 51/-1/-1->50->43 [6] 51/-1/-1->50->49 [7] 51/-1/-1->50->49
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_51 Trees [0] 52/-1/-1->51->50 [1] 52/42/-1->51->50 [2] -1/-1/-1->51->50 [3] 52/-1/-1->51->50 [4] 52/-1/-1->51->50 [5] 52/-1/-1->51->50 [6] -1/-1/-1->51->50 [7] 52/-1/-1->51->50
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_52 Trees [0] 53/-1/-1->52->51 [1] 53/-1/-1->52->51 [2] 53/60/-1->52->36 [3] 53/-1/-1->52->51 [4] 53/-1/-1->52->51 [5] 53/-1/-1->52->51 [6] 53/-1/-1->52->45 [7] 53/-1/-1->52->51
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_53 Trees [0] 54/-1/-1->53->52 [1] 54/-1/-1->53->52 [2] 54/44/-1->53->52 [3] -1/-1/-1->53->52 [4] 54/-1/-1->53->52 [5] 54/-1/-1->53->52 [6] 54/-1/-1->53->52 [7] -1/-1/-1->53->52
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_54 Trees [0] 55/-1/-1->54->53 [1] 55/-1/-1->54->53 [2] 55/-1/-1->54->53 [3] 55/62/-1->54->38 [4] 55/-1/-1->54->53 [5] 55/-1/-1->54->53 [6] 55/-1/-1->54->53 [7] 55/-1/-1->54->47
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_55 Trees [0] -1/-1/-1->55->54 [1] 48/-1/-1->55->54 [2] 48/-1/-1->55->54 [3] 48/46/-1->55->54 [4] -1/-1/-1->55->54 [5] 48/-1/-1->55->54 [6] 48/-1/-1->55->54 [7] 48/-1/-1->55->54
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_56 Trees [0] 57/-1/-1->56->48 [1] 57/-1/-1->56->63 [2] 57/-1/-1->56->63 [3] 57/-1/-1->56->63 [4] 57/24/-1->56->-1 [5] 57/-1/-1->56->63 [6] 57/-1/-1->56->63 [7] 57/-1/-1->56->63
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_57 Trees [0] 58/-1/-1->57->56 [1] -1/-1/-1->57->56 [2] 58/-1/-1->57->56 [3] 58/-1/-1->57->56 [4] 58/-1/-1->57->56 [5] -1/-1/-1->57->56 [6] 58/-1/-1->57->56 [7] 58/-1/-1->57->56
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_58 Trees [0] 59/-1/-1->58->57 [1] 59/-1/-1->58->50 [2] 59/-1/-1->58->57 [3] 59/-1/-1->58->57 [4] 59/-1/-1->58->57 [5] 59/26/-1->58->-1 [6] 59/-1/-1->58->57 [7] 59/-1/-1->58->57
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_59 Trees [0] 60/-1/-1->59->58 [1] 60/-1/-1->59->58 [2] -1/-1/-1->59->58 [3] 60/-1/-1->59->58 [4] 60/-1/-1->59->58 [5] 60/-1/-1->59->58 [6] -1/-1/-1->59->58 [7] 60/-1/-1->59->58
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_60 Trees [0] 61/-1/-1->60->59 [1] 61/-1/-1->60->59 [2] 61/-1/-1->60->52 [3] 61/-1/-1->60->59 [4] 61/-1/-1->60->59 [5] 61/-1/-1->60->59 [6] 61/28/-1->60->-1 [7] 61/-1/-1->60->59
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_61 Trees [0] 62/-1/-1->61->60 [1] 62/-1/-1->61->60 [2] 62/-1/-1->61->60 [3] -1/-1/-1->61->60 [4] 62/-1/-1->61->60 [5] 62/-1/-1->61->60 [6] 62/-1/-1->61->60 [7] -1/-1/-1->61->60
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_62 Trees [0] 63/-1/-1->62->61 [1] 63/-1/-1->62->61 [2] 63/-1/-1->62->61 [3] 63/-1/-1->62->54 [4] 63/-1/-1->62->61 [5] 63/-1/-1->62->61 [6] 63/-1/-1->62->61 [7] 63/30/-1->62->-1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO Grank_63 Trees [0] -1/-1/-1->63->62 [1] 56/-1/-1->63->62 [2] 56/-1/-1->63->62 [3] 56/-1/-1->63->62 [4] -1/-1/-1->63->62 [5] 56/-1/-1->63->62 [6] 56/-1/-1->63->62 [7] 56/-1/-1->63->62
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->buffSizes[NCCL_PROTO_LL]: 524288
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->buffSizes[NCCL_PROTO_LL128]: 4915200
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->buffSizes[NCCL_PROTO_SIMPLE]: 4194304
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->p2pChunkSize: 131072
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO comm->p2pnChannels: 8, comm->p2pnChannelsPerPeer: 1
inspur-gpu-server-15:2563676:2563678 [0] NCCL INFO 8 coll channels, 0 collnet channels, 0 nvls channels, 8 p2p channels, 1 p2p channels per peer
run.sh: 第 11 行： 2563676 段错误               （核心已转储） LD_LIBRARY_PATH="$SCRIPT_DIR/../build/lib/" NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=MOCK_ROOT,MOCK NCCL_CROSS_NIC=1 NCCL_TOPO_FILE=$SCRIPT_DIR/topo_8A100_4CX6.xml NCCL_NVLS_ENABLE=0 NCCL_NUM_MOCK_GPU=64 NCCL_NUM_MOCK_NODE=8 $1 ./MultiDevOneP $CORE_FILE

If I run the script with another xml file (e.g. topo_4A100_2CX6.xml), there is a different bug:

NCCL version 2.20.5-MockNCCL+cuda12.6

inspur-gpu-server-15:2570998:2571000 [0] misc/cudawrap.cc:34 NCCL WARN Cuda failure 'no CUDA-capable device is detected'

inspur-gpu-server-15:2570998:2571000 [0] init.cc:270 NCCL WARN Cuda failure 3 'initialization error'
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Mocking start: mock_nNodes(8), mock_nRanks(64), mock_nRanksPerNode(8).
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO info_index(0): 0x7f08f0099440, rank: 0, cudaDev: 0, nvmlDev: 0, gdrSupport: 1,hostHash: 10710501475456838185, pidHash: 13668843625570461005, busId: ffffffffffffffff:ff:ff.f, cudaCompCap: -1.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO comm_index(0): 0x55eaf7ae4c00, node: 0, nNodes: 0, localRank: 0, localRanks: 0, maxLocalRanks: 0, intraRank: 0, intraRanks: 0, ncclCollNet: 0.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO nNodes: 64.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO comm_index(0): 0x55eaf7ae4c00, node: 0, nNodes: 0, localRank: 0, localRanks: 0, maxLocalRanks: 0, intraRank: 0, intraRanks: 1, ncclCollNet: 0.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO After ncclTopoGetSystem, system_info{nodes[NET].count: 3, nodes[GPU].count: 4, maxBW: 0.000000, totalBW: 0.000000}.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[0].id: 0000:13:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[1].id: 0000:19:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[2].id: 0000:48:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[3].id: 0000:4d:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO After ncclTopoGetSystem, system_info{nodes[NET].count: 3, nodes[GPU].count: 4, maxBW: 0.000000, totalBW: 0.000000}.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[0].id: 0000:13:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[1].id: 0000:19:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[2].id: 0000:48:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO (*system)->nodes[GPU].nodes[3].id: 0000:4d:00.0, gdr = 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO After ncclTopoSearchInit, system_info{nodes[NET].count: 3, nodes[GPU].count: 4, maxBW: 24.000000, totalBW: 240.000000}.
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO GPU/13000 :GPU/13000 (0/5000.000000/LOC) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (4/24.000000/PXB) NET/2 (6/24.000000/PXN)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO GPU/19000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (0/5000.000000/LOC) GPU/48000 (2/240.000000/NVL) GPU/4D000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (4/24.000000/PXB) NET/2 (6/24.000000/PXN)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO GPU/48000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (0/5000.000000/LOC) GPU/4D000 (2/240.000000/NVL) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PXN) NET/2 (4/24.000000/PXB)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO GPU/4D000 :GPU/13000 (2/240.000000/NVL) GPU/19000 (2/240.000000/NVL) GPU/48000 (2/240.000000/NVL) GPU/4D000 (0/5000.000000/LOC) NVS/0 (1/240.000000/NVL) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PXN) NET/2 (4/24.000000/PXB)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO NET/0 :GPU/13000 (5/24.000000/PHB) GPU/19000 (5/24.000000/PHB) GPU/48000 (5/24.000000/PHB) GPU/4D000 (5/24.000000/PHB) CPU/0 (2/24.000000/PHB) NET/0 (0/5000.000000/LOC) NET/1 (5/24.000000/PHB) NET/2 (5/24.000000/PHB)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO NET/1 :GPU/13000 (4/24.000000/PXB) GPU/19000 (4/24.000000/PXB) GPU/48000 (6/24.000000/PHB) GPU/4D000 (6/24.000000/PHB) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (0/5000.000000/LOC) NET/2 (6/24.000000/PHB)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO NET/2 :GPU/13000 (6/24.000000/PHB) GPU/19000 (6/24.000000/PHB) GPU/48000 (4/24.000000/PXB) GPU/4D000 (4/24.000000/PXB) CPU/0 (3/24.000000/PHB) NET/0 (5/24.000000/PHB) NET/1 (6/24.000000/PHB) NET/2 (0/5000.000000/LOC)
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO s->nodes[GPU].nodes[0].gpu.rank: 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO s->nodes[GPU].nodes[1].gpu.rank: 1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO s->nodes[GPU].nodes[2].gpu.rank: 2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO s->nodes[GPU].nodes[3].gpu.rank: 3
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO comm->compCap: 80, comm->minCompCap: 80, comm->maxCompCap: 80
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO mock_comm[0].compCap: 80, mock_comm[0].minCompCap: 80, mock_comm[0].maxCompCap: 80
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/3 GPU/1 GPU/2 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/1 GPU/3 GPU/0 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/1 GPU/2 GPU/3 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/3 GPU/0 GPU/1 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 0]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/3 GPU/1 GPU/2 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/1 GPU/3 GPU/0 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 0]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/0 GPU/1 GPU/2 GPU/3 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/2 GPU/3 GPU/0 GPU/1 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 1]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/8 GPU/11 GPU/9 GPU/10 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/10 GPU/9 GPU/11 GPU/8 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 1]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/8 GPU/9 GPU/10 GPU/11 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/10 GPU/11 GPU/8 GPU/9 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 2]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/16 GPU/19 GPU/17 GPU/18 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/18 GPU/17 GPU/19 GPU/16 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 2]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/16 GPU/17 GPU/18 GPU/19 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/18 GPU/19 GPU/16 GPU/17 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 3]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/24 GPU/27 GPU/25 GPU/26 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/26 GPU/25 GPU/27 GPU/24 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 3]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/24 GPU/25 GPU/26 GPU/27 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/26 GPU/27 GPU/24 GPU/25 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 4]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/32 GPU/35 GPU/33 GPU/34 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/34 GPU/33 GPU/35 GPU/32 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 4]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/32 GPU/33 GPU/34 GPU/35 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/34 GPU/35 GPU/32 GPU/33 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 5]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/40 GPU/43 GPU/41 GPU/42 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/42 GPU/41 GPU/43 GPU/40 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 5]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/40 GPU/41 GPU/42 GPU/43 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/42 GPU/43 GPU/40 GPU/41 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 6]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/48 GPU/51 GPU/49 GPU/50 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/50 GPU/49 GPU/51 GPU/48 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 6]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/48 GPU/49 GPU/50 GPU/51 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/50 GPU/51 GPU/48 GPU/49 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock ring Graph[node: 7]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 4, crossNic 1, nChannels 2, bw 24.000000/24.000000, type NVL/PXB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/56 GPU/59 GPU/57 GPU/58 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/58 GPU/57 GPU/59 GPU/56 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO ------ Mock tree Graph[node: 7]: ------
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Pattern 1, crossNic 1, nChannels 2, bw 48.000000/24.000000, type NVL/PHB, sameChannels 0
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  0 : NET/1 GPU/56 GPU/57 GPU/58 GPU/59 NET/2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO  1 : NET/2 GPU/58 GPU/59 GPU/56 GPU/57 NET/1
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO -------------------------------

inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO comm->nNodes: 8
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO treeGraph.nChannels: 2, ringGraph.nChannels: 2
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO Channel 00/04 :     0    3    1    2   10    9   11    8   16   19   17   18   26   25   27   24   32   35   33   34   42   41   43   40   48   51   49   50   58   57   59   56    0    3    1    2   10    9   11    8   16   19   17   18   26   25   27   24   32   35   33   34   42   41   43   40   48   51   49   50   58   57   59   56

inspur-gpu-server-15:2570998:2571000 [0] graph/rings.cc:58 NCCL WARN Error : ring 0 does not contain rank 4
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO graph/connect.cc:507 -> 3
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO init.cc:1492 -> 3
inspur-gpu-server-15:2570998:2571000 [0] NCCL INFO init.cc:1913 -> 3
inspur-gpu-server-15:2570998:2570998 [0] NCCL INFO group.cc:418 -> 3
inspur-gpu-server-15:2570998:2570998 [0] NCCL INFO group.cc:95 -> 3
inspur-gpu-server-15:2570998:2570998 [0] NCCL INFO init.cc:2254 -> 3
Failed, NCCL error MultiDevOneP.cc:134 'internal error - please report this issue to the NCCL developers'

Besides, if run in the VM, there will be another error:

NCCL version 2.20.5-MockNCCL+cuda12.9
localhost:799:801 [0] NCCL INFO Mocking start: mock_nNodes(1), mock_nRanks(1), mock_nRanksPerNode(1).
localhost:799:801 [0] NCCL INFO info_index(0): 0x7fa4fc0038d0, rank: 0, cudaDev: 0, nvmlDev: 0, gdrSupport: 1,hostHash: 8982114052393301127, pidHash: 7559370754094648493, busId: ffffffffffffffff:ff:ff.f, cudaCompCap: -1.
run.sh: line 11:   799 Bus error               LD_LIBRARY_PATH="$SCRIPT_DIR/../build/lib/" NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=MOCK_ROOT,MOCK NCCL_CROSS_NIC=1 NCCL_TOPO_FILE=$SCRIPT_DIR/topo_8A100_4CX6.xml NCCL_NVLS_ENABLE=0 NCCL_NUM_MOCK_GPU=1 NCCL_NUM_MOCK_NODE=1 $1 ./MultiDevOneP $CORE_FILE

Expected Behavior

Should run successfully

Actual Behavior

Error

Environment

CUDA: 12.6, installed in /usr/local/cuda
OS: 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported sequence from the SimCCL folder: make -j src.build, then run make and bash run.sh in test, after checking src/device/generate.py permissions. Inspect the complete output around the reported segmentation error; the issue log is truncated. Done means the test run completes without the reported crash, with the expected result documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.