Manual single node with 1 nvswitch and 0 switch network topo caused coredump

I manually wrote a network topo file which describes a single node with 6 GPUs, no NVSwitch, and no swtich case.
Although theoretically nvswitch_num should be 0, but I found that this will cause MockNcclGroup function crashed:

        std::vector<int>NVSwitch;
        for(int k = 0;k<nNodesPerTPGroup;k++){
          NVSwitch.push_back(_NVSwitch[nvswitch_idx]);
          nvswitch_idx++;

Therefore, for the case of TP=4 and DP=1, I created a virtual nvswitch and wrote the below topo file:

~
7 6 1 0 14 A800
6
0 6 202Gbps 0.00173ms 0
1 6 202Gbps 0.0017ms 0
2 6 1481Gbps 0.00242ms 0
3 6 1481Gbps 0.00253ms 0
4 0 147Gbps 0.00225ms 0
4 1 146Gbps 0.00224ms 0
4 2 147Gbps 0.00222ms 0
4 3 147Gbps 0.00244ms 0
4 5 1481Gbps 0.00253ms 0
5 0 148Gbps 0.00228ms 0
5 1 148Gbps 0.00227ms 0
5 2 148Gbps 0.00227ms 0
5 3 147Gbps 0.00226ms 0
5 4 1481Gbps 0.00252ms 0

My command is as below, and I found the coredump happened in function uint32_t nic_idx = GetNicIdxOfQp(qp);:

AS_LOG_LEVEL=DEBUG AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w example/workload_analytical.txt -n ./one_server_A800_gpus_6 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

The log is as below:

id: mlp_moelayer , depen: -1 , wg_comp_time: 1
id: mlp_moelayer , depen: -1 , wg_comp_time: 1
id: final_column , depen: -1 , wg_comp_time: 875420
id: cross_entropy1 , depen: -1 , wg_comp_time: 0
id: cross_entropy2 , depen: -1 , wg_comp_time: 0
id: cross_entropy3 , depen: -1 , wg_comp_time: 0
id: optimizer1 , depen: -1 , wg_comp_time: 0
id: optimizer2 , depen: -1 , wg_comp_time: 0
id: optimizer3 , depen: -1 , wg_comp_time: 0
id: optimizer4 , depen: -1 , wg_comp_time: 0
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 1789 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_7.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
info: no forward pass collective for layer: grad_gather
info: no forward pass collective for layer: grad_param_comm
info: no forward pass collective for layer: grad_param_compute
info: no forward pass collective for layer: embedding_grads
info: no forward pass collective for layer: moe_grad_norm1
info: no forward pass collective for layer: moe_grad_norm2
chunk size is: 50331648 , size is: 50331648 , layer_num is: 6 , node: 0
info: all-reduce forward pass collective issued for layer: embedding_layer, involved dimensions:  1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
assert failed. cond="false", msg="We assume at least one NIC is alive", +0.000625753s assert failed. cond="4 file=/src/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc, line=251
falseterminate called without an active exception
", msg="We assume at least one NIC is alive", Aborted (core dumped)

Start at extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc:251, where the log reports that no NIC is alive, and trace the GetNicIdxOfQp(qp) call back through MockNcclGroup. Compare that path with the supplied one_server_A800_gpus_6 topology and its zero-NVSwitch case. Done means the reported single-node topology no longer aborts or core dumps when running the supplied command.