在自定义拓扑测试中 Aborted (core dumped)

我用如下代码生成了一个自定义拓扑，其中有16台机器，每个机器的4个GPU和3个ToR switch全连接，所有的48个ToR switch和64个交换层switch全连接。

def main():

    num_gpus = 64
    num_gpu_per_node = 4
    num_switch_per_node = 3
    num_board = 6
    num_switch_per_board = 4
    num_node = num_gpus // num_gpu_per_node
    
    counter = 0
    gpu_idxes = []
    for i in range(num_node):
        gpu_idxes.append([j for j in range(counter, counter + num_gpu_per_node)])
        counter += num_gpu_per_node
    
    
    node_switch_idxes = []
    for i in range(num_node):
        node_switch_idxes.append([j for j in range(counter, counter + num_switch_per_node)])
        counter += num_switch_per_node
    
    board_switch_idxes = []
    for i in range(num_board):
        board_switch_idxes.append([j for j in range(counter, counter + num_switch_per_board)])
        counter += num_switch_per_board
        
    node_switch_flat_idxes = [items for row in node_switch_idxes for items in row]
    board_switch_flat_idxes = [items for row in board_switch_idxes for items in row]
    
    with open("pcyn_topo", "w") as f:
        node_switch_link = num_gpu_per_node * num_switch_per_node * num_node
        board_switch_link = (num_board * num_switch_per_board) * (num_switch_per_node * num_node)
        #first_line = str(num_gpus + len(node_switch_flat_idxes) + len(board_switch_flat_idxes)) + " " + str(0) + " " + str(len(board_switch_flat_idxes) + len(node_switch_flat_idxes)) + " " + str(node_switch_link+board_switch_link) + " " + "A100" + "\n"
        first_line = str(num_gpus + len(node_switch_flat_idxes) + len(board_switch_flat_idxes)) + " " + str(num_gpu_per_node) + " " + str(len(node_switch_flat_idxes)) + " " + str(len(board_switch_flat_idxes)) + " " + str(node_switch_link+board_switch_link) + " " + "A100" + "\n"
        f.write(first_line)
        all_switches = node_switch_flat_idxes.copy()
        all_switches.extend(board_switch_flat_idxes)
        second_line = " ".join([str(i) for i in all_switches]) + "\n"
        f.write(second_line)
        
        # GPU to node switch links
        for i in range(num_node):
            for gpu_idx in gpu_idxes[i]:
                for switch_idx in node_switch_idxes[i]:
                    f.write(f"{gpu_idx} {switch_idx} 800Gbps 0.000025ms 0\n")
        
        # Node switch to board switch links
        for node_switch_idx in node_switch_flat_idxes:
            for board_switch_idx in board_switch_flat_idxes:
                f.write(f"{node_switch_idx} {board_switch_idx} 800Gbps 0.0005ms 0\n")


if __name__ == "__main__":
    main()

然后运行AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./pcyn_topo -c astra-sim-alibabacloud/inputs/config/SimAI.conf
会报错
assert failed. cond="rate2kmin.find(rate) != rate2kmin.end()", msg="must set kmin for each link speed", +0.000000000s -1 file=/workspace/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/scratch/common.h, line=850
terminate called without an active exception
Aborted (core dumped

但是如果把f.write(f"{node_switch_idx} {board_switch_idx} 800Gbps 0.0005ms 0\n")中的800Gbps换成400Gbps就不会有问题，似乎交换层连接的带宽是有限制的？

Reproduce the failure with the provided topology generator, simulator command, pcyn_topo, and SimAI.conf. Start at common.h line 850 and compare the 400Gbps and 800Gbps link cases, focusing on the rate2kmin lookup. Done means the custom topology runs without the assertion, or the required configuration for the 800Gbps link speed is clearly identified.