An error occurred while running the noNVLink topology

Sorry, I met a problem while running the noNVLink topology.

topology

I generated a noNVLink topology.
My code is:

python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -g 16 -gps 4 -nsps 0 -bw 200Gbps -gt H100 -asn 4 -psn 2 --ro

The topology file is:

workload

This workload can run normally in the NVLink topology, so there is no problem with the workload.

Instruction

sudo AS_SEND_LAT=2 AS_PXN_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce_1lines_16g_tp1_256M_dp.txt -n  ./Rail_Opti_Single
ToR_16g_4gps_200Gbps_H100_400G_noNvlink -c astra-sim-alibabacloud/inputs/config/SimAI.conf

However, an error occurred:

maxRtt=4488 maxBdp=112200
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 16 index in ring: 0 offset: 1total nodes in ring: 16
ring of node 0, id: 0 dimension: local total nodes in ring: 16 index in ring: 0 offset: 1total nodes in ring: 16
ring of node 0, id: 0 dimension: local total nodes in ring: 16 index in ring: 0 offset: 1total nodes in ring: 16
ring of node 0, id: 0 dimension: local total nodes in ring: 16 index in ring: 0 offset: 1total nodes in ring: 16
total nodes: 16
Success in opening workload file
model_parallel_NPU_group is 1
checkpoints layers are: 
layers initiating fwd_in_bckwd are: 
pp_commize:0
*****Warining: Input workload format mismatch. It may cause simulation error. Pleased use the latest AICB to generate.*****
id: embedding_layer , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 1 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_16.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
Segmentation fault

I don't know the reason.

Reproduce the failure with the shown SimAI_simulator command, the generated noNvlink topology, SimAI.conf, and microAllReduce workload. Compare this run with the working NVLink topology and inspect the simulator stage reached immediately before the Segmentation fault, including CSV output generation. Done means identifying the cause and confirming the same workload completes without the fault.