仿真异常结束，weight grad事件无法触发

请教下各位老师，是否遇到过仿真异常结束的问题：
1、修改： astra-sim-alibabacloud/inputs/config/SimAI.conf中CC_MODE 3
2、正常仿真：
命令：
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 80 -w ./aicb/results/workload/None-gpt_7B-world_size8-tp4-pp1-ep1-gbs2-mbs1-seq2048-MOE-False-GEMM-False-flash_attn-True.txt -n ./HPN_7_0_8_gpus_8_in_one_server_with_single_plane_100Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
输出：

***** info: input gradient collective for layer: layernorm is finished************
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
info: no weight grad collective for layer: layernorm
info: no input grad collective for layer: grad_param_compute
info: no weight grad collective for layer: grad_param_compute
info: no input grad collective for layer: grad_param_comm
chunk size is: 6488588288 , size is: 6488588288 , layer_num is: 1 , node: 0
info: reduce-scatter weight grad collective issued for layer: grad_param_comm, involved dimensions:  0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
info: no input grad collective for layer: grad_gather
Layer issue wg all gather at tick: 540210436
chunk size is: 3244294144 , size is: 3244294144 , layer_num is: 0 , node: 0
info: all-gather weight grad collective issued for layer: grad_gather, involved dimensions:  0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 540210436
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 12 in layer: grad_param_comm
DEBUG - Processing event: 15 in layer: grad_param_comm
DEBUG - PHY_MTP defined: NO
DEBUG - Current generator ID: 7 (0 is required for print)
DEBUG - Processing event: 15 in layer: grad_param_comm

3、异常仿真：
命令：
AS_LOG_LEVEL=INFO AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 80 -w ./aicb/results/workload/None-gpt_7B-world_size1024-tp4-pp1-ep1-gbs256-mbs1-seq4096-MOE-False-GEMM-False-flash_attn-True.txt -n ./HPN_7_0_1024_gpus_8_in_one_server_with_single_plane_100Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
输出：

DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
info: no weight grad collective for layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
DEBUG - Processing event: 16 in layer: layernorm
info: no input grad collective for layer: grad_param_compute
info: no weight grad collective for layer: grad_param_compute
info: no input grad collective for layer: grad_param_comm
chunk size is: 6522142720 , size is: 6522142720 , layer_num is: 1 , node: 0
info: reduce-scatter weight grad collective issued for layer: grad_param_comm, involved dimensions:  0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
info: no input grad collective for layer: grad_gather
Layer issue wg all gather at tick: 546518210
chunk size is: 3261071360 , size is: 3261071360 , layer_num is: 0 , node: 0
info: all-gather weight grad collective issued for layer: grad_gather, involved dimensions:  0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 546518210
root@05-06-78-999:/data/SimAI# 

4、根据打印分析，千卡7B CC_MODE 10时候无法触发weight grad事件
5、另外不修改astra-sim-alibabacloud/inputs/config/SimAI.conf配置，在千卡70B大模型仿真也遇到同样无法触发weight grad事件的问题

期待各位老师指点，万分感谢！

Start with astra-sim-alibabacloud/inputs/config/SimAI.conf and reproduce the two commands using the referenced workload and network files. Compare the normal and abnormal logs around the weight-grad event, including CC_MODE and the 70B case; done means identifying why the event is not triggered and documenting a reproducible resolution.