tp=24的仿真问题

【问题】
拿workload做了一个实验，截取了几个layer的workload传给simulation，同样的通信size下，TP=24（跨机）的仿真结果中，fwd total comm的值每层不太一样，TP=8时各层是一致的。所以有些疑惑，为啥会有这样的差异。

“fwd total comm的值每层不太一样”，计算时间一样的
fwd exposed comm = fwd total comm，好像是这个出问题，这个是咋算的，好像有overlap还是做了啥？

【目的】
GPU=24, TP=24, 使用ring allreduce集合通信算法，实验TP跨机时的通信流量

【文件说明】
Rail_Opti_SingleToR_24g_8gps_400Gbps_H100：拓扑文件

SimAI.conf : conf文件

test_tp_24.txt： workload，mlp fwd通信size=33554432 (对应flow.txt中流的执行时间无规律跳变)

test_tp_24_mlp15360.txt： workload，mlp fwd通信size=改小为15360 (对应flow.txt中流的执行时间固定不变)

【使用的命令】
AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 1 -w ./example/test_tp_24.txt -n ./Rail_Opti_SingleToR_24g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 1 -w ./example/test_tp_24_mlp15360.txt -n ./Rail_Opti_SingleToR_24g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

【Rail_Opti_SingleToR_24g_8gps_400Gbps_H100：拓扑文件】

38 8 3 11 72 H100
24 25 26 27 28 29 30 31 32 33 34 35 36 37
0 24 2880Gbps 0.000025ms 0
0 27 400Gbps 0.0005ms 0
1 24 2880Gbps 0.000025ms 0
1 28 400Gbps 0.0005ms 0
2 24 2880Gbps 0.000025ms 0
2 29 400Gbps 0.0005ms 0
3 24 2880Gbps 0.000025ms 0
3 30 400Gbps 0.0005ms 0
4 24 2880Gbps 0.000025ms 0
4 31 400Gbps 0.0005ms 0
5 24 2880Gbps 0.000025ms 0
5 32 400Gbps 0.0005ms 0
6 24 2880Gbps 0.000025ms 0
6 33 400Gbps 0.0005ms 0
7 24 2880Gbps 0.000025ms 0
7 34 400Gbps 0.0005ms 0
8 25 2880Gbps 0.000025ms 0
8 27 400Gbps 0.0005ms 0
9 25 2880Gbps 0.000025ms 0
9 28 400Gbps 0.0005ms 0
10 25 2880Gbps 0.000025ms 0
10 29 400Gbps 0.0005ms 0
11 25 2880Gbps 0.000025ms 0
11 30 400Gbps 0.0005ms 0
12 25 2880Gbps 0.000025ms 0
12 31 400Gbps 0.0005ms 0
13 25 2880Gbps 0.000025ms 0
13 32 400Gbps 0.0005ms 0
14 25 2880Gbps 0.000025ms 0
14 33 400Gbps 0.0005ms 0
15 25 2880Gbps 0.000025ms 0
15 34 400Gbps 0.0005ms 0
16 26 2880Gbps 0.000025ms 0
16 27 400Gbps 0.0005ms 0
17 26 2880Gbps 0.000025ms 0
17 28 400Gbps 0.0005ms 0
18 26 2880Gbps 0.000025ms 0
18 29 400Gbps 0.0005ms 0
19 26 2880Gbps 0.000025ms 0
19 30 400Gbps 0.0005ms 0
20 26 2880Gbps 0.000025ms 0
20 31 400Gbps 0.0005ms 0
21 26 2880Gbps 0.000025ms 0
21 32 400Gbps 0.0005ms 0
22 26 2880Gbps 0.000025ms 0
22 33 400Gbps 0.0005ms 0
23 26 2880Gbps 0.000025ms 0
23 34 400Gbps 0.0005ms 0
27 35 400Gbps 0.0005ms 0
27 36 400Gbps 0.0005ms 0
27 37 400Gbps 0.0005ms 0
28 35 400Gbps 0.0005ms 0
28 36 400Gbps 0.0005ms 0
28 37 400Gbps 0.0005ms 0
29 35 400Gbps 0.0005ms 0
29 36 400Gbps 0.0005ms 0
29 37 400Gbps 0.0005ms 0
30 35 400Gbps 0.0005ms 0
30 36 400Gbps 0.0005ms 0
30 37 400Gbps 0.0005ms 0
31 35 400Gbps 0.0005ms 0
31 36 400Gbps 0.0005ms 0
31 37 400Gbps 0.0005ms 0
32 35 400Gbps 0.0005ms 0
32 36 400Gbps 0.0005ms 0
32 37 400Gbps 0.0005ms 0
33 35 400Gbps 0.0005ms 0
33 36 400Gbps 0.0005ms 0
33 37 400Gbps 0.0005ms 0
34 35 400Gbps 0.0005ms 0
34 36 400Gbps 0.0005ms 0
34 37 400Gbps 0.0005ms 0

[SimAI.conf : conf文件]

ENABLE_QCN 1
USE_DYNAMIC_PFC_THRESHOLD 1

PACKET_PAYLOAD_SIZE 9000

FLOW_FILE /etc/astra-sim/simulation/flow1.txt
TRACE_FILE /etc/astra-sim/simulation/trace1.txt
TRACE_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_mix.tr
FCT_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_fct.txt
PFC_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_pfc.txt
TRACE_FLOW_FILE trace_flow.txt

SIMULATOR_STOP_TIME 40000000000000.00

CC_MODE 1
ALPHA_RESUME_INTERVAL 1
RATE_DECREASE_INTERVAL 4
CLAMP_TARGET_RATE 0
RP_TIMER 900
EWMA_GAIN 0.00390625
FAST_RECOVERY_TIMES 1
RATE_AI 50Mb/s
RATE_HAI 100Mb/s
MIN_RATE 100Mb/s
DCTCP_RATE_AI 1000Mb/s

ERROR_RATE_PER_LINK 0.0000
L2_CHUNK_SIZE 4000
L2_ACK_INTERVAL 1
L2_BACK_TO_ZERO 0

HAS_WIN 1
GLOBAL_T 0
VAR_WIN 1
FAST_REACT 1
U_TARGET 0.95
MI_THRESH 0
INT_MULTI 1
MULTI_RATE 0
SAMPLE_FEEDBACK 0
PINT_LOG_BASE 1.05
PINT_PROB 1.0

RATE_BOUND 1

ACK_HIGH_PRIO 0

LINK_DOWN 0 0 0

ENABLE_TRACE 1

KMAX_MAP 7 25000000000 400 50000000000 800 100000000000 1600 200000000000 1200 400000000000 3200 800000000000 2800 1600000000000 2400
KMIN_MAP 7 25000000000 100 50000000000 200 100000000000 400 200000000000 300 400000000000 800 800000000000 600 1600000000000 600
PMAX_MAP 7 25000000000 0.2 50000000000 0.2 100000000000 0.2 200000000000 0.8 400000000000 0.2 800000000000 0.2 1600000000000 0.2

BUFFER_SIZE 32

QLEN_MON_FILE /etc/astra-sim/simulation/llama_hpn7_qlen.txt
BW_MON_FILE /etc/astra-sim/simulation/llama_hpn7_bw.txt
RATE_MON_FILE /etc/astra-sim/simulation/llama_hpn7_rate.txt
CNP_MON_FILE /etc/astra-sim/simulation/llama_hpn7_cnp.txt
MON_START 0
MON_END 20000
QP_MON_INTERVAL 100
QLEN_MON_INTERVAL 10000
BW_MON_INTERVAL 10000

SKIP_NCCL 0
COLLECTION_INTERVAL 0.005
ECMP_MODE 0
LEAF_HASH_ALGO 10
SPINE_HASH_ALGO 11
SCHEDULE_MONITOR 0
PFC_ALPHA 3

[test_tp_24_mlp15360.txt]

HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 24 ep: 1 pp: 1 vpp: 32 ga: 8 all_gpus: 24 checkpoints: 0 checkpoint_initiates: 0 pp_comm: 2437156
9
embedding_grads -1 1 NONE 0 1 ALLREDUCE 13554432 1 NONE 0 100
embedding_layer -1 427384 ALLREDUCE 23554432 1 NONE 0 98872 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 15360 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 53554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 15360 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 15360 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100

[test_tp_24.txt]
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 24 ep: 1 pp: 1 vpp: 32 ga: 8 all_gpus: 24 checkpoints: 0 checkpoint_initiates: 0 pp_comm: 2437156
9
embedding_grads -1 1 NONE 0 1 ALLREDUCE 13554432 1 NONE 0 100
embedding_layer -1 427384 ALLREDUCE 23554432 1 NONE 0 98872 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 33554432 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 53554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 33554432 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100
mlp_layer -1 3563830 ALLREDUCE 33554432 712766 NONE 0 811638 NONE 0 100
attention_layer -1 1169142 ALLREDUCE 43554432 2338284 NONE 0 2437156 NONE 0 100

[运行命令1]
AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 1 -w /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/test_tp_24.txt -n /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/Rail_Opti_SingleToR_24g_8gps_400Gbps_H100 -c /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/SimAI.conf

【运行输出】
“fwd total comm的值每层不太一样”
fwd exposed comm = fwd total comm，好像是这个出问题，这个是咋算的，好像有overlap还是做了啥？

【运行时 output】
AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 1 -w /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/test_tp_24.txt -n /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/Rail_Opti_SingleToR_24g_8gps_400Gbps_H100 -c /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/SimAI.conf
maxRtt=4720 maxBdp=236000
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 27 index in ring: 0 offset: 1total nodes in ring: 27
ring of node 0, id: 0 dimension: local total nodes in ring: 27 index in ring: 0 offset: 1total nodes in ring: 27
ring of node 0, id: 0 dimension: local total nodes in ring: 27 index in ring: 0 offset: 1total nodes in ring: 27
ring of node 0, id: 0 dimension: local total nodes in ring: 27 index in ring: 0 offset: 1total nodes in ring: 27
total nodes: 27
Success in opening workload file
model_parallel_NPU_group is 24
checkpoints layers are:
layers initiating fwd_in_bckwd are:
ring of node 0, id: 0 dimension: local total nodes in ring: 24 index in ring: 0 offset: 1total nodes in ring: 24
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 24total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 24 index in ring: 0 offset: 1total nodes in ring: 24
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 24total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 24 index in ring: 0 offset: 1total nodes in ring: 24
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 24total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 24 index in ring: 0 offset: 1total nodes in ring: 24
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 24total nodes in ring: 1
pp_commize:2437156
Warining: Input workload format mismatch. It may cause simulation error. Pleased use the latest AICB to generate.
id: embedding_grads , depen: -1 , wg_comp_time: 1
id: embedding_layer , depen: -1 , wg_comp_time: 98872
id: attention_layer , depen: -1 , wg_comp_time: 2437156
id: mlp_layer , depen: -1 , wg_comp_time: 811638
id: attention_layer , depen: -1 , wg_comp_time: 2437156
id: mlp_layer , depen: -1 , wg_comp_time: 811638
id: attention_layer , depen: -1 , wg_comp_time: 2437156
id: mlp_layer , depen: -1 , wg_comp_time: 811638
id: attention_layer , depen: -1 , wg_comp_time: 2437156
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 9 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_27.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
info: no forward pass collective for layer: embedding_grads
chunk size is: 23554432 , size is: 23554432 , layer_num is: 1 , node: 0
info: all-reduce forward pass collective issued for layer: embedding_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: embedding_layer is finished************
chunk size is: 43554432 , size is: 43554432 , layer_num is: 2 , node: 0
info: all-reduce forward pass collective issued for layer: attention_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: attention_layer is finished************
chunk size is: 33554432 , size is: 33554432 , layer_num is: 3 , node: 0
info: all-reduce forward pass collective issued for layer: mlp_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: mlp_layer is finished************
chunk size is: 53554432 , size is: 53554432 , layer_num is: 4 , node: 0
info: all-reduce forward pass collective issued for layer: attention_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: attention_layer is finished************
chunk size is: 33554432 , size is: 33554432 , layer_num is: 5 , node: 0
info: all-reduce forward pass collective issued for layer: mlp_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: mlp_layer is finished************
chunk size is: 43554432 , size is: 43554432 , layer_num is: 6 , node: 0
info: all-reduce forward pass collective issued for layer: attention_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: attention_layer is finished************
chunk size is: 33554432 , size is: 33554432 , layer_num is: 7 , node: 0
info: all-reduce forward pass collective issued for layer: mlp_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: mlp_layer is finished************
chunk size is: 43554432 , size is: 43554432 , layer_num is: 8 , node: 0
info: all-reduce forward pass collective issued for layer: attention_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: fwd pass comm collective for layer: attention_layer is finished************
info: no input grad collective for layer: attention_layer
info: no weight grad collective for layer: attention_layer
info: no input grad collective for layer: mlp_layer
info: no weight grad collective for layer: mlp_layer
info: no input grad collective for layer: attention_layer
info: no weight grad collective for layer: attention_layer
info: no input grad collective for layer: mlp_layer
info: no weight grad collective for layer: mlp_layer
info: no input grad collective for layer: attention_layer
info: no weight grad collective for layer: attention_layer
info: no input grad collective for layer: mlp_layer
info: no weight grad collective for layer: mlp_layer
info: no input grad collective for layer: attention_layer
info: no weight grad collective for layer: attention_layer
info: no input grad collective for layer: embedding_layer
info: no weight grad collective for layer: embedding_layer
chunk size is: 13554432 , size is: 13554432 , layer_num is: 0 , node: 0
info: all-reduce input grad collective issued for layer: embedding_grads, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
***** info: input gradient collective for layer: embedding_grads is finished************
info: no weight grad collective for layer: embedding_grads
pass: 0 finished at time: 43851036
workload stats for the job scheduled at NPU offset: 0

Layer id: embedding_grads
Total collectives issued for this layer: 1
************************* Workload stats ************************* embedding_grads
id: embedding_grads ,Total cycles spent on fwd pass compute: 1
id: embedding_grads ,Total cycles spent on weight grad compute: 1
id: embedding_grads ,Total cycles spent on input grad compute: 1
id: embedding_grads ,Total cycles spent idle waiting for fwd finish: 0
id: embedding_grads ,Total cycles spent idle waiting for weight grad finish: 0
id: embedding_grads ,Total cycles spent idle waiting for input grad finish: 228954
id: embedding_grads ,Total cycles spent on fwd pass comm: 0
id: embedding_grads ,Total cycles spent on weight grad comm: 0
id: embedding_grads ,Total cycles spent on input grad comm: 228954

Layer id: embedding_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* embedding_layer
id: embedding_layer ,Total cycles spent on fwd pass compute: 427384
id: embedding_layer ,Total cycles spent on weight grad compute: 98872
id: embedding_layer ,Total cycles spent on input grad compute: 1
id: embedding_layer ,Total cycles spent idle waiting for fwd finish: 273746
id: embedding_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: embedding_layer ,Total cycles spent idle waiting for input grad finish: 0
id: embedding_layer ,Total cycles spent on fwd pass comm: 273746
id: embedding_layer ,Total cycles spent on weight grad comm: 0
id: embedding_layer ,Total cycles spent on input grad comm: 0

Layer id: attention_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* attention_layer
id: attention_layer ,Total cycles spent on fwd pass compute: 1169142
id: attention_layer ,Total cycles spent on weight grad compute: 2437156
id: attention_layer ,Total cycles spent on input grad compute: 2338284
id: attention_layer ,Total cycles spent idle waiting for fwd finish: 515449
id: attention_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: attention_layer ,Total cycles spent idle waiting for input grad finish: 0
id: attention_layer ,Total cycles spent on fwd pass comm: 515449
id: attention_layer ,Total cycles spent on weight grad comm: 0
id: attention_layer ,Total cycles spent on input grad comm: 0

Layer id: mlp_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* mlp_layer
id: mlp_layer ,Total cycles spent on fwd pass compute: 3563830
id: mlp_layer ,Total cycles spent on weight grad compute: 811638
id: mlp_layer ,Total cycles spent on input grad compute: 712766
id: mlp_layer ,Total cycles spent idle waiting for fwd finish: 456379
id: mlp_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: mlp_layer ,Total cycles spent idle waiting for input grad finish: 0
id: mlp_layer ,Total cycles spent on fwd pass comm: 456379
id: mlp_layer ,Total cycles spent on weight grad comm: 0
id: mlp_layer ,Total cycles spent on input grad comm: 0

Layer id: attention_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* attention_layer
id: attention_layer ,Total cycles spent on fwd pass compute: 1169142
id: attention_layer ,Total cycles spent on weight grad compute: 2437156
id: attention_layer ,Total cycles spent on input grad compute: 2338284
id: attention_layer ,Total cycles spent idle waiting for fwd finish: 494570
id: attention_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: attention_layer ,Total cycles spent idle waiting for input grad finish: 0
id: attention_layer ,Total cycles spent on fwd pass comm: 494570
id: attention_layer ,Total cycles spent on weight grad comm: 0
id: attention_layer ,Total cycles spent on input grad comm: 0

Layer id: mlp_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* mlp_layer
id: mlp_layer ,Total cycles spent on fwd pass compute: 3563830
id: mlp_layer ,Total cycles spent on weight grad compute: 811638
id: mlp_layer ,Total cycles spent on input grad compute: 712766
id: mlp_layer ,Total cycles spent idle waiting for fwd finish: 400241
id: mlp_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: mlp_layer ,Total cycles spent idle waiting for input grad finish: 0
id: mlp_layer ,Total cycles spent on fwd pass comm: 400241
id: mlp_layer ,Total cycles spent on weight grad comm: 0
id: mlp_layer ,Total cycles spent on input grad comm: 0

Layer id: attention_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* attention_layer
id: attention_layer ,Total cycles spent on fwd pass compute: 1169142
id: attention_layer ,Total cycles spent on weight grad compute: 2437156
id: attention_layer ,Total cycles spent on input grad compute: 2338284
id: attention_layer ,Total cycles spent idle waiting for fwd finish: 521723
id: attention_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: attention_layer ,Total cycles spent idle waiting for input grad finish: 0
id: attention_layer ,Total cycles spent on fwd pass comm: 521723
id: attention_layer ,Total cycles spent on weight grad comm: 0
id: attention_layer ,Total cycles spent on input grad comm: 0

Layer id: mlp_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* mlp_layer
id: mlp_layer ,Total cycles spent on fwd pass compute: 3563830
id: mlp_layer ,Total cycles spent on weight grad compute: 811638
id: mlp_layer ,Total cycles spent on input grad compute: 712766
id: mlp_layer ,Total cycles spent idle waiting for fwd finish: 678441
id: mlp_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: mlp_layer ,Total cycles spent idle waiting for input grad finish: 0
id: mlp_layer ,Total cycles spent on fwd pass comm: 678441
id: mlp_layer ,Total cycles spent on weight grad comm: 0
id: mlp_layer ,Total cycles spent on input grad comm: 0

Layer id: attention_layer
Total collectives issued for this layer: 1
************************* Workload stats ************************* attention_layer
id: attention_layer ,Total cycles spent on fwd pass compute: 1169142
id: attention_layer ,Total cycles spent on weight grad compute: 2437156
id: attention_layer ,Total cycles spent on input grad compute: 2338284
id: attention_layer ,Total cycles spent idle waiting for fwd finish: 712217
id: attention_layer ,Total cycles spent idle waiting for weight grad finish: 0
id: attention_layer ,Total cycles spent idle waiting for input grad finish: 0
id: attention_layer ,Total cycles spent on fwd pass comm: 712217
id: attention_layer ,Total cycles spent on weight grad comm: 0
id: attention_layer ,Total cycles spent on input grad comm: 0

all passes finished at time: 43851037, id of first layer: embedding_grads
path to create csvs is: ./ncclFlowModel_
success in openning file

Time to exit: Mon Jun 9 13:18:26 2025
all-reduce Collective implementation: NcclFlowModel
reduce-scatter Collective implementation: NcclFlowModel
all-gather Collective implementation: NcclFlowModel
all-to-all Collective implementation: NcclFlowModel
Collective optimization: baseline
Total sim duration: 0:2 hours
Total streams injected: 9
Total streams finished: 9
Percentage of finished streams: 100 %

sim_finish on sent, Thread id: 139704076007232
All data sent from node 0 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 0 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 1 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 1 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 2 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 2 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 3 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 3 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 4 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 4 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 5 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 5 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 6 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 6 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 7 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 7 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 8 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 8 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 9 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 9 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 10 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 10 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 11 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 11 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 12 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 12 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 13 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 13 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 14 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 14 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 15 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 15 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 16 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 16 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 17 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 17 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 18 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 18 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 19 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 19 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 20 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 20 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 21 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 21 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 22 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 22 is 617146304
sim_finish on sent, Thread id: 139704076007232
All data sent from node 23 is 617146304
sim_finish on received, Thread id: 139704076007232
All data received by node 23 is 617146304

【运行命令2】
AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 1 -w /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/test_tp_24_mlp15360.txt -n /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/Rail_Opti_SingleToR_24g_8gps_400Gbps_H100 -c /disk2/futianhao/software2/SimAI/fth_test/workload_forSimAI/SimAI.conf

Description

Contributor guide

First steps

Research direction

Assessment