仿真程序跑着跑着就停下来了，停下来后也没其余的提示

钉钉群提问“ 感觉这个sim-ai是不是还有bug，我设置了13b的模型，tp=2,pp=1,world_size=16,micro_batch=1,global_batch=8,seq_len=2048，加速卡为A100，仿真程序跑着跑着就停下来了，停下来后也没其余的提示 ”

运行命令

(base) root@x08j03287:/disk2/futianhao/software2/SimAI# python ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo Spectrum-X -g 16 -gt A100 -bw 200Gbps -nvbw 600Gbps
asw_switch_num: 8
psw_switch_num: 64
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Spectrum-X_16g_8gps_200Gbps_A100

sh ./scripts/megatron_workload_with_aiob.sh -m 13 --world_size 16 --tensor_model_parallel_size 2 --pipeline_model_parallel 1 --frame Megatron --global_batch 8 --micro_batch 1 --seq_length 2048 --swiglu --use_flash_attn

这是workload命令

/disk2/futianhao/software2/SimAI/aicb/results/workload/None-gpt_13B-world_size16-tp2-pp1-ep1-gbs8-mbs1-seq2048-MOE-False-GEMM-False-flash_attn-True.txt

运行命令
AS_LOG_LEVEL=DEBUG AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w /disk2/futianhao/software2/SimAI/aicb/results/workload/None-gpt_13B-world_size16-tp2-pp1-ep1-gbs8-mbs1-seq2048-MOE-False-GEMM-False-flash_attn-True.txt -n ./Spectrum-X_16g_8gps_200Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

具体log：
生成的ncclFlowModel_EndToEnd.csv是空的

(base) root@x08j03287:/disk2/futianhao/software2/SimAI# AS_LOG_LEVEL=DEBUG AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w /disk2/futianhao/software2/SimAI/aicb/results/workload/None-gpt_13B-world_size16-tp2-pp1-ep1-gbs8-mbs1-seq2048-MOE-False-GEMM-False-flash_attn-True.txt -n ./Spectrum-X_16g_8gps_200Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
maxRtt=5080 maxBdp=127000
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 18 index in ring: 0 offset: 1total nodes in ring: 18
ring of node 0, id: 0 dimension: local total nodes in ring: 18 index in ring: 0 offset: 1total nodes in ring: 18
ring of node 0, id: 0 dimension: local total nodes in ring: 18 index in ring: 0 offset: 1total nodes in ring: 18
ring of node 0, id: 0 dimension: local total nodes in ring: 18 index in ring: 0 offset: 1total nodes in ring: 18
total nodes: 18
Success in opening workload file
model_parallel_NPU_group is 2
checkpoints layers are:
layers initiating fwd_in_bckwd are:
ring of node 0, id: 0 dimension: local total nodes in ring: 2 index in ring: 0 offset: 1total nodes in ring: 2
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 2total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 2 index in ring: 0 offset: 1total nodes in ring: 2
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 2total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 2 index in ring: 0 offset: 1total nodes in ring: 2
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 2total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 2 index in ring: 0 offset: 1total nodes in ring: 2
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 2total nodes in ring: 9
pp_commize:0
id: grad_norm , depen: -1 , wg_comp_time: 1
id: layernorm , depen: -1 , wg_comp_time: 1
id: moe_grad_norm1 , depen: -1 , wg_comp_time: 1
id: moe_grad_norm2 , depen: -1 , wg_comp_time: 1
id: embedding_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: attention_layer , depen: -1 , wg_comp_time: 1
id: mlp_layer , depen: -1 , wg_comp_time: 1
id: embedding_norm , depen: -1 , wg_comp_time: 1
id: cross_entropy1 , depen: -1 , wg_comp_time: 0
id: cross_entropy2 , depen: -1 , wg_comp_time: 0
id: cross_entropy3 , depen: -1 , wg_comp_time: 0
id: optimizer1 , depen: -1 , wg_comp_time: 0
id: optimizer2 , depen: -1 , wg_comp_time: 0
id: optimizer3 , depen: -1 , wg_comp_time: 0
id: optimizer4 , depen: -1 , wg_comp_time: 0
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 93 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_18.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
chunk size is: 10831790080 , size is: 10831790080 , layer_num is: 0 , node: 0
info: all-gather forward pass collective issued for layer: grad_norm, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,

运行analytical可以

./bin/SimAI_analytical -w /disk2/futianhao/software2/SimAI/aicb/results/workload/None-gpt_13B-world_size16-tp2-pp1-ep1-gbs8-mbs1-seq2048-MOE-False-GEMM-False-flash_attn-True.txt -g 9216 -g_p_s 8 -r test- -busbw example/busbw.yaml

有输出结果
/disk2/futianhao/software2/SimAI/results/test-EndToEnd.csv

File name, Expose DP comm, Expose DP_EP comm, Expose TP comm, Expose_EP_comm, Expose_PP_comm, bubble time, total comp, total exposed comm, Total time
test-, 53083 (45.42%), 0 (0.00%), 63785 (54.58%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 116868 (100.00%), 116868
layer_name,Analytical_test,fwd compute,wg compute,ig compute,fwd exposed comm,wg exposed comm,ig exposed comm,fwd total comm,algbw,busbw,wg total comm,algbw,busbw,ig total comm,algbw,busbw
grad_norm,Analytical_test,0,0,0,18014,53083,0,18014,560.00,280.00,53083,380.08,380.00,0,-nan,0.00
layernorm,Analytical_test,0,0,0,0,0,33626,0,-nan,0.00,0,-nan,0.00,33626,300.00,300.00
moe_grad_norm1,Analytical_test,0,0,0,0,0,0,0,-nan,0.00,0,-nan,-nan,0,-nan,0.00
moe_grad_norm2,Analytical_test,0,0,0,0,0,0,0,-nan,0.00,0,-nan,-nan,0,-nan,0.00
embedding_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
attention_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
mlp_layer,Analytical_test,0,0,0,65,0,65,65,300.00,300.00,0,-nan,0.00,65,300.00,300.00
embedding_norm,Analytical_test,0,0,0,1598,0,0,1598,300.00,300.00,0,-nan,0.00,0,-nan,0.00
cross_entropy1,Analytical_test,0,0,0,0,0,0,0,305.18,305.18,0,-nan,0.00,0,-nan,0.00
cross_entropy2,Analytical_test,0,0,0,0,0,0,0,305.18,305.18,0,-nan,0.00,0,-nan,0.00
cross_entropy3,Analytical_test,0,0,0,0,0,0,0,305.18,305.18,0,-nan,0.00,0,-nan,0.00
optimizer1,Analytical_test,0,0,0,0,0,0,0,317.89,317.89,0,-nan,0.00,0,-nan,0.00
optimizer2,Analytical_test,0,0,0,0,0,0,0,317.89,317.89,0,-nan,0.00,0,-nan,0.00
optimizer3,Analytical_test,0,0,0,0,0,0,0,317.89,317.89,0,-nan,0.00,0,-nan,0.00
optimizer4,Analytical_test,0,0,0,0,0,0,0,317.89,317.89,0,-nan,0.00,0,-nan,0.00

Reproduce the workload with scripts/megatron_workload_with_aiob.sh, then run bin/SimAI_simulator using the generated Spectrum-X topology and astra-sim-alibabacloud/inputs/config/SimAI.conf. Start from the simulator log after “simulator run” and compare the empty ncclFlowModel_EndToEnd.csv with the analytical output. Done means the simulation completes with a meaningful EndToEnd CSV for this workload, or reports the failure clearly.