aliyun / aliyun/SimAI

问1:例子文件中,ALLREDUCE是不是只走tp=8,所以改gpu=64和topo=64不会改变ncclFlowModel_EndToEnd.csv的结果?问2:h100的topo,tp>8为什么会没有输出

Open
#139 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

64 gpu test

python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 64 -gt H100 -bw 400Gbps -nvbw 1360Gbps
asw_switch_num: 8
psw_switch_num: 64
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Rail_Opti_SingleToR_64g_8gps_400Gbps_H100

AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce_64gpu.txt -n ./Rail_Opti_SingleToR_64g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

workload:
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 64 checkpoints: 0 checkpoint_initiates: 0
2
embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1
embedding_layer -1 556000 ALLREDUCE 67108864 1 NONE 0 1 NONE 0 1

result:
File name, Expose DP comm, Expose DP_EP comm, Expose TP comm, Expose_EP_comm, Expose_PP_comm, bubble time, total comp, total exposed comm, Total time
test1, 0 (0.00%), 0 (0.00%), 648 (36.81%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 1112 (63.19%), 648 (36.81%), 1760
layer_name,test1,fwd compute,wg compute,ig compute,fwd exposed comm,wg exposed comm,ig exposed comm,fwd total comm,algbw,busbw,wg total comm,algbw,busbw,ig total comm,algbw,busbw,workload finished at
embedding_layer,test1,556.000000,0.001000,0.001000,138.667000,0.000000,0.000000,138.667000,112.680016,197.190033,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
embedding_layer,test1,556.000000,0.001000,0.001000,509.147000,0.000000,0.000000,509.147000,122.754333,214.820084,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
SUM,test1,1112.000000,0.002000,0.002000,647.814000,0.000000,0.000000,647.814000,NONE,NONE,0.000000,NONE,NONE,0.000000,NONE,NONE
total exposed comm,647.820000,total comp,1112.004000,total time,1759.824000

32 gpu test

python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 32 -gt H100 -bw 400Gbps -nvbw 1360Gbps
asw_switch_num: 8
psw_switch_num: 64
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Rail_Opti_SingleToR_32g_8gps_400Gbps_H100

cmd:
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce_32gpu.txt -n ./Rail_Opti_SingleToR_32g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

/SimAI/ncclFlowModel_EndToEnd.csv

workload:
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 32 checkpoints: 0 checkpoint_initiates: 0
2
embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1
embedding_layer -1 556000 ALLREDUCE 67108864 1 NONE 0 1 NONE 0 1

result:
File name, Expose DP comm, Expose DP_EP comm, Expose TP comm, Expose_EP_comm, Expose_PP_comm, bubble time, total comp, total exposed comm, Total time
test1, 0 (0.00%), 0 (0.00%), 648 (36.81%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 1112 (63.19%), 648 (36.81%), 1760
layer_name,test1,fwd compute,wg compute,ig compute,fwd exposed comm,wg exposed comm,ig exposed comm,fwd total comm,algbw,busbw,wg total comm,algbw,busbw,ig total comm,algbw,busbw,workload finished at
embedding_layer,test1,556.000000,0.001000,0.001000,138.667000,0.000000,0.000000,138.667000,112.680016,197.190033,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
embedding_layer,test1,556.000000,0.001000,0.001000,509.147000,0.000000,0.000000,509.147000,122.754333,214.820084,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
SUM,test1,1112.000000,0.002000,0.002000,647.814000,0.000000,0.000000,647.814000,NONE,NONE,0.000000,NONE,NONE,0.000000,NONE,NONE
total exposed comm,647.820000,total comp,1112.004000,total time,1759.824000

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with inputs/topo/gen_Topo_Template.py and compare the generated H100 topologies for 32 and 64 GPUs. Run the corresponding microAllReduce workloads through bin/SimAI_simulator using inputs/config/SimAI.conf, then inspect ncclFlowModel_EndToEnd.csv and the ALLREDUCE setup. Done means documenting whether TP=8 makes the two results identical and identifying why H100 topologies with TP>8 produce no output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.