问1:例子文件中,ALLREDUCE是不是只走tp=8,所以改gpu=64和topo=64不会改变ncclFlowModel_EndToEnd.csv的结果?问2:h100的topo,tp>8为什么会没有输出
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 13h 4m
- Merged PRs (30d)
- 1
Description
64 gpu test
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 64 -gt H100 -bw 400Gbps -nvbw 1360Gbps
asw_switch_num: 8
psw_switch_num: 64
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Rail_Opti_SingleToR_64g_8gps_400Gbps_H100
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce_64gpu.txt -n ./Rail_Opti_SingleToR_64g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
workload:
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 64 checkpoints: 0 checkpoint_initiates: 0
2
embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1
embedding_layer -1 556000 ALLREDUCE 67108864 1 NONE 0 1 NONE 0 1
result:
File name, Expose DP comm, Expose DP_EP comm, Expose TP comm, Expose_EP_comm, Expose_PP_comm, bubble time, total comp, total exposed comm, Total time
test1, 0 (0.00%), 0 (0.00%), 648 (36.81%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 1112 (63.19%), 648 (36.81%), 1760
layer_name,test1,fwd compute,wg compute,ig compute,fwd exposed comm,wg exposed comm,ig exposed comm,fwd total comm,algbw,busbw,wg total comm,algbw,busbw,ig total comm,algbw,busbw,workload finished at
embedding_layer,test1,556.000000,0.001000,0.001000,138.667000,0.000000,0.000000,138.667000,112.680016,197.190033,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
embedding_layer,test1,556.000000,0.001000,0.001000,509.147000,0.000000,0.000000,509.147000,122.754333,214.820084,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
SUM,test1,1112.000000,0.002000,0.002000,647.814000,0.000000,0.000000,647.814000,NONE,NONE,0.000000,NONE,NONE,0.000000,NONE,NONE
total exposed comm,647.820000,total comp,1112.004000,total time,1759.824000
32 gpu test
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 32 -gt H100 -bw 400Gbps -nvbw 1360Gbps
asw_switch_num: 8
psw_switch_num: 64
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Rail_Opti_SingleToR_32g_8gps_400Gbps_H100
cmd:
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce_32gpu.txt -n ./Rail_Opti_SingleToR_32g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
/SimAI/ncclFlowModel_EndToEnd.csv
workload:
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 32 checkpoints: 0 checkpoint_initiates: 0
2
embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1
embedding_layer -1 556000 ALLREDUCE 67108864 1 NONE 0 1 NONE 0 1
result:
File name, Expose DP comm, Expose DP_EP comm, Expose TP comm, Expose_EP_comm, Expose_PP_comm, bubble time, total comp, total exposed comm, Total time
test1, 0 (0.00%), 0 (0.00%), 648 (36.81%), 0 (0.00%), 0 (0.00%), 0 (0.00%), 1112 (63.19%), 648 (36.81%), 1760
layer_name,test1,fwd compute,wg compute,ig compute,fwd exposed comm,wg exposed comm,ig exposed comm,fwd total comm,algbw,busbw,wg total comm,algbw,busbw,ig total comm,algbw,busbw,workload finished at
embedding_layer,test1,556.000000,0.001000,0.001000,138.667000,0.000000,0.000000,138.667000,112.680016,197.190033,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
embedding_layer,test1,556.000000,0.001000,0.001000,509.147000,0.000000,0.000000,509.147000,122.754333,214.820084,0.000000,-nan,0.000000,0.000000,-nan,0.000000,1759.824000
SUM,test1,1112.000000,0.002000,0.002000,647.814000,0.000000,0.000000,647.814000,NONE,NONE,0.000000,NONE,NONE,0.000000,NONE,NONE
total exposed comm,647.820000,total comp,1112.004000,total time,1759.824000
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with inputs/topo/gen_Topo_Template.py and compare the generated H100 topologies for 32 and 64 GPUs. Run the corresponding microAllReduce workloads through bin/SimAI_simulator using inputs/config/SimAI.conf, then inspect ncclFlowModel_EndToEnd.csv and the ALLREDUCE setup. Done means documenting whether TP=8 makes the two results identical and identifying why H100 topologies with TP>8 produce no output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100