aliyun / aliyun/SimAI

仿真结果没有DP通信

Open
#199 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

我使用aicb生成workload命令:python -m workload_generator.SimAI_training_workload_generator --frame Megatron --gpu_type A100 --world_size=128 --tensor_model_parallel_size=4 --pipeline_model_parallel=4 --global_batch=512 --micro_batch=2 --epoch_num=10 --model_name=GPT-7.5B --hidden_size=4096 --num_layers=36 --seq_length=4096 --num_attention_heads=32 --vocab_size=50304 --max_position_embeddings=4096 --ffn_hidden_size=16384 --activation_func=gelu --dtype=bfloat16 --use_flash_attn --aiob_enable,生成了一个A100-GPT-7.5B-world_size128-tp4-pp4-ep1-gbs512-mbs2-seq4096-MOE-False-GEMM-False-flash_attn-True.txt文件,直接跑SimAI模型训练仿真(TP=4,PP=4,DP=8,GPU卡数=128),会不报错退出,打印日志如下:

Image 删除workload前面7行(参考issue:https://github.com/aliyun/SimAI/issues/107): Image

使用命令:AS_NVLS_ENABLE=1 AS_SEND_LAT=3 AS_LOG_LEVEL=DEBUG ./bin/SimAI_simulator -t 16 -w A100-GPT-7.5B-world_size128-tp4-pp4-ep1-gbs512-mbs2-seq4096-MOE-False-GEMM-False-flash_attn-True.txt -n Spectrum-X_128g_8gps_200Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf,可以生成ncclFlowModel_EndToEnd.csv文件,但是文件内只有TP通信,没有DP通信。
Image
A100-GPT-7.5B-world_size128-tp4-pp2-ep1-gbs256-mbs1-seq4096-MOE-False-GEMM-False-flash_attn-True.txt

ncclFlowModel_EndToEnd.csv

能帮忙看一下是哪里出现问题了么?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the workload with the provided Python command, then run SimAI_simulator with the attached workload, SimAI.conf, and output settings. Compare the generated ncclFlowModel_EndToEnd.csv with the TP=4, PP=4, DP=8 configuration and trace why DP communication is absent. Done means the simulator completes without errors and the CSV includes the expected DP communication.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.