aliyun / aliyun/SimAI

SimAI_simulator不输出结果,bug修复求助

Open
#264 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

使用aicb生成了workload,执行命令如下
sh ./scripts/megatron_workload_with_aiob.sh -m 7 \ --world_size 128 --tp 8 --pp 1 \ --frame Megatron --gpu_type H800 \ --global_batch 1024 --micro_batch 8 \ --seq_length 4096 --swiglu \ --use_flash_attn --aiob_enable
生成的workload是没有问题的。
运行./bin/SimAI_analytical可以输出结果,没有问题。
但是simulation仿真没有结果输出,我的执行环境也是和仓库一致的。GCC/G++ 9.4.0, python 3.8.10 in Ubuntu 20.04
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/H800-gpt_7B-world_size128-tp8-pp1-ep1-gbs1024-mbs8-seq4096-MOE-False-GEMM-False-flash_attn-True.txt \ -n ./Spectrum-X_128g_8gps_400Gbps_H800 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
如果用仓库给的workload例子,可以输出结果
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 32 checkpoints: 0 checkpoint_initiates: 0 2 embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1 embedding_layer -1 556000 ALLREDUCE 67108864 1
但是这个workload过于简单,不是一个完整的模型。
所以代码应该是有bug的,这个问题在很多issue中已经提到,并未解决。你们有可能解决这个bug,并更新代码仓库吗?
公司领导现在想要用这个仓库做很多工作,目前simai不是很work。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command using scripts/megatron_workload_with_aiob.sh, the generated workload file, bin/SimAI_simulator, and astra-sim-alibabacloud/inputs/config/SimAI.conf. Compare its behavior with the repository workload example and inspect where the simulator reads or processes the larger workload. Done means the provided workload produces simulation output under the documented environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell, ubuntu
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.