aliyun / aliyun/SimAI

ncclFlowModel 输出文件为空 & 已知 Bug 长期未修复

Open
#247 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

在执行 SimAI 仿真后,ncclFlowModel_EndToEnd.csv 和 ncclFlowModel_detailed_72.csv 输出为空,参考 Issue #91 删除 workload 中前面DP和最后的优化器相关行后虽可输出,但导致仿真结果缺失 DP 通信数据;此外,使用过程中遇到多个已在历史 Issue 中报告的 Bug 长期未修复,想了解造成 NCCL 输出异常的根本原因及正确解决方案,同时询问仓库目前的维护状态和 Bug 修复计划。
simulation执行命令
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/H800-gpt_7B-world_size64-tp8-pp1-ep1-gbs512-mbs8-seq1440-MOE-False-GEMM-False-flash_attn-True.txt -n ./Spectrum-X_64g_8gps_400Gbps_H800 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
workload删除的部分

grad_gather -1 1 NONE 0 1 NONE 0 1 ALLGATHER 1627652096 100
grad_param_comm -1 1 NONE 0 1 NONE 0 1 REDUCESCATTER 3255304192 100
grad_param_compute -1 1 NONE 0 611776 NONE 0 1 NONE 0 100
layernorm -1 1 NONE 0 1 ALLREDUCE 1627652096 1 NONE 0 100
embedding_grads -1 1 NONE 0 1 ALLREDUCE 94371840 1 NONE

cross_entropy1 -1 0 ALLREDUCE 46080 0 NONE 0 0 NONE 0 100
cross_entropy2 -1 0 ALLREDUCE 46080 0 NONE 0 0 NONE 0 100
cross_entropy3 -1 0 ALLREDUCE 46080 0 NONE 0 0 NONE 0 100
optimizer1 -1 0 ALLREDUCE 4 0 NONE 0 0 NONE 0 100
optimizer2 -1 0 ALLREDUCE 4 0 NONE 0 0 NONE 0 100
optimizer3 -1 0 ALLREDUCE

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command shown with the H800 workload and SimAI.conf, then inspect the generated ncclFlowModel_EndToEnd.csv and ncclFlowModel_detailed_72.csv. Compare this with the workload changes referenced from Issue #91 to identify why output disappears and whether DP communication is lost; done means a documented root cause and a correct fix or reproduction-based explanation.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.