aliyun / aliyun/SimAI

DCN+拓扑下ALLTOALL原语出错

Open
#177 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

使用官方示例的拓扑生成

DCN+ topofile (Single ToR, non rail-optimized)

python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo DCN+ -g 256 -psn 64 -bw 400Gbps
工作负载为
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 256 checkpoints: 0 checkpoint_initiates: 0
1
embedding_layer -1 556000 NONE 0 1 NONE 0 1 ALLTOALL 67108864 1
报错
info: all-to-all weight grad collective issued for layer: embedding_layer with size: 67108864, involved dimensions: 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 556004
assert failed. cond="m_ptr", msg="Attempted to dereference zero pointer", +0.001378159s 332 file=/home/chuyunliu/桌面/SimAI2/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h, line=649
terminate called without an active exception
查看代码感觉主要问题似乎在于weight_grad阶段选择了DP组进行通信,而其他阶段选择的是TP。
同时,当ALLTOALL消息尺寸非常小时也不会发生这个问题,我把它改为16777216就可以正常完成

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the DCN+ command from the report and the HYBRID_TRANSFORMER workload, comparing ALLTOALL sizes 67108864 and 16777216. Start by tracing weight_grad collective group selection and the ns-3 network backend path that reaches extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h:649. Done means the larger ALLTOALL completes without the zero-pointer assertion and the intended communication group is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.