DCN+拓扑下ALLTOALL原语出错
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 13h 4m
- Merged PRs (30d)
- 1
Description
使用官方示例的拓扑生成
DCN+ topofile (Single ToR, non rail-optimized)
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo DCN+ -g 256 -psn 64 -bw 400Gbps
工作负载为
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 256 checkpoints: 0 checkpoint_initiates: 0
1
embedding_layer -1 556000 NONE 0 1 NONE 0 1 ALLTOALL 67108864 1
报错
info: all-to-all weight grad collective issued for layer: embedding_layer with size: 67108864, involved dimensions: 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 556004
assert failed. cond="m_ptr", msg="Attempted to dereference zero pointer", +0.001378159s 332 file=/home/chuyunliu/桌面/SimAI2/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h, line=649
terminate called without an active exception
查看代码感觉主要问题似乎在于weight_grad阶段选择了DP组进行通信,而其他阶段选择的是TP。
同时,当ALLTOALL消息尺寸非常小时也不会发生这个问题,我把它改为16777216就可以正常完成
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the DCN+ command from the report and the HYBRID_TRANSFORMER workload, comparing ALLTOALL sizes 67108864 and 16777216. Start by tracing weight_grad collective group selection and the ns-3 network backend path that reaches extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h:649. Done means the larger ALLTOALL completes without the zero-pointer assertion and the intended communication group is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100