aliyun / aliyun/SimAI

How to run alltoall?

Open
#44 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

When I replaced ALLREDUCE in microAllReduce.txt with ALLTOALL, the following error was reported. Are there any other places that need to be modified?Thanks!

id: embedding_layer , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 6 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_36.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
chunk size is: 16777216 , size is: 16777216 , layer_num is: 0 , node: 0
info: all-to-all forward pass collective issued for layer: embedding_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
assert failed. cond="false", msg="We assume at least one NIC is alive", +0.000568010s 35 file=/root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.ccassert failed. cond="false", msg="We assume at least one NIC is alive", , line=251
+0.000568010sterminate called without an active exception
32 file=/root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc, line=Aborted (core dumped)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with microAllReduce.txt and reproduce the ALLTOALL run that reaches the assertion in extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc at line 251. Trace how the all-to-all collective is configured and identify the additional setup needed so the simulation completes without the NIC assertion.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.