How to run alltoall?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 13h 4m
- Merged PRs (30d)
- 1
Description
When I replaced ALLREDUCE in microAllReduce.txt with ALLTOALL, the following error was reported. Are there any other places that need to be modified?Thanks!
id: embedding_layer , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
id: , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 6 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_36.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
chunk size is: 16777216 , size is: 16777216 , layer_num is: 0 , node: 0
info: all-to-all forward pass collective issued for layer: embedding_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
assert failed. cond="false", msg="We assume at least one NIC is alive", +0.000568010s 35 file=/root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.ccassert failed. cond="false", msg="We assume at least one NIC is alive", , line=251
+0.000568010sterminate called without an active exception
32 file=/root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc, line=Aborted (core dumped)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with microAllReduce.txt and reproduce the ALLTOALL run that reaches the assertion in extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc at line 251. Trace how the all-to-all collective is configured and identify the additional setup needed so the simulation completes without the NIC assertion.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100