Azure / Azure/msccl

Question about synthesizing Allreduce

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
67
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Hi. Thanks for previous prompt response. I'm currently tring to synthesize the Allreduce for a custom topology(let's say a ring with 4 or 8 nodes as an example). Some strange problems occurs when doing so. I wonder if you can help.

My Codes:
```python
topology = Ring(num_Nodes=4)
from msccl.collectives import allgather,allreduce,reduce_scatter,reduce,alltoall
collective_allgather = allgather(topology.num_nodes())
collective_reduce_scatter = reduce_scatter(topology.num_nodes())
save_msccl_object(topology,'SG2260_topo_Ring4.json')
save_msccl_object(collective_allgather,'coll_allgather.json')
save_msccl_object(collective_reduce_scatter,'coll_reducescatter.json')
assert 0 == os.system('msccl solve pareto-optimal custom custom --topology-file SG2260_topo_Ring4.json --collective-file coll_allgather.json')
assert 0 == os.system('msccl solve pareto-optimal custom custom --topology-file SG2260_topo_Ring4.json --collective-file coll_reducescatter.json')
assert 0 == os.system('msccl compose allreduce ReduceScatter.n4-MYTP-steps2.rounds3.chunks2.msccl.json Allgather.n4-MYTP-steps2.rounds3.chunks2.msccl.json -o allreduce_ring4.json')
```
I stored the collective also into json file for better debug. The logged allreduce json has strange input and output map as follows:
```
"input_map": { "0": [0, 1], "1": [0, 1], "2": [0, 1], "3": [0, 1] },
"output_map": { "0": [0, 1], "1": [0, 1], "2": [0, 1], "3": [0, 1] },
"steps": [
{
"msccl_type": "step",
"rounds": 1,
"sends": [
[0, 2, 1],
[1, 2, 3],
[2, 3, 0],
[3, 3, 2],
[4, 0, 3],
[5, 0, 1],
[6, 1, 0],
[7, 1, 2]
]
},
{
"msccl_type": "step",
"rounds": 2,
"sends": [
[0, 1, 0],
[0, 3, 0],
[1, 1, 0],
[1, 3, 0],
[2, 0, 1],
[2, 2, 1],
[3, 0, 1],
[3, 2, 1],
[4, 1, 2],
[4, 3, 2],
[5, 1, 2],
[5, 3, 2],
[6, 0, 3],
[6, 2, 3],
[7, 0, 3],
[7, 2, 3]
]
},
{
"msccl_type": "step",
"rounds": 2,
"sends": [
[0, 0, 1],
[0, 0, 3],
[1, 0, 1],
[1, 0, 3],
[2, 1, 0],
[2, 1, 2],
[3, 1, 0],
[3, 1, 2],
[4, 2, 1],
[4, 2, 3],
[5, 2, 1],
[5, 2, 3],
[6, 3, 0],
[6, 3, 2],
[7, 3, 0],
[7, 3, 2]
]
},
{
"msccl_type": "step",
"rounds": 1,
"sends": [
[0, 1, 2],
[1, 3, 2],
[2, 0, 3],
[3, 2, 3],
[4, 3, 0],
[5, 1, 0],
[6, 0, 1],
[7, 2, 1]
]
}
],
"collective": {
"msccl_type": "collective",
"name": "Allreduce(n=4)",
"nodes": 4,
"chunks": [
{ "msccl_type": "chunk", "pre": [0], "post": [0, 1, 2, 3], "addr": 0 },
{ "msccl_type": "chunk", "pre": [1], "post": [0, 1, 2, 3], "addr": 0 },
{ "msccl_type": "chunk", "pre": [2], "post": [0, 1, 2, 3], "addr": 0 },
{ "msccl_type": "chunk", "pre": [3], "post": [0, 1, 2, 3], "addr": 0 }
```
I want to know how to understand this output. The chunck id seems to not match with each other. And the input/output map is not a proper solution for allreduce.
I'll be really appreciated and happy to offer other trail logs if anyone can help.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.