NVLink Topology inconsistent with real cases?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 13h 4m
- Merged PRs (30d)
- 1
Description
I noticed the generated topology has something like:
0 8 3600Gbps 0.000025ms 0
1 8 3600Gbps 0.000025ms 0
2 8 3600Gbps 0.000025ms 0
3 8 3600Gbps 0.000025ms 0
4 8 3600Gbps 0.000025ms 0
5 8 3600Gbps 0.000025ms 0
6 8 3600Gbps 0.000025ms 0
7 8 3600Gbps 0.000025ms 0
But in an actual DGX H100 system, we have 8 GPUs and 4 NVSwitchs, each connecting link bandwidth is 1000Gbps or 800 Gbps actually, so I manually created a test topo:
0 8 1000Gbps 0.000025ms 0
1 8 1000Gbps 0.000025ms 0
2 8 1000Gbps 0.000025ms 0
3 8 1000Gbps 0.000025ms 0
4 8 1000Gbps 0.000025ms 0
5 8 1000Gbps 0.000025ms 0
6 8 1000Gbps 0.000025ms 0
7 8 1000Gbps 0.000025ms 0
0 9 800Gbps 0.000025ms 0
1 9 800Gbps 0.000025ms 0
2 9 800Gbps 0.000025ms 0
3 9 800Gbps 0.000025ms 0
4 9 800Gbps 0.000025ms 0
5 9 800Gbps 0.000025ms 0
6 9 800Gbps 0.000025ms 0
7 9 800Gbps 0.000025ms 0
0 10 800Gbps 0.000025ms 0
1 10 800Gbps 0.000025ms 0
2 10 800Gbps 0.000025ms 0
3 10 800Gbps 0.000025ms 0
4 10 800Gbps 0.000025ms 0
5 10 800Gbps 0.000025ms 0
6 10 800Gbps 0.000025ms 0
7 10 800Gbps 0.000025ms 0
0 11 1000Gbps 0.000025ms 0
1 11 1000Gbps 0.000025ms 0
2 11 1000Gbps 0.000025ms 0
3 11 1000Gbps 0.000025ms 0
4 11 1000Gbps 0.000025ms 0
5 11 1000Gbps 0.000025ms 0
6 11 1000Gbps 0.000025ms 0
7 11 1000Gbps 0.000025ms 0
But the allreduce result showed that the maximum busbw is just 200GB/s, and I can't enable NVLS , is this a routing problem with NS-3?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the generated NVLink topology and the NS-3 routing and NVLS configuration involved in the allreduce run. Compare the generated 3600Gbps topology with the manually specified DGX H100 links and inspect how the simulator reports bus bandwidth. Done means identifying whether routing or topology modeling explains the 200GB/s limit and documenting the required correction.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100