aliyun / aliyun/SimAI

NVLink Topology inconsistent with real cases?

Open
#60 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

I noticed the generated topology has something like:

0 8 3600Gbps 0.000025ms 0
1 8 3600Gbps 0.000025ms 0
2 8 3600Gbps 0.000025ms 0
3 8 3600Gbps 0.000025ms 0
4 8 3600Gbps 0.000025ms 0
5 8 3600Gbps 0.000025ms 0
6 8 3600Gbps 0.000025ms 0
7 8 3600Gbps 0.000025ms 0

But in an actual DGX H100 system, we have 8 GPUs and 4 NVSwitchs, each connecting link bandwidth is 1000Gbps or 800 Gbps actually, so I manually created a test topo:

0 8 1000Gbps 0.000025ms 0
1 8 1000Gbps 0.000025ms 0
2 8 1000Gbps 0.000025ms 0
3 8 1000Gbps 0.000025ms 0
4 8 1000Gbps 0.000025ms 0
5 8 1000Gbps 0.000025ms 0
6 8 1000Gbps 0.000025ms 0
7 8 1000Gbps 0.000025ms 0
0 9 800Gbps 0.000025ms 0
1 9 800Gbps 0.000025ms 0
2 9 800Gbps 0.000025ms 0
3 9 800Gbps 0.000025ms 0
4 9 800Gbps 0.000025ms 0
5 9 800Gbps 0.000025ms 0
6 9 800Gbps 0.000025ms 0
7 9 800Gbps 0.000025ms 0
0 10 800Gbps 0.000025ms 0
1 10 800Gbps 0.000025ms 0
2 10 800Gbps 0.000025ms 0
3 10 800Gbps 0.000025ms 0
4 10 800Gbps 0.000025ms 0
5 10 800Gbps 0.000025ms 0
6 10 800Gbps 0.000025ms 0
7 10 800Gbps 0.000025ms 0
0 11 1000Gbps 0.000025ms 0
1 11 1000Gbps 0.000025ms 0
2 11 1000Gbps 0.000025ms 0
3 11 1000Gbps 0.000025ms 0
4 11 1000Gbps 0.000025ms 0
5 11 1000Gbps 0.000025ms 0
6 11 1000Gbps 0.000025ms 0
7 11 1000Gbps 0.000025ms 0

But the allreduce result showed that the maximum busbw is just 200GB/s, and I can't enable NVLS , is this a routing problem with NS-3?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the generated NVLink topology and the NS-3 routing and NVLS configuration involved in the allreduce run. Compare the generated 3600Gbps topology with the manually specified DGX H100 links and inspect how the simulator reports bus bandwidth. Done means identifying whether routing or topology modeling explains the 200GB/s limit and documenting the required correction.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.