bytedance / bytedance/InfiniStore
GPU NIC topology awareness improvement
- Dominant language
- C++
- Stars
- 437
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
GPU servers usually have more than one GPU and NIC. The latency between different GPUs and NICs can vary significantly, so it is necessary to be aware of the topology. we could use `nvidia-smi topo -m` to check
# Observation:
1. tensor on each GPU should always use the nearest NIC.
2. inner node: client should use the same NIC for local RDMA access. for example tensor on GPU7 will use NIC7 and connect to local NIC7
3. across nodes: client may choose random remote NIC for load balance. (although we have NUMA, but I found the latency between CPU memory and NIC is negligible( to be confirmed later)
# Solution:
## Server
Server will listen to all available NICs, and publish NIC's GID/LID/IP to all cluster.
## Client
`connect` function will add a new parameter(`preferred device`). if `preferred device` is specified, client will only open `preferred device` to save memory, if not, client will open all available local NICs.
* when connect remote node, client will choose random remote NIC(knowleges from cluster)
* when connect local node, client will always select the same NIC as the tensor's
Contributor guide
Assessment
This issue has not been assessed yet.