bytedance / bytedance/InfiniStore

GPU NIC topology awareness improvement

Open
#81 0 comments 0 reactions 1 assignee Claimed by @thesues View on GitHub
Dominant language
C++
Stars
437
Forks
44
PR merge metrics
No merged PRs in 30d

Description

GPU servers usually have more than one GPU and NIC. The latency between different GPUs and NICs can vary significantly, so it is necessary to be aware of the topology. we could use `nvidia-smi topo -m` to check
# Observation:

1. tensor on each GPU should always use the nearest NIC.
2. inner node: client should use the same NIC for local RDMA access. for example tensor on GPU7 will use NIC7 and connect to local NIC7
3. across nodes: client may choose random remote NIC for load balance. (although we have NUMA, but I found the latency between CPU memory and NIC is negligible( to be confirmed later)

# Solution:

## Server
Server will listen to all available NICs, and publish NIC's GID/LID/IP to all cluster.

## Client
`connect` function will add a new parameter(`preferred device`). if `preferred device` is specified, client will only open `preferred device` to save memory, if not, client will open all available local NICs.

* when connect remote node, client will choose random remote NIC(knowleges from cluster)
* when connect local node, client will always select the same NIC as the tensor's

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.