deepseek-ai / deepseek-ai/DeepEP
Why is so much memory used when initializing nvshmem? test_intranode
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
I created a large tensor and found that initializing nvshmem takes up a huge amount of memory.
```
test_ll_compatibility, num_rdma_bytes = True,128
if test_ll_compatibility:
ll_num_tokens, ll_hidden, ll_num_experts, ll_num_topk = 16, 5120, 256, 9
num_rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint(ll_num_tokens, ll_hidden, num_ranks, ll_num_experts)
buffer = deep_ep.Buffer(group, int(2e9), num_rdma_bytes, low_latency_mode=test_ll_compatibility,
num_qps_per_rank=(ll_num_experts // num_ranks if test_ll_compatibility else 1), explicitly_destroy=True)
tensor = torch.empty(138*2**30,device = "cuda",dtype = torch.int8)
torch.manual_seed(rank)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.