deepseek-ai / deepseek-ai/DeepEP
Ways to do overlapping of computation and commucation.
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
Hello, I'm trying to use DeepEP and intergate it into Megatron (with this commit https://github.com/NVIDIA/Megatron-LM/commit/0d389f5e9935957b146c12b1dcdddc35cf9deb85), and accomplish overlapping of computation and commucation, I've noticed DeepEP is using previous_event and async_finish args in dispatch and combine methods to support the overlapping , whereas the FlexTokenDispatcher in that commit doesn't adopt any of them. For me, there are two ways to do that:
* modify FlexTokenDispatcher and pass args previous_event, async_finish etc. to make use of DeepEP's overlapping interface。
* keep FlexTokenDispatcher as it is, and manage the stream and event outside by myself before calling dispatch and combine under the context of torch.cuda.stream(), which means DeepEP's pevious_event, async_finish won't be used.
Can you guys give me some suggestions about the two ways in term of performance?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.