deepseek-ai / deepseek-ai/DeepEP
Why doesn't low-latency dispatch use system-level fence before issuing IBGDA writes?
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
(Clarification: https://github.com/deepseek-ai/DeepEP/issues/113 also inquires about the `__threadfence` in `ibgda_post_send`, but in this issue I am asking specifically about the consistency guarantee of the token data, not the WQE itself, so this is technically not a duplicate 😃)
To quote from `internode_ll::dispatch`, it first performs casting and writes FP8 data to the send buffer:
https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L213-L252
Then it issues IBGDA writes. `nvshmemi_ibgda_put_nbi_warp` is implemented such that the request is posted to NIC when `slot_idx % 4 == 3`.
https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L267
The problem is I don't see any use of system-level primitives here, to guarantee **the NIC will read the freshly cast data from the send buffer when it composes IB packets**. The closest thing I see is the `__threadfence` in `ibgda_post_send`, but it is not system-level (and I'm curious about that one, too).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with csrc/kernels/internode_ll.cu at the dispatch casting and send-buffer writes, the nvshmemi_ibgda_put_nbi_warp call, and ibgda_post_send. Trace the ordering and visibility guarantees between GPU memory, the NIC, and IB packet composition; done means documenting whether the existing synchronization is sufficient or identifying the required change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100