deepseek-ai / deepseek-ai/DeepEP

Why doesn't low-latency dispatch use system-level fence before issuing IBGDA writes?

Open
#587 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

(Clarification: https://github.com/deepseek-ai/DeepEP/issues/113 also inquires about the `__threadfence` in `ibgda_post_send`, but in this issue I am asking specifically about the consistency guarantee of the token data, not the WQE itself, so this is technically not a duplicate 😃)

To quote from `internode_ll::dispatch`, it first performs casting and writes FP8 data to the send buffer:

https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L213-L252

Then it issues IBGDA writes. `nvshmemi_ibgda_put_nbi_warp` is implemented such that the request is posted to NIC when `slot_idx % 4 == 3`.

https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L267

The problem is I don't see any use of system-level primitives here, to guarantee **the NIC will read the freshly cast data from the send buffer when it composes IB packets**. The closest thing I see is the `__threadfence` in `ibgda_post_send`, but it is not system-level (and I'm curious about that one, too).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with csrc/kernels/internode_ll.cu at the dispatch casting and send-buffer writes, the nvshmemi_ibgda_put_nbi_warp call, and ibgda_post_send. Trace the ordering and visibility guarantees between GPU memory, the NIC, and IB packet composition; done means documenting whether the existing synchronization is sufficient or identifying the required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.