deepseek-ai / deepseek-ai/DeepEP

[Performance] Question About Dispatch Prologue Barrier and Excessive P99 ITL With V2

Open
#652 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

# Question

Hello, I am curious as to the reasoning behind this dispatch barrier in the prologue:

```
// Barrier without TMA store flush, without prologue grid sync
comm::gpu_barrier(
gin, workspace_layout, 0, rank_idx, sm_idx, thread_idx);
```

As far as I understand it, it's there to prevent hazard around the the workspace buffer with chained dispatch (or even combine) calls? In practice, we noticed that V2 has bad worst case behavior when a first dispatch is called in a forward pass due to rank token imbalance causing a straggler, leading to bad P99 ITL behavior.

My question is: what would it take to disable this barrier?

# Background and Further Information

While running R1 NVFP4 with decode only uniform routing and running two simultaneous deployments with 2 nodes on NVL72 (D8):
* deployment A - V2 Kernels
* deployment B - legacy kernels

under a benchmark that runs ISL 1 OSL 1500 (ISL 1 is not unusual for decode only benchmarks), I noticed these large P99 spikes that occur when the benchmark tool ramps up concurrency to get to the next target concurrency target:

Image

Note that the blue spikes (V2 Kernels deployment) correlate directly with the moment that the benchmarks ramps up concurrency.

At the same time I was doing kernel level measurements and noticed that when doing isolated dispatch/combine runs, the V2 kernels have much worse worst case behavior compared to the legacy kernels.

See benchmark results here: [results](https://gist.github.com/elvircrn/8f97e566ed5f6298a7db9484752bc734) and the corresponding source code reproducing them: [benchmark script](https://gist.github.com/elvircrn/b7e528721fbed31c0a9fda8bc1450ec1). Would also appreciate any feedback on this benchmark script!

Please note that the purple spikes for legacy kernels are due to KV$ utilization getting filled up and as such should be ignored.

Relevant results snippet:

```
local-rand bf16 elastic 159.9 207.2 232.1 1850.7 42.3 1.45 126 20 ***
```

(note the severe `1850.7` result and see the full table for context).

I then noticed a global barrier during dispatch prologue which I thought could make a straggler rank problem worse because it now makes it so that the V2 kernels cannot overlap the send phase with a straggler. I decided to try commenting it out:

```
// // Barrier without TMA store flush, without prologue grid sync
// comm::gpu_barrier(
// gin, workspace_layout, 0, rank_idx, sm_idx, thread_idx);
```

and noticed that the bad P99 ITL behavior for V2 kernels goes away:

Image

The spikes are still there due to token imbalance at this moment, but the spike is now higher for legacy kernels!

So my hypothesis is that when the concurrency spike happens, our router makes a wrong decision at one point and a token imbalance over mutiple ranks occurs and we get a straggler rank and the dispatch prologue barrier causes massive slowdowns during the first dispatch of the forward pass.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.