deepseek-ai / deepseek-ai/DeepEP

[BUG] test_internode hangs and times out on repeated runs

Open
#364 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

When test_internode is run in a tight loop for stress testing, it randomly hangs and eventually times out.
It seems to be related to TMA function.

Screenshot of the hang:
Image

### Environment
- Commit Hash: The latest commit on main (e3908bf5bd0cc6265bcb225d15cd8c996d4759ef)
- Hardware: Reproducible on both 2/4-node H20 and 2/4-node H200 setups. ConnectX-7 with dual port 200G. RoCEv2 with PFC.
- Image: ngc 24.02
- Driver / CUDA Version: <570.148.08 / 12.8>

### Steps to Reproduce
I modified the test script for stress testing.

https://github.com/deepseek-ai/DeepEP/blob/e3908bf5bd0cc6265bcb225d15cd8c996d4759ef/tests/test_internode.py#L174-L176
Adding a return to skip performance test.
```python
if local_rank == 0:
print('', flush=True)
return
```
https://github.com/deepseek-ai/DeepEP/blob/e3908bf5bd0cc6265bcb225d15cd8c996d4759ef/tests/test_internode.py#L242-L245
Running continuously for multiple rounds.
```python
for i in range(10000):
test_main(args, num_sms, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
if local_rank == 0:
print('', flush=True)
```

Moving torch.manual_seed(rank) inside the loop to ensure identical inputs for each iteration does not fix the issue.
I changed 'assert' to 'print' to prevent interrupting the testing program, but it still hang then timeout. calc_diff and check_data will get inf or nan.

Last Known Good Commit: 7705f53 (The stress test passes reliably on this commit). This is the last commit before the internode TMA feature was merged.
First Known Bad Commit: a2fa3b7 (The stress test fails on this commit). This is the first commit that introduces the internode TMA functionality.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.