[BUG]stuck in the communication when pp = 4
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
Hi there , my training task got stuck when I try to train at PP = 4, I use austin to identify the exactly code and found out that Rank 0 got Stuck at isent() and Rank 1 got stuck at iRecv() .
**To Reproduce**
using L40s to train a multimodal 7B on TP =1 and PP = 4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Stack trace/logs**
see attachment.
**Environment (please complete the following information):**
- Megatron-LM commit ID
- torch Version: 2.5.0a0+e000cf0ad9.nv24.10
- CUDA version
**Proposed fix**
replace below code ( megatron/core/P2P_communication.py #def _p2p_ops Line 159) :
```
even_send_odd_recv_group = group
if get_pipeline_model_parallel_world_size() == 2:
# Use the global process group for one of the two p2p communications
# to allow the overlap of the independent communications.
# Using the global process group is compatible because the pipeline-parallel
# communications set the source and destination by global rank.
even_recv_odd_send_group = torch.distributed.group.WORLD
else:
even_recv_odd_send_group = group
```
with
```
even_send_odd_recv_group = torch.distributed.group.WORLD
even_recv_odd_send_group = torch.distributed.group.WORLD
```
[pp4_error1.log](https://github.com/user-attachments/files/20452604/pp4_error1.log)
[pp4_error_austin.log](https://github.com/user-attachments/files/20452609/pp4_error_austin.log)
Contributor guide
Assessment
This issue has not been assessed yet.