NVIDIA / NVIDIA/Megatron-LM

[BUG]stuck in the communication when pp = 4

Open
#1,597 3 comments 1 reaction 0 assignees View on GitHub
bug community-request waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**
Hi there , my training task got stuck when I try to train at PP = 4, I use austin to identify the exactly code and found out that Rank 0 got Stuck at isent() and Rank 1 got stuck at iRecv() .

**To Reproduce**
using L40s to train a multimodal 7B on TP =1 and PP = 4.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Stack trace/logs**
see attachment.

**Environment (please complete the following information):**
- Megatron-LM commit ID
- torch Version: 2.5.0a0+e000cf0ad9.nv24.10
- CUDA version

**Proposed fix**

replace below code ( megatron/core/P2P_communication.py #def _p2p_ops Line 159) :
```
even_send_odd_recv_group = group
if get_pipeline_model_parallel_world_size() == 2:
# Use the global process group for one of the two p2p communications
# to allow the overlap of the independent communications.
# Using the global process group is compatible because the pipeline-parallel
# communications set the source and destination by global rank.
even_recv_odd_send_group = torch.distributed.group.WORLD
else:
even_recv_odd_send_group = group
```
with
```
even_send_odd_recv_group = torch.distributed.group.WORLD
even_recv_odd_send_group = torch.distributed.group.WORLD
```

[pp4_error1.log](https://github.com/user-attachments/files/20452604/pp4_error1.log)

[pp4_error_austin.log](https://github.com/user-attachments/files/20452609/pp4_error_austin.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.