huggingface / huggingface/picotron
Change process group order to optimize inter-node communication
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 203
- PR merge metrics
- No merged PRs in 30d
Description
To decrease inter-node communication volume, we intend to configure the ranks within the Pipeline Parallel process group to facilitate inter-node communication. Maybe self.grid of ProcessGroupManager should be:
```python
self.grid = torch.arange(self.world_size).view(pp_size, dp_size, cp_size, tp_size) # PP * DP * CP * TP grid
```
instead of
```python
# https://github.com/huggingface/picotron/blob/df3ae8a5f0cce213816b6b287b7febc75ab98a53/picotron/process_group_manager.py#L13
self.grid = torch.arange(self.world_size).view(dp_size, pp_size, cp_size, tp_size) # DP * PP * CP * TP grid
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.