huggingface / huggingface/picotron

Change process group order to optimize inter-node communication

Open
#17 5 comments 0 reactions 1 assignee Claimed by @3outeille View on GitHub
Dominant language
Python
Stars
2.3k
Forks
203
PR merge metrics
No merged PRs in 30d

Description

To decrease inter-node communication volume, we intend to configure the ranks within the Pipeline Parallel process group to facilitate inter-node communication. Maybe self.grid of ProcessGroupManager should be:
```python
self.grid = torch.arange(self.world_size).view(pp_size, dp_size, cp_size, tp_size) # PP * DP * CP * TP grid
```
instead of
```python
# https://github.com/huggingface/picotron/blob/df3ae8a5f0cce213816b6b287b7febc75ab98a53/picotron/process_group_manager.py#L13
self.grid = torch.arange(self.world_size).view(dp_size, pp_size, cp_size, tp_size) # DP * PP * CP * TP grid
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.