facebookresearch / facebookresearch/fairscale
new nccl_base_collectives slow down the FSDP performance when GPU memory usage is high
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 293
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
After this pull request: FSDP uses _allgather_base and _reduce_scatter_base,
https://github.com/facebookresearch/fairscale/pull/729/files, people observe some performance slow down when GPU usage is high.
The reason is that if GPU memory usage is increased to some point, cuda cache allocator will has some issue and caused cudaMalloc to be very slow.
## Command
## To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
## Expected behavior
## Environment
Please copy and paste the output from the
environment collection script from PyTorch
(or fill out the checklist below manually).
You can run the script with:
```
# For security purposes, please check the contents of collect_env.py before running it.
python -m torch.utils.collect_env
```
- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux):
- How you installed PyTorch (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- Any other relevant information:
## Additional context
Contributor guide
Research direction
Review the files changed by pull request #729 and trace the _allgather_base and _reduce_scatter_base call paths. Run python -m torch.utils.collect_env to record the environment, then reproduce the slowdown at high GPU memory usage. Done means the cause is isolated and the fix is validated with a performance comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100