facebookresearch / facebookresearch/fairscale

new nccl_base_collectives slow down the FSDP performance when GPU memory usage is high

Open
#801 3 comments 0 reactions 0 assignees View on GitHub
better_eng FSDP
Dominant language
Python
Stars
3.4k
Forks
293
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug
After this pull request: FSDP uses _allgather_base and _reduce_scatter_base,
https://github.com/facebookresearch/fairscale/pull/729/files, people observe some performance slow down when GPU usage is high.
The reason is that if GPU memory usage is increased to some point, cuda cache allocator will has some issue and caused cudaMalloc to be very slow.

## Command

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

## Expected behavior

## Environment

Please copy and paste the output from the
environment collection script from PyTorch
(or fill out the checklist below manually).

You can run the script with:
```
# For security purposes, please check the contents of collect_env.py before running it.
python -m torch.utils.collect_env
```

- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux):
- How you installed PyTorch (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- Any other relevant information:

## Additional context

Contributor guide

Open the contributing guide

Research direction

Review the files changed by pull request #729 and trace the _allgather_base and _reduce_scatter_base call paths. Run python -m torch.utils.collect_env to record the environment, then reproduce the slowdown at high GPU memory usage. Done means the cause is isolated and the fix is validated with a performance comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.