Lightning-AI / Lightning-AI/pytorch-lightning
training with 500 gpus is almost 2x slower then training with 300 gpus in my case
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
In my case, training with 500 GPUs is almost 2x slower than training with 300 GPUs. Does anyone know what might be the causes and how to solve them? For more information, we use NVIDIA Nsight, and it has been observed that there are numerous broadcasts. Thank you in advance to anyone who can provide insights or information on these topics!
### What version are you seeing the problem on?
pytorch-lightning==1.4.2
### How to reproduce the bug
base on https://github.com/CompVis/stable-diffusion
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
torch==1.12.1+cuda113.cudnn8.3.2
pytorch-lightning==1.4.2
### More info
_No response_
cc @justusschock
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the stable-diffusion reproduction linked in the issue and compare the 300-GPU and 500-GPU runs using NVIDIA Nsight. Review the reported broadcasts and the supplied PyTorch and PyTorch Lightning versions; the issue is done when the scaling slowdown is explained and a verified remedy or actionable diagnosis is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100