Lightning-AI / Lightning-AI/pytorch-lightning
forced CPU-GPU optimization at the end of every training step resulting in underutilization of GPU
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
I just realized that the feature implemented in this PR: #20825 causes CPU-GPU synchronization at the end of each training step, which negatively affects the GPU utilization. In particular the `torch.distributed.broadcast(sigterm_tensor, src=0)` in method: `_broadcast_sigterm_tensor` needs CPU to wait for GPU. This is not the case before adding this feature.
You can see this in Pytorch profiler below:
_Originally posted by @mojtababahrami in https://github.com/Lightning-AI/pytorch-lightning/issues/20825#issuecomment-3649816413_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #20825 and the `_broadcast_sigterm_tensor` method, especially the `torch.distributed.broadcast(sigterm_tensor, src=0)` call. Use the provided PyTorch profiler example to compare synchronization at the end of training steps before and after the feature. Done means the signal handling remains correct without forcing CPU-GPU synchronization on every step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100