Support for Pytorch-Lightning multi-GPU
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Add multi-gpu support when models are trained using Pytorch-Lightning.
## Motivation
Pytorch-lightning is one of the most popular packages for training models in pytorch. It is simple, full-featured, and allows to handle distributed training easily. However, it currently crashes when training DGL models with a multi-gpu setting since the batches do not get converted appropriately. This is related to the objectives of DGL 0.7 to the roadmap of supporting pytorch-lightning #2888 .
## Additional context
I tried training a model on a single machine with multi-GPUs, so the training is not distributed, but rather the batches are split evenly and spread across GPUs for parallel computation.
There are 2 problems that I noticed:
- The `device` of the `DGLHeteroGraph` objects stays on the CPU and is not sent to the GPU automatically, as is the case with the labels. At each iteration, the `DataLoader` loads data from the RAM, and pytorch-lightning is supposed to send them to the GPU, but it does not happen for some reason.
- Manually pushing the device to the right GPU does not help either, since the batches are not split accordingly. For example, if I have a batch size of 128 on 2 GPUs, it should be split into 2 batches of size 64. Here, the batches are not split and each GPU receives the full batch of 128 elements.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.