dmlc / dmlc/dgl

Support for Pytorch-Lightning multi-GPU

Open
#3,053 3 comments 1 reaction 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature
Add multi-gpu support when models are trained using Pytorch-Lightning.

## Motivation
Pytorch-lightning is one of the most popular packages for training models in pytorch. It is simple, full-featured, and allows to handle distributed training easily. However, it currently crashes when training DGL models with a multi-gpu setting since the batches do not get converted appropriately. This is related to the objectives of DGL 0.7 to the roadmap of supporting pytorch-lightning #2888 .

## Additional context

I tried training a model on a single machine with multi-GPUs, so the training is not distributed, but rather the batches are split evenly and spread across GPUs for parallel computation.

There are 2 problems that I noticed:

- The `device` of the `DGLHeteroGraph` objects stays on the CPU and is not sent to the GPU automatically, as is the case with the labels. At each iteration, the `DataLoader` loads data from the RAM, and pytorch-lightning is supposed to send them to the GPU, but it does not happen for some reason.
- Manually pushing the device to the right GPU does not help either, since the batches are not split accordingly. For example, if I have a batch size of 128 on 2 GPUs, it should be split into 2 batches of size 64. Here, the batches are not split and each GPU receives the full batch of 128 elements.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.