AnswerDotAI / AnswerDotAI/minai
Multi-GPU Training with DataParallel Results in RuntimeError
- Dominant language
- Jupyter Notebook
- Stars
- 43
- Forks
- 9
- Avg merge
- 11m
- Merged PRs (30d)
- 1
Description
**Description**
I am trying to perform multi-GPU training using the DataParallel wrapper from PyTorch. When I try to run the fit method, I encounter a RuntimeError saying that the parameters and buffers must be on the same device.
Here's a snippet of the code that I am using:
```
# Initialize learner and model
learn = Learner(...)
learn.model = ...
# Attempt to use DataParallel
model = nn.DataParallel(learn.model, device_ids=[1, 2, 3])
learn.model = model
# Update DataLoader device
learn.dls.device = torch.device("cuda:1")
# Clear cache
torch.cuda.empty_cache()
# Start training
learn.fit(1)
```
Error Message
The error message I receive is:
_RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:3_
**Environment**
PyTorch version: (e.g., 1.9.0)
Library version: (e.g., 0.2.0)
CUDA/cuDNN version: (e.g., CUDA 11.8, cuDNN 8.2.1)
GPU models and configuration: (e.g., 4x Tesla T4)
Operating System: (e.g., Ubuntu 18.04)
**Additional Context**
I've tried to set both the model and the DataLoader to the same device but without success. It seems like the model parameters and DataLoader are ending up on different devices during the training, causing the error.
Would appreciate any guidance on how to resolve this issue or if it's something that needs to be addressed in the library.
Contributor guide
Assessment
This issue has not been assessed yet.