AnswerDotAI / AnswerDotAI/minai

Multi-GPU Training with DataParallel Results in RuntimeError

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
43
Forks
9
Avg merge
11m
Merged PRs (30d)
1

Description

**Description**
I am trying to perform multi-GPU training using the DataParallel wrapper from PyTorch. When I try to run the fit method, I encounter a RuntimeError saying that the parameters and buffers must be on the same device.

Here's a snippet of the code that I am using:

```
# Initialize learner and model
learn = Learner(...)
learn.model = ...

# Attempt to use DataParallel
model = nn.DataParallel(learn.model, device_ids=[1, 2, 3])
learn.model = model

# Update DataLoader device
learn.dls.device = torch.device("cuda:1")

# Clear cache
torch.cuda.empty_cache()

# Start training
learn.fit(1)
```
Error Message
The error message I receive is:

_RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:3_

**Environment**
PyTorch version: (e.g., 1.9.0)
Library version: (e.g., 0.2.0)
CUDA/cuDNN version: (e.g., CUDA 11.8, cuDNN 8.2.1)
GPU models and configuration: (e.g., 4x Tesla T4)
Operating System: (e.g., Ubuntu 18.04)

**Additional Context**
I've tried to set both the model and the DataLoader to the same device but without success. It seems like the model parameters and DataLoader are ending up on different devices during the training, causing the error.

Would appreciate any guidance on how to resolve this issue or if it's something that needs to be addressed in the library.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.