ContinualAI / ContinualAI/avalanche

Feature request: Add an option to preload the dataset into GPU

Open
#839 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

It would be great to add an option to preload a dataset directly to GPU before starting to train.
I was training small FCNs and CNNs on small datasets like mnist and cifar10 and I had a 100% CPU load because it was busy in continuously transfering minibatches to the GPU.

I've coded a small code snippet that monkey patches torch.data.utils.DataLoader to move the dataset to GPU which gets executed in the following line in avalanche dataloaders, and I got 20x speedup and a higher GPU utilization.
https://github.com/ContinualAI/avalanche/blob/a299bd43155f191d8f2be117e01f1366cda552c2/avalanche/benchmarks/utils/data_loader.py#L127

```python3
import torch

data_device = "cuda:0"

# Monkey Patching torch DataLoader
# Do this before importing avalanche
def patch_dataloader_init(oldinit):
def newinit(self, dataset, *args, pin_memory=None, **kwargs):
# Define a new dataset after loading all the initial dataset into device
dsdata = [
tuple((
tensor.to(data_device) if isinstance(tensor, torch.Tensor) else tensor
for tensor in tensortup
))
for tensortup in dataset
]
class NewDataset(torch.utils.data.Dataset):
def __len__(self):
return len(dataset)

def __getitem__(self, idx):
return dsdata[idx]
newdataset = NewDataset()
# NOTE: GPU memory cannot be pinned, thus we set pin_memory to False
oldinit(self, newdataset, *args, pin_memory=False, **kwargs)
return newinit

torch.utils.data.dataloader.DataLoader.__init__ = patch_dataloader_init(torch.utils.data.dataloader.DataLoader.__init__)
```

Note that the above snippet can still be optimized for tensor datasets, since it currently copies each example of the dataset into GPU, while probably storing the whole dataset as a single tensor in GPU and reimplementing the common subsampling operations using torch operations would give faster speedups.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.