ContinualAI / ContinualAI/avalanche
Feature request: Add an option to preload the dataset into GPU
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
It would be great to add an option to preload a dataset directly to GPU before starting to train.
I was training small FCNs and CNNs on small datasets like mnist and cifar10 and I had a 100% CPU load because it was busy in continuously transfering minibatches to the GPU.
I've coded a small code snippet that monkey patches torch.data.utils.DataLoader to move the dataset to GPU which gets executed in the following line in avalanche dataloaders, and I got 20x speedup and a higher GPU utilization.
https://github.com/ContinualAI/avalanche/blob/a299bd43155f191d8f2be117e01f1366cda552c2/avalanche/benchmarks/utils/data_loader.py#L127
```python3
import torch
data_device = "cuda:0"
# Monkey Patching torch DataLoader
# Do this before importing avalanche
def patch_dataloader_init(oldinit):
def newinit(self, dataset, *args, pin_memory=None, **kwargs):
# Define a new dataset after loading all the initial dataset into device
dsdata = [
tuple((
tensor.to(data_device) if isinstance(tensor, torch.Tensor) else tensor
for tensor in tensortup
))
for tensortup in dataset
]
class NewDataset(torch.utils.data.Dataset):
def __len__(self):
return len(dataset)
def __getitem__(self, idx):
return dsdata[idx]
newdataset = NewDataset()
# NOTE: GPU memory cannot be pinned, thus we set pin_memory to False
oldinit(self, newdataset, *args, pin_memory=False, **kwargs)
return newinit
torch.utils.data.dataloader.DataLoader.__init__ = patch_dataloader_init(torch.utils.data.dataloader.DataLoader.__init__)
```
Note that the above snippet can still be optimized for tensor datasets, since it currently copies each example of the dataset into GPU, while probably storing the whole dataset as a single tensor in GPU and reimplementing the common subsampling operations using torch operations would give faster speedups.
Contributor guide
Assessment
This issue has not been assessed yet.