dmlc / dmlc/dgl

Dataloading requires 5x the memory of the indices tensor when a dictionary is passed

Open
#3,918 5 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug
When running UVA training on an edge-centric dataset, and doing sampling on a subset of indces, these lines require 5x the size of the indices tensor worth of GPU memory, causing OOMs with medium sized datasets:

https://github.com/dmlc/dgl/blob/master/python/dgl/dataloading/dataloader.py#L88-L95

## To Reproduce

Steps to reproduce the behavior:

1. Create a moderately large graph (in my case ~ 1B edges)
2. Create formats for the graph (required for UVA)
3. Create a sampler that samples most of those edges (with UVA on), with indices stored on GPU.
4. Run sampling

## Expected behavior
Ideally, if we need to create these tensors, they should be stored in host memory. The best case scenario is that we can somehow do this operation in place.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.