dmlc / dmlc/dgl

MultiLayerFullNeighborSampler takes too much memory

Open
#3,476 7 comments 0 reactions 0 assignees View on GitHub
bug:unconfirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug
Related discussion: https://discuss.dgl.ai/t/why-does-multilayerfullneighborsampler-consume-a-large-amount-of-memory/2454/12

MultilayerFullNeighborSampler used too much memory than expected. If change `MultilayerFullNeighborSampler(num_layers)` to `MultiLayerNeighborSampler([-1 for _ in range(num_layers)])`. The memory consumption goes much lower.

Potentially this might due to the implementation using in_subgraph but not sample_neighbors

Confirmed on the master branch

## To reproduce
using examples/pytorch/graphsage/train_sampling.py

Change sampler to `sampler = dgl.dataloading.MultiLayerFullNeighborSampler(4)` and `sampler = dgl.dataloading.MultiLayerNeighborSampler([-1 for _ in range(num_layers)])`.

cc @BarclayII

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.