dmlc / dmlc/dgl

[RFC][DataLoader] Support `sample_blocks` API for non-DGLGraph objects

Open
#4,498 8 comments 0 reactions 2 assignees Claimed by @VibhuJawa View on GitHub
feature request
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature
We should add support to `sample_blocks` API for non-DGLGraph objects to use in the sampling pipeline.

This will follow how we currently support `sample_neighbors` for Non-DGL Graph objects.

Document: https://hackmd.io/6PJixJj7Ssy19I-N9CV1Ag#Extending-DGL-Graph-Sampling-Pipeline
Related Issue: https://github.com/dmlc/dgl/issues/3600

## Motivation

Currently, in sample blocks (see below), we iterate for each value of `fanout.`

Though this approach works when the data is present locally when we want to do this through an external service (say something like `GAAS`) this can become a bottleneck as we will need to transfer graphs back and forth from the service for each value of fanout.

This will mean we do a bunch of calls like below which can become really slow.
```
External Sampling Process -> Training Process -> External Sampling Process -> .....
```


https://github.com/dmlc/dgl/blob/d41d07d0f6cbed17993644b58057e280a9e8f011/python/dgl/dataloading/neighbor_sampler.py#L106-L120

## Pitch

We will have to modify the current [sample_blocks](https://github.com/dmlc/dgl/blob/2b76674002cb1df4de237024838513b6679dfbf2/python/dgl/dataloading/neighbor_sampler.py#L106-L120) to check if the underlying object has an attribute for `sample_blocks` and if yes call that directly and if not call the above iterative sample_neighbors.py

## Additional context

Related issue: https://github.com/dmlc/dgl/issues/4326

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.