More lightweight create_block is needed.
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## ❓ Questions and Help
Now, there are many third-party graph sampling frameworks, like [torch-quiver](https://github.com/quiver-team/torch-quiver), which may be more flexible or have higher performance. DGL provides [create_block](https://docs.dgl.ai/generated/dgl.create_block.html?highlight=create_block#dgl.create_block) to help developers finish their adaptor. Unfortunately, compared with PyG, the adaptor for DGL is a little heavy, due to its complex packaging for graph.
For graphsage with `dataset = reddit`, `fan_out = [25,10]`, `batch_size=1024`, using `torch-quiver` sampling on the GPU, caching all data on GPU memory, the E2E training one epoch time cost are:
| DGL(create_block) | PyG | DGL(my_create_block) |
| --- | --- | --- |
| 3.13 sec | 2.88 sec | 3.03 sec |
DGL(create_block) is 8.7% slower than PyG.
By breaking down, we can find that in DGL adaptor `create_block` can account up for **40%+** of the time cost in sampling stage. It's too heavy.

To reduce the overhead, I write a simple `create_block` called `my_create_block`. Code is following
```python
from dgl.heterograph import DGLBlock
def my_create_block(arrays, num_src_nodes, num_dst_nodes):
torch.cuda.nvtx.range_push('1')
hgidx = dgl.heterograph_index.create_unitgraph_from_coo(
2, num_src_nodes, num_dst_nodes, arrays[0], arrays[1], ['coo', 'csr', 'csc'],
row_sorted=False, col_sorted=True)
torch.cuda.nvtx.range_pop()
torch.cuda.nvtx.range_push('2')
retg = DGLBlock(hgidx, (['_N'], ['_N']), ['_E'])
torch.cuda.nvtx.range_pop()
return retg
```
Although, `DGL(my_create_block)` can reach `3.03` sec per epoch. `my_create_block` still take **15.3%** time cost in sampling stage.

Is there any way to provide a more lightweight `create_block` API or very low-level (`C++` is ok) but high performance APIs for developers to write an efficient Adaptor?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.