[Graphbolt][Performance] Reduce the memory usage of `preprocess_ondisk_dataset`
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
## Motivation
Currently, [preprocess_ondisk_dataset](https://github.com/dmlc/dgl/blob/4ee0a8bddbd93963b5f078c475381f4ab521d2e1/python/dgl/graphbolt/impl/ondisk_dataset.py#L41) consumes much more memory than the topology of a graph itself during the preprocessing. When loading a graph with 2B nodes and 8B edges, it cannot be finished in a machine with 380 GB memory. After a rough profiling, I found that the peak memory usage is reached when converting a DGL graph to a fused sampling graph. https://github.com/dmlc/dgl/blob/4ee0a8bddbd93963b5f078c475381f4ab521d2e1/python/dgl/graphbolt/impl/ondisk_dataset.py#L212
There could be two factors contributing to the peak memory usage.
1. The input DGL graph is passed to the function, which consumes about 160 GB memory.
2. `from_dglgraph` creates a temporary homogeneous graph and also its CSC format.
## Alternatives
## Pitch
## Additional context
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.