dmlc / dmlc/dgl

[Graphbolt][Performance] Reduce the memory usage of `preprocess_ondisk_dataset`

Open
#7,086 2 comments 0 reactions 2 assignees Claimed by @Skeleton003 View on GitHub
stale-issue
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

## Motivation

Currently, [preprocess_ondisk_dataset](https://github.com/dmlc/dgl/blob/4ee0a8bddbd93963b5f078c475381f4ab521d2e1/python/dgl/graphbolt/impl/ondisk_dataset.py#L41) consumes much more memory than the topology of a graph itself during the preprocessing. When loading a graph with 2B nodes and 8B edges, it cannot be finished in a machine with 380 GB memory. After a rough profiling, I found that the peak memory usage is reached when converting a DGL graph to a fused sampling graph. https://github.com/dmlc/dgl/blob/4ee0a8bddbd93963b5f078c475381f4ab521d2e1/python/dgl/graphbolt/impl/ondisk_dataset.py#L212
There could be two factors contributing to the peak memory usage.

1. The input DGL graph is passed to the function, which consumes about 160 GB memory.
2. `from_dglgraph` creates a temporary homogeneous graph and also its CSC format.

## Alternatives

## Pitch

## Additional context

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.