dmlc / dmlc/dgl

Slicing large messages when exchanging node/edge features into multiple small messages

Open
#5,505 1 comment 0 reactions 1 assignee Claimed by @kylasa View on GitHub
Work Item
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

### ## 🔨Work Item

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description

In the current implementation of Dist. Graph Partitioning Pipeline when exchanging/shuffling node/edge feature(s) it uses only one single message for node/edge-type.

Dist. Graph Partitioning Pipeline uses 'gloo' messaging layer for exchanging messages and 'gloo' does not slice/split large messages into smaller messages to void OOM issues. This is an application level functionality which needs to be addressed by the graph partitioning pipeline itself. Per testing results, gloo fails with OOM issues on a 4 node cluster when every node is sending a message of size 10 GB to every other node in the 4-node cluster.

To address future OOM issues when shuffling large node/edge features, we need some mechanism to split large messages into multiple small messages. And the outgoing message size (on a per node basis) can be controlled by a user specified command line argument.

## Depending work items or issues

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.