dmlc / dmlc/dgl

Better understanding on the usage of memory (allocation/deallocation) of a python process - distributed graph partitioning pipeline

Open
#4,784 1 comment 0 reactions 0 assignees View on GitHub
stale-issue
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🔨Work Item

This issue will be used to track the discussion on memory related issues of a python process (in the context of distributed graph partitioning pipeline).

The goal of this effort is to effectively handle very large no. of nodes/edges per graph partition on each ec2 instance during normal pipeline's processing.

Currently we use "/proc/[pid]/status" to print memory usage of pipeline during the course of its execution. We take a snapshot of the following items:

VmPeak - Peak memory used, so far, during the course of the execution of the pipeline
VmSize - Total size of the address space used by the process
VmRSS - Size of the memory which is in the RAM at the moment
VmData - Size of the Data portion of the current process

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description

## Depending work items or issues

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.