alibaba / alibaba/euler

RAM overflow while initializing graph

Open
#150 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

I have 294 Gb binary .dat file and one machine having 126 Gb of RAM.
When I execute a command
```
python -m tf_euler \
--data_dir euler \
--max_id 177416798 \
--model graphsage --mode train
```
memory overflow occurs. What can I do to process this file in chunks on my machine? Can HDFS help me to do it locally?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the shown `python -m tf_euler` training command with the 294 GB `.dat` input and 126 GB of RAM. Investigate where graph initialization exhausts memory and whether chunked or HDFS-backed processing is supported; done means the dataset can be processed without memory overflow on the reported machine.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, python
Domain
data-engineering, distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.