RAM overflow while initializing graph
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
I have 294 Gb binary .dat file and one machine having 126 Gb of RAM.
When I execute a command
```
python -m tf_euler \
--data_dir euler \
--max_id 177416798 \
--model graphsage --mode train
```
memory overflow occurs. What can I do to process this file in chunks on my machine? Can HDFS help me to do it locally?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the shown `python -m tf_euler` training command with the 294 GB `.dat` input and 126 GB of RAM. Investigate where graph initialization exhausts memory and whether chunked or HDFS-backed processing is supported; done means the dataset can be processed without memory overflow on the reported machine.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, python
- Domain
- data-engineering, distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100