apache / apache/incubator-graphar

[Feat][Spark] Memory tuning for GraphAr Spark

Open
#324 0 comments 0 reactions 1 assignee Claimed by @acezen View on GitHub
enhancement
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

**Is your feature request related to a problem? Please describe.**
Currently, GraphAr spark implementation can not handle very large dataset efficiency and got OOM error. We need to tuning the memory usage of GraphAr spark and make it scalable.

**Describe the solution you'd like**
Tuning the memory usage with [spark guide](https://spark.apache.org/docs/latest/tuning.html)
- [x] serialize RDD object with [persist](https://spark.apache.org/docs/latest/tuning.html#serialized-rdd-storage) #326
- [ ] tuning GC (especially when vertex chunk size is very small that would generate a large amount of rdd object)
- [ ] tuning data structure

**Additional context**
related to #119

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.