apache / apache/incubator-graphar
[Feat][Spark] Memory tuning for GraphAr Spark
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
**Is your feature request related to a problem? Please describe.**
Currently, GraphAr spark implementation can not handle very large dataset efficiency and got OOM error. We need to tuning the memory usage of GraphAr spark and make it scalable.
**Describe the solution you'd like**
Tuning the memory usage with [spark guide](https://spark.apache.org/docs/latest/tuning.html)
- [x] serialize RDD object with [persist](https://spark.apache.org/docs/latest/tuning.html#serialized-rdd-storage) #326
- [ ] tuning GC (especially when vertex chunk size is very small that would generate a large amount of rdd object)
- [ ] tuning data structure
**Additional context**
related to #119
Contributor guide
Assessment
This issue has not been assessed yet.