JanusGraph / JanusGraph/janusgraph
Feature request : spark graphx integration?
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
There are docs about this : https://docs.janusgraph.org/latest/hadoop-tp3.html , but this way of integration can not utilize existing graphx code. If there are ways to export to RDD(VertexRDD[VD] and EdgeRDD[ED] extend and are optimized versions of RDD[(VertexId, VD)] and RDD[Edge[ED]]) then lots of existing graphx optimizations can be reused
In addition,with RDD we can impose a more fine-grained control like explicitly control the caching behavior of RDDs,as well as lots of existing spark tricks,which can be used to optimize speed
Contributor guide
Research direction
Start with the Hadoop TP3 integration documentation linked in the issue and examine how JanusGraph currently integrates with Spark. The requested outcome is an export to GraphX-compatible VertexRDD and EdgeRDD types, with enough control over RDD caching and Spark optimizations; no source files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100