JanusGraph / JanusGraph/janusgraph
Make reindexing compatible to Spark RDDs
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Currently, one can use
```java
new InputFormatRDD().readGraphRDD(new MapConfiguration(configurationMap), sparkContext)
```
to load the entire graph to Spark as Spark RDDs. FYI I also have a [PR](https://github.com/apache/tinkerpop/pull/1758) for [TINKERPOP-2772](https://issues.apache.org/jira/projects/TINKERPOP/issues/TINKERPOP-2772) to make Spark load more accessible to users.
One can do any custom analysis on these RDDs, but it becomes difficult when one wants to run reindexing in Spark. The curse is, these RDDs are in TinkerPop's StarGraph.StarVertex type and thus not acceptable by `IndexSerializer::reindexElement` method. I am not sure if this is doable but it would be awesome if we could refactor reindexing APIs to make them accept any kind of `Vertex` rather than just `JanusGraphVertex`.
FWIW, the workaround I use in my project is to only use the vertex id in the RDD and query JanusGraph to retrieve the vertex, which is not that efficient and is definitely a waste of resources.
Contributor guide
Research direction
Start with InputFormatRDD.readGraphRDD and trace how Spark RDDs produce StarGraph.StarVertex values. Then inspect IndexSerializer::reindexElement and the JanusGraphVertex requirement to determine whether a Vertex-based API is feasible. Done means Spark RDD vertices can be reindexed without the vertex-id lookup workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100