JanusGraph / JanusGraph/janusgraph

Make reindexing compatible to Spark RDDs

Open
#3,156 0 comments 0 reactions 0 assignees View on GitHub
area/hadoop area/spark area/tinkerpop
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Currently, one can use

```java
new InputFormatRDD().readGraphRDD(new MapConfiguration(configurationMap), sparkContext)
```

to load the entire graph to Spark as Spark RDDs. FYI I also have a [PR](https://github.com/apache/tinkerpop/pull/1758) for [TINKERPOP-2772](https://issues.apache.org/jira/projects/TINKERPOP/issues/TINKERPOP-2772) to make Spark load more accessible to users.

One can do any custom analysis on these RDDs, but it becomes difficult when one wants to run reindexing in Spark. The curse is, these RDDs are in TinkerPop's StarGraph.StarVertex type and thus not acceptable by `IndexSerializer::reindexElement` method. I am not sure if this is doable but it would be awesome if we could refactor reindexing APIs to make them accept any kind of `Vertex` rather than just `JanusGraphVertex`.

FWIW, the workaround I use in my project is to only use the vertex id in the RDD and query JanusGraph to retrieve the vertex, which is not that efficient and is definitely a waste of resources.

Contributor guide

Open the contributing guide

Research direction

Start with InputFormatRDD.readGraphRDD and trace how Spark RDDs produce StarGraph.StarVertex values. Then inspect IndexSerializer::reindexElement and the JanusGraphVertex requirement to determine whether a Vertex-based API is feasible. Done means Spark RDD vertices can be reindexed without the vertex-id lookup workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.