JanusGraph / JanusGraph/janusgraph

Feature request : spark graphx integration?

Open
#1,419 1 comment 1 reaction 0 assignees View on GitHub
area/spark
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

There are docs about this : https://docs.janusgraph.org/latest/hadoop-tp3.html , but this way of integration can not utilize existing graphx code. If there are ways to export to RDD(VertexRDD[VD] and EdgeRDD[ED] extend and are optimized versions of RDD[(VertexId, VD)] and RDD[Edge[ED]]) then lots of existing graphx optimizations can be reused

In addition,with RDD we can impose a more fine-grained control like explicitly control the caching behavior of RDDs,as well as lots of existing spark tricks,which can be used to optimize speed

Contributor guide

Open the contributing guide

Research direction

Start with the Hadoop TP3 integration documentation linked in the issue and examine how JanusGraph currently integrates with Spark. The requested outcome is an export to GraphX-compatible VertexRDD and EdgeRDD types, with enough control over RDD caching and Spark optimizations; no source files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.