JanusGraph / JanusGraph/janusgraph

Suggestion: Extend the Hadoop-Gremlin documentation

Open
#253 1 comment 0 reactions 0 assignees View on GitHub
area/docs
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

The [current documentation of JanusGraph about Hadoop-Gremlin](http://docs.janusgraph.org/0.1.0/hadoop-tp3.html) is very short and doesn't really explain how to execute a vertex program on JanusGraph or how to execute traversals in OLAP mode. This led to many questions on the mailing list and on StackOverflow. It also doesn't help that there exist a variety of configuration options specific to Hadoop-Gremlin that aren't explained anywhere [1] and that the documentation is split between the JanusGraph and the TinkerPop docs, where the TinkerPop docs of course lack JanusGraph specific config options.

I would suggest that at least one minimal but complete example for the different use cases of Hadoop-Gremlin are added to the JanusGraph documentation:

- Importing data into JanusGraph
- Exporting data with JanusGraph
- Executing OLAP traversals (a `g.V().out().out().count()` should be enough)
- Executing a `VertexProgram` like `PageRank`
- As soon as it's supported: Modifying the graph for example for repair jobs that are [already mentioned in the docs](http://docs.janusgraph.org/0.1.0/eventual-consistency.html#ghost-vertices)

Another aspect concerning Hadoop-Gremlin is that [HDFS is not required anymore with Spark in all cases since TinkerPop 3.1.1](https://github.com/apache/tinkerpop/blob/master/CHANGELOG.asciidoc#tinkerpop-311-release-date-february-8-2016). A short explanation of when the HDFS can be omitted might also be a good idea here.

Apart from that, I also noticed that the JanusGraph compatibility matrix lacks information about the supported Hadoop and Spark versions. Of course, this is specific to TinkerPop, but wouldn't it make sense to also include the information in this matrix? Otherwise every user has to search in the TinkerPop documentation for this information and as far as I know, it is only provided in TinkerPop's changelog which is not really the first place a typical JanusGraph user will look into.

In short: I think that getting JanusGraph to work with Spark (or Giraph) is one of the most complicated aspects of JanusGraph and that a good documentation avoids that users stumble upon the same problems again and again.

[1] For example `cassandra.input.predicate` which [led to some confusion in the Aurelius Google Group](https://groups.google.com/forum/#!topic/aureliusgraphs/RTidD6j1JG0).

Contributor guide

Open the contributing guide

Research direction

Start with the current Hadoop-Gremlin documentation linked in the issue, the TinkerPop 3.1.1 changelog, and JanusGraph's compatibility matrix. Document minimal examples for import, export, OLAP traversals, and PageRank, explain relevant configuration and when HDFS can be omitted, and add supported Hadoop and Spark versions to the matrix.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
databases, distributed-systems, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.