JanusGraph / JanusGraph/janusgraph
Suggestion: Extend the Hadoop-Gremlin documentation
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
The [current documentation of JanusGraph about Hadoop-Gremlin](http://docs.janusgraph.org/0.1.0/hadoop-tp3.html) is very short and doesn't really explain how to execute a vertex program on JanusGraph or how to execute traversals in OLAP mode. This led to many questions on the mailing list and on StackOverflow. It also doesn't help that there exist a variety of configuration options specific to Hadoop-Gremlin that aren't explained anywhere [1] and that the documentation is split between the JanusGraph and the TinkerPop docs, where the TinkerPop docs of course lack JanusGraph specific config options.
I would suggest that at least one minimal but complete example for the different use cases of Hadoop-Gremlin are added to the JanusGraph documentation:
- Importing data into JanusGraph
- Exporting data with JanusGraph
- Executing OLAP traversals (a `g.V().out().out().count()` should be enough)
- Executing a `VertexProgram` like `PageRank`
- As soon as it's supported: Modifying the graph for example for repair jobs that are [already mentioned in the docs](http://docs.janusgraph.org/0.1.0/eventual-consistency.html#ghost-vertices)
Another aspect concerning Hadoop-Gremlin is that [HDFS is not required anymore with Spark in all cases since TinkerPop 3.1.1](https://github.com/apache/tinkerpop/blob/master/CHANGELOG.asciidoc#tinkerpop-311-release-date-february-8-2016). A short explanation of when the HDFS can be omitted might also be a good idea here.
Apart from that, I also noticed that the JanusGraph compatibility matrix lacks information about the supported Hadoop and Spark versions. Of course, this is specific to TinkerPop, but wouldn't it make sense to also include the information in this matrix? Otherwise every user has to search in the TinkerPop documentation for this information and as far as I know, it is only provided in TinkerPop's changelog which is not really the first place a typical JanusGraph user will look into.
In short: I think that getting JanusGraph to work with Spark (or Giraph) is one of the most complicated aspects of JanusGraph and that a good documentation avoids that users stumble upon the same problems again and again.
[1] For example `cassandra.input.predicate` which [led to some confusion in the Aurelius Google Group](https://groups.google.com/forum/#!topic/aureliusgraphs/RTidD6j1JG0).
Contributor guide
Research direction
Start with the current Hadoop-Gremlin documentation linked in the issue, the TinkerPop 3.1.1 changelog, and JanusGraph's compatibility matrix. Document minimal examples for import, export, OLAP traversals, and PageRank, explain relevant configuration and when HDFS can be omitted, and add supported Hadoop and Spark versions to the matrix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- databases, distributed-systems, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100