JanusGraph / JanusGraph/janusgraph

Kryo registrator options for Spark

Open
#1,194 2 comments 0 reactions 0 assignees View on GitHub
area/docs area/spark help wanted kind/enhancement
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

This came up during a [documentation review](https://github.com/JanusGraph/janusgraph/pull/1177#issuecomment-408504616):

https://github.com/JanusGraph/janusgraph/blob/0.2/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/test/resources/cassandra-read.properties#L30
Should we use:

> spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator

Or:

> spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator

`JanusGraphKryoRegistrator` only registers a serializer for `Geoshape`, while the TinkerPop `GryoRegistrator` handles many different classes. `JanusGraphKryoRegistrator` does not extend `GryoRegistrator`.

We need some clarity on how to configure the Kryo registrator to cover all required TinkerPop and JanusGraph classes. Also will need updates on the documentation and the properties files in the distribution.

Contributor guide

Open the contributing guide

Research direction

Start with janusgraph-hadoop-parent/janusgraph-hadoop-core/src/test/resources/cassandra-read.properties at the referenced Spark setting, then review the linked documentation review. Determine which registrator covers the required TinkerPop and JanusGraph classes, and identify the distribution properties files and documentation that need matching updates. Done means the configuration choice is resolved and those references are consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.