JanusGraph / JanusGraph/janusgraph
Add support for Geoshape predicates in OLAP traversals
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Geoshape predicates are currently not supported in OLAP mode.
```
gremlin> a.V().has('location', Geoshape.point(1.0, 2.0))
Caused by: java.io.NotSerializableException: org.janusgraph.core.attribute.Geoshape
Serialization stack:
- object not serializable (class: org.janusgraph.core.attribute.Geoshape, value: POINT (2 1))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.P, name: originalValue, type: class java.lang.Object)
- object (class org.apache.tinkerpop.gremlin.process.traversal.P, eq(POINT (2 1)))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.step.util.HasContainer, name: predicate, type: class org.apache.tinkerpop.gremlin.process.traversal.P)
- object (class org.apache.tinkerpop.gremlin.process.traversal.step.util.HasContainer, location.eq(POINT (2 1)))
- writeObject data (class: java.util.ArrayList)
- object (class java.util.ArrayList, [location.eq(POINT (2 1))])
- field (class: org.apache.tinkerpop.gremlin.process.traversal.step.filter.HasStep, name: hasContainers, type: interface java.util.List)
- object (class org.apache.tinkerpop.gremlin.process.traversal.step.filter.HasStep, HasStep([location.eq(POINT (2 1))]))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal, name: finalEndStep, type: interface org.apache.tinkerpop.gremlin.process.traversal.Step)
- object (class org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal, [GraphStep(vertex,[]), HasStep([location.eq(POINT (2 1))])])
- field (class: org.apache.tinkerpop.gremlin.process.traversal.util.PureTraversal, name: pureTraversal, type: interface org.apache.tinkerpop.gremlin.process.traversal.Traversal$Admin)
- object (class org.apache.tinkerpop.gremlin.process.traversal.util.PureTraversal, [GraphStep(vertex,[]), HasStep([location.eq(POINT (2 1))])])
- writeObject data (class: java.util.HashMap)
- object (class java.util.HashMap, {gremlin.vertexProgram=org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram, gremlin.traversalVertexProgram.traversal=[GraphStep(vertex,[]), Ha
sStep([location.eq(POINT (2 1))])]})
- field (class: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration, name: properties, type: interface java.util.Map)
- object (class org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration, org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration@cd7a7cd)
- element of array (index: 1)
- array (class [Ljava.lang.Object;, size 3)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
- object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor, functionalInterfaceMethod=org/apache/spark/api/j
ava/function/PairFlatMapFunction.call:(Ljava/lang/Object;)Ljava/lang/Iterable;, implementation=invokeStatic org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.lambda$executeVertexProgramIteration$3
5c6b113$1:(Lorg/apache/commons/configuration/Configuration;Lorg/apache/commons/configuration/Configuration;Lorg/apache/tinkerpop/gremlin/spark/process/computer/SparkMemory;Ljava/util/Iterator;)Ljava/lang/Iterable
;, instantiatedMethodType=(Ljava/util/Iterator;)Ljava/lang/Iterable;, numCaptured=3])
- writeReplace data (class: java.lang.invoke.SerializedLambda)
- object (class org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor$$Lambda$91/739815532, org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor$$Lambda$91/739815532@1da9efbd)
- field (class: org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$9$1, name: f$11, type: interface org.apache.spark.api.java.function.PairFlatMapFunction)
- object (class org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$9$1, )
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 17 more
```
@sjudeng did some investigation and deduced that this is probably a closure serialization issue:
```
@twilmes It looks like serialization of Geoshape predicates in OLAP traversals has never been supported? I checked JanusGraph 0.1 and Titan 1.0 and both failed with the same error. I think the underlying issue is that when the traversal has a Geoshape predicate it forces the Geoshape into the task's closure, and Spark does not support Kryo for closure serialization.
https://issues.apache.org/jira/browse/SPARK-12414
https://stackoverflow.com/a/40261550
Can we create a separate issue for adding support for Geoshape predicates in OLAP traversals? The current updates here fix #373 (e.g. OLAP traversals with Geoshape properties), which I view as a bug fix since this worked in 0.1
```
Contributor guide
Research direction
Start by reproducing the Geoshape predicate failure in an OLAP traversal and trace how the traversal and predicate reach Spark task serialization. Review the OLAP traversal and Spark execution paths, along with the investigation linked in the issue. Done means Geoshape predicates execute in OLAP mode without NotSerializableException, with coverage for the reported traversal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100