JanusGraph / JanusGraph/janusgraph

Add support for Geoshape predicates in OLAP traversals

Open
#390 0 comments 0 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Geoshape predicates are currently not supported in OLAP mode.
```
gremlin> a.V().has('location', Geoshape.point(1.0, 2.0))
Caused by: java.io.NotSerializableException: org.janusgraph.core.attribute.Geoshape
Serialization stack:
- object not serializable (class: org.janusgraph.core.attribute.Geoshape, value: POINT (2 1))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.P, name: originalValue, type: class java.lang.Object)
- object (class org.apache.tinkerpop.gremlin.process.traversal.P, eq(POINT (2 1)))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.step.util.HasContainer, name: predicate, type: class org.apache.tinkerpop.gremlin.process.traversal.P)
- object (class org.apache.tinkerpop.gremlin.process.traversal.step.util.HasContainer, location.eq(POINT (2 1)))
- writeObject data (class: java.util.ArrayList)
- object (class java.util.ArrayList, [location.eq(POINT (2 1))])
- field (class: org.apache.tinkerpop.gremlin.process.traversal.step.filter.HasStep, name: hasContainers, type: interface java.util.List)
- object (class org.apache.tinkerpop.gremlin.process.traversal.step.filter.HasStep, HasStep([location.eq(POINT (2 1))]))
- field (class: org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal, name: finalEndStep, type: interface org.apache.tinkerpop.gremlin.process.traversal.Step)
- object (class org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal, [GraphStep(vertex,[]), HasStep([location.eq(POINT (2 1))])])
- field (class: org.apache.tinkerpop.gremlin.process.traversal.util.PureTraversal, name: pureTraversal, type: interface org.apache.tinkerpop.gremlin.process.traversal.Traversal$Admin)
- object (class org.apache.tinkerpop.gremlin.process.traversal.util.PureTraversal, [GraphStep(vertex,[]), HasStep([location.eq(POINT (2 1))])])
- writeObject data (class: java.util.HashMap)
- object (class java.util.HashMap, {gremlin.vertexProgram=org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram, gremlin.traversalVertexProgram.traversal=[GraphStep(vertex,[]), Ha
sStep([location.eq(POINT (2 1))])]})
- field (class: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration, name: properties, type: interface java.util.Map)
- object (class org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration, org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration@cd7a7cd)
- element of array (index: 1)
- array (class [Ljava.lang.Object;, size 3)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
- object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor, functionalInterfaceMethod=org/apache/spark/api/j
ava/function/PairFlatMapFunction.call:(Ljava/lang/Object;)Ljava/lang/Iterable;, implementation=invokeStatic org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.lambda$executeVertexProgramIteration$3
5c6b113$1:(Lorg/apache/commons/configuration/Configuration;Lorg/apache/commons/configuration/Configuration;Lorg/apache/tinkerpop/gremlin/spark/process/computer/SparkMemory;Ljava/util/Iterator;)Ljava/lang/Iterable
;, instantiatedMethodType=(Ljava/util/Iterator;)Ljava/lang/Iterable;, numCaptured=3])
- writeReplace data (class: java.lang.invoke.SerializedLambda)
- object (class org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor$$Lambda$91/739815532, org.apache.tinkerpop.gremlin.spark.process.computer.SparkExecutor$$Lambda$91/739815532@1da9efbd)
- field (class: org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$9$1, name: f$11, type: interface org.apache.spark.api.java.function.PairFlatMapFunction)
- object (class org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$9$1, )
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 17 more
```

@sjudeng did some investigation and deduced that this is probably a closure serialization issue:

```
@twilmes It looks like serialization of Geoshape predicates in OLAP traversals has never been supported? I checked JanusGraph 0.1 and Titan 1.0 and both failed with the same error. I think the underlying issue is that when the traversal has a Geoshape predicate it forces the Geoshape into the task's closure, and Spark does not support Kryo for closure serialization.

https://issues.apache.org/jira/browse/SPARK-12414
https://stackoverflow.com/a/40261550

Can we create a separate issue for adding support for Geoshape predicates in OLAP traversals? The current updates here fix #373 (e.g. OLAP traversals with Geoshape properties), which I view as a bug fix since this worked in 0.1
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Geoshape predicate failure in an OLAP traversal and trace how the traversal and predicate reach Spark task serialization. Review the OLAP traversal and Spark execution paths, along with the investigation linked in the issue. Done means Geoshape predicates execute in OLAP mode without NotSerializableException, with coverage for the reported traversal.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.