apache / apache/incubator-graphar
feat(spark): Nebula spark connector does not support spark 3.4+
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
### Describe the enhancement requested
We should decide what to do with it. At the moment it is a part of Spark CI.
```
Exception in thread "main" java.lang.RuntimeException: Your current spark version 3.4.3 is not supported by the current NebulaGraph Spark Connector.
please visit https://github.com/vesoft-inc/nebula-spark-connector#version-match to know which Connector you need.
at com.vesoft.nebula.connector.utils.SparkValidate$.validate(SparkValidate.scala:15)
at com.vesoft.nebula.connector.connector.package$NebulaDataFrameReader.nebula(package.scala:34)
at org.apache.graphar.example.Nebula2GraphAr$.readVertexDF(Nebula2GraphAr.scala:135)
at org.apache.graphar.example.Nebula2GraphAr$.readAndPutDataIntoWriter(Nebula2GraphAr.scala:83)
at org.apache.graphar.example.Nebula2GraphAr$.main(Nebula2GraphAr.scala:53)
at org.apache.graphar.example.Nebula2GraphAr.main(Nebula2GraphAr.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Component(s)
Spark
Contributor guide
Research direction
Inspect the Spark CI setup and the failing Nebula2GraphAr.scala call path, starting with readVertexDF, readAndPutDataIntoWriter, and the SparkValidate.scala:15 failure shown here. Decide how Spark 3.4.3 support should be handled, then rerun the Spark CI example and confirm it no longer fails because the connector rejects that version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- ci-cd, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100