apache / apache/incubator-graphar

feat(spark): Nebula spark connector does not support spark 3.4+

Open
#583 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

### Describe the enhancement requested

We should decide what to do with it. At the moment it is a part of Spark CI.

```
Exception in thread "main" java.lang.RuntimeException: Your current spark version 3.4.3 is not supported by the current NebulaGraph Spark Connector.
please visit https://github.com/vesoft-inc/nebula-spark-connector#version-match to know which Connector you need.

at com.vesoft.nebula.connector.utils.SparkValidate$.validate(SparkValidate.scala:15)
at com.vesoft.nebula.connector.connector.package$NebulaDataFrameReader.nebula(package.scala:34)
at org.apache.graphar.example.Nebula2GraphAr$.readVertexDF(Nebula2GraphAr.scala:135)
at org.apache.graphar.example.Nebula2GraphAr$.readAndPutDataIntoWriter(Nebula2GraphAr.scala:83)
at org.apache.graphar.example.Nebula2GraphAr$.main(Nebula2GraphAr.scala:53)
at org.apache.graphar.example.Nebula2GraphAr.main(Nebula2GraphAr.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

### Component(s)

Spark

Contributor guide

Open the contributing guide

Research direction

Inspect the Spark CI setup and the failing Nebula2GraphAr.scala call path, starting with readVertexDF, readAndPutDataIntoWriter, and the SparkValidate.scala:15 failure shown here. Decide how Spark 3.4.3 support should be handled, then rerun the Spark CI example and confirm it no longer fails because the connector rejects that version.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
ci-cd, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.