apache / apache/seatunnel

[Bug] [Spark Engine] SeaTunnel with Spark engine doesn't support Spark 3.5

Open
#9,855 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

When trying to use SeaTunnel with Spark 3.5 as the engine, the job fails with a `NoSuchMethodError` exception. It appears that SeaTunnel is not compatible with Spark 3.5.

### SeaTunnel Version

2.3.11

### SeaTunnel Config

```conf
env {
# common parameter
parallelism = 1

# spark special parameter
spark.app.name = "spark"
spark.sql.catalogImplementation = "hive"
spark.executor.memory= "2g"
spark.executor.instances = "1"
spark.yarn.priority = "100"
hive.exec.dynamic.partition.mode = "nonstrict"
spark.dynamicAllocation.enabled="false"
}

source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
plugin_output = "fake"
row.num = 16
schema = {
fields {
name = "string"
age = "int"
}
}
}

# If you would like to get more information about how to configure SeaTunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/connector-v2/source
}

transform {
FieldMapper {
plugin_input = "fake"
plugin_output = "fake1"
field_mapper = {
age = age
name = new_name
}
}
}

sink {
Console {
plugin_input = "fake1"
}

# If you would like to get more information about how to configure SeaTunnel and see full list of sink plugins,
# please go to https://seatunnel.apache.org/docs/connector-v2/sink
}
```

### Running Command

```shell
sh ./bin/start-seatunnel-spark-3-connector-v2.sh --master local[4] --deploy-mode client --config ./job/spark.conf
```

### Error Exception

```log
Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.encoders.ExpressionEncoder org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(org.apache.spark.sql.types.StructType)'
at org.apache.seatunnel.core.starter.spark.execution.TransformExecuteProcessor.sparkTransform(TransformExecuteProcessor.java:180)
at org.apache.seatunnel.core.starter.spark.execution.TransformExecuteProcessor.execute(TransformExecuteProcessor.java:151)
at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.execute(SparkExecution.java:73)
at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:60)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:1075)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$anon$2.doSubmit(SparkSubmit.scala:1167)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1176)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

### Zeta or Flink or Spark Version

Spark 3.5.0

### Java or Scala Version

javac 1.8.0_462

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.seatunnel.core.starter.spark.execution.TransformExecuteProcessor.java around line 180 and inspect the Spark 3.5 RowEncoder failure shown in the stack trace. Run the provided start-seatunnel-spark-3-connector-v2.sh command with job/spark.conf, then verify that the example Spark 3.5 job completes without the NoSuchMethodError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.