Submitting seatunnel to spark k8s 3.5 get error with useCommitCoordinator()
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 210
Description
Hi, When I submit seatunnel to my k8s cluster with spark 3.5 version image, I got this exception:
`Exception in thread "main" java.lang.AbstractMethodError: Receiver class org.apache.seatunnel.translation.spark.sink.SeaTunnelBatchWrite does not define or inherit an implementation of the resolved method 'boolean useCommitCoordinator()' of interface org.apache.spark.sql.connector.write.BatchWrite.`
I manually add useCommitCoordinator() function in SeaTunnelBatchWrite.java and use my modified spark starter jar, but this issue still exists.
Could you help find why this issue comes? Many Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.seatunnel.translation.spark.sink.SeaTunnelBatchWrite.java and the Spark 3.5 BatchWrite interface referenced by the exception. Reproduce the submission using the Spark 3.5 image on Kubernetes, including the starter jar, and compare the runtime interface with the compiled implementation. Done means the submission no longer raises AbstractMethodError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100