apache / apache/seatunnel

Submitting seatunnel to spark k8s 3.5 get error with useCommitCoordinator()

Open
#9,799 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

Hi, When I submit seatunnel to my k8s cluster with spark 3.5 version image, I got this exception:

`Exception in thread "main" java.lang.AbstractMethodError: Receiver class org.apache.seatunnel.translation.spark.sink.SeaTunnelBatchWrite does not define or inherit an implementation of the resolved method 'boolean useCommitCoordinator()' of interface org.apache.spark.sql.connector.write.BatchWrite.`

I manually add useCommitCoordinator() function in SeaTunnelBatchWrite.java and use my modified spark starter jar, but this issue still exists.

Image

Could you help find why this issue comes? Many Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.seatunnel.translation.spark.sink.SeaTunnelBatchWrite.java and the Spark 3.5 BatchWrite interface referenced by the exception. Reproduce the submission using the Spark 3.5 image on Kubernetes, including the starter jar, and compare the runtime interface with the compiled implementation. Done means the submission no longer raises AbstractMethodError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.