[Improvement] Support Apache Spark 4.1 in the Spark connector
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
The Spark connector supports Spark 3.5 and 4.0 after #12414, so users on Spark 4.1 have no build to use. Spark 4.1 is also the newest line the connector can support today: Iceberg 1.11.0, the version already pinned for the connector, publishes `iceberg-spark-runtime-4.1_2.13` and nothing for 4.2.
### How should we improve?
Add a `spark-connector/v4.1` module alongside `v3.5` and `v4.0`, publishing `gravitino-spark-4.1_2.13` and `gravitino-spark-connector-runtime-4.1_2.13`, and run the shared integration suite against it in CI.
The module is not a pure copy of the 4.0 one. `ParserInterface` gained a defaulted `parsePlanWithParameters` that `SparkSession.sql` routes through, `ProcedureCatalog.listProcedures` became abstract, and the constructors of Spark's `JDBCTable` and Iceberg's `SparkTable` both changed. Paimon stays unavailable on 4.1, since no released Paimon version publishes `paimon-spark-4.1`.
Contributor guide
Research direction
Start by comparing spark-connector/v4.0 and v3.5, then inspect the shared integration suite and its CI configuration. Account for the listed Spark, Iceberg, and catalog API changes while adding the v4.1 module and artifacts; done means the 4.1 connector publishes successfully and passes the shared integration suite, with Paimon unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100