apache / apache/gravitino

[Improvement] Support Apache Spark 4.1 in the Spark connector

Open
#13,018 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

The Spark connector supports Spark 3.5 and 4.0 after #12414, so users on Spark 4.1 have no build to use. Spark 4.1 is also the newest line the connector can support today: Iceberg 1.11.0, the version already pinned for the connector, publishes `iceberg-spark-runtime-4.1_2.13` and nothing for 4.2.

### How should we improve?

Add a `spark-connector/v4.1` module alongside `v3.5` and `v4.0`, publishing `gravitino-spark-4.1_2.13` and `gravitino-spark-connector-runtime-4.1_2.13`, and run the shared integration suite against it in CI.

The module is not a pure copy of the 4.0 one. `ParserInterface` gained a defaulted `parsePlanWithParameters` that `SparkSession.sql` routes through, `ProcedureCatalog.listProcedures` became abstract, and the constructors of Spark's `JDBCTable` and Iceberg's `SparkTable` both changed. Paimon stays unavailable on 4.1, since no released Paimon version publishes `paimon-spark-4.1`.

Contributor guide

Open the contributing guide

Research direction

Start by comparing spark-connector/v4.0 and v3.5, then inspect the shared integration suite and its CI configuration. Account for the listed Spark, Iceberg, and catalog API changes while adding the v4.1 module and artifacts; done means the 4.1 connector publishes successfully and passes the shared integration suite, with Paimon unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.