apache / apache/hudi

Drop Apache Spark 3.3 integration support

Open
#18,784 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Proposal
Remove the `hudi-spark3.3.x` module and `spark3.3` Maven profile. After this change, the minimum supported Spark version is 3.4.

We discussed that this will be done in future releases to support existing Hudi users better, based on the discussion in #18917.

### Motivation
- Spark 3.3 reached end-of-life upstream.
- Maintaining the 3.3 adapter blocks simplifications in the shared Spark glue code.

### Scope
- Delete `hudi-spark-datasource/hudi-spark3.3.x/`.
- Remove the `spark3.3` profile and `spark33.version` property from the root `pom.xml`.
- Drop Spark 3.3 jobs from CI workflows, release scripts, bundle validation, docker, and docs.
- Strip `isSpark3_3` / `gteqSpark3_3_2` branches and the `Spark3_3Adapter` fallback in shared code.

### Out of scope (follow-up)
Refactoring shared code that is now redundant given the higher minimum Spark version (e.g., merging branches that only existed to fork 3.3 vs. 3.4+).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with hudi-spark-datasource/hudi-spark3.3.x/ and the root pom.xml to identify the module, profile, and version property to remove. Then search the CI workflows, release scripts, bundle validation, docker, and docs for Spark 3.3 references, along with isSpark3_3, gteqSpark3_3_2, and Spark3_3Adapter. Done means Spark 3.3 support and its references are removed while the stated follow-up refactoring remains out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, spark
Domain
build-system, ci-cd, data-engineering, distributed-systems, documentation
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.