Drop Apache Spark 3.3 integration support
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Proposal
Remove the `hudi-spark3.3.x` module and `spark3.3` Maven profile. After this change, the minimum supported Spark version is 3.4.
We discussed that this will be done in future releases to support existing Hudi users better, based on the discussion in #18917.
### Motivation
- Spark 3.3 reached end-of-life upstream.
- Maintaining the 3.3 adapter blocks simplifications in the shared Spark glue code.
### Scope
- Delete `hudi-spark-datasource/hudi-spark3.3.x/`.
- Remove the `spark3.3` profile and `spark33.version` property from the root `pom.xml`.
- Drop Spark 3.3 jobs from CI workflows, release scripts, bundle validation, docker, and docs.
- Strip `isSpark3_3` / `gteqSpark3_3_2` branches and the `Spark3_3Adapter` fallback in shared code.
### Out of scope (follow-up)
Refactoring shared code that is now redundant given the higher minimum Spark version (e.g., merging branches that only existed to fork 3.3 vs. 3.4+).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with hudi-spark-datasource/hudi-spark3.3.x/ and the root pom.xml to identify the module, profile, and version property to remove. Then search the CI workflows, release scripts, bundle validation, docker, and docs for Spark 3.3 references, along with isSpark3_3, gteqSpark3_3_2, and Spark3_3Adapter. Done means Spark 3.3 support and its references are removed while the stated follow-up refactoring remains out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, spark
- Domain
- build-system, ci-cd, data-engineering, distributed-systems, documentation
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100