[CORE] Support Spark 4.0 and Deprecate JDK 8
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Current Spark and JDK support status are in below
Spark 3.4.4 support JDK8(8u362) or above version
Spark 3.5.4 support JDK8(8u371) or above version
Spark 4.0 can only support JDK 11 and 17
For better CI resource usage, we can consider to deprecate JDK 8 support when adding Spark 4.0 support.
The plan to support Spark 4.0 will be initialized in Gluten 1.4 and done the work in Gluten 1.5.
One other consideration is in Iceberg cannot support JDK 8 with Spark 3.4.4, we have Iceberg UT may consider to separate the UT testing for Spark 3.4.4 and Iceberg or just support JDK 11/17 for better alignment.
Contributor guide
Research direction
Begin by tracing the current Spark/JDK support matrix and CI jobs, then inspect the Iceberg UT setup for its Spark 3.4.4 path. Done means Spark 4.0 runs on JDK 11/17, JDK 8 is deprecated as planned, and Iceberg tests have an explicit compatible test arrangement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100