apache / apache/gluten

[CORE] Support Spark 4.0 and Deprecate JDK 8

Open
#8,574 6 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Current Spark and JDK support status are in below
Spark 3.4.4 support JDK8(8u362) or above version
Spark 3.5.4 support JDK8(8u371) or above version
Spark 4.0 can only support JDK 11 and 17
For better CI resource usage, we can consider to deprecate JDK 8 support when adding Spark 4.0 support.
The plan to support Spark 4.0 will be initialized in Gluten 1.4 and done the work in Gluten 1.5.

One other consideration is in Iceberg cannot support JDK 8 with Spark 3.4.4, we have Iceberg UT may consider to separate the UT testing for Spark 3.4.4 and Iceberg or just support JDK 11/17 for better alignment.

Contributor guide

Open the contributing guide

Research direction

Begin by tracing the current Spark/JDK support matrix and CI jobs, then inspect the Iceberg UT setup for its Spark 3.4.4 path. Done means Spark 4.0 runs on JDK 11/17, JDK 8 is deprecated as planned, and Iceberg tests have an explicit compatible test arrangement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.