[CORE] Drop Spark 3.2 support
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Discussed in https://github.com/apache/incubator-gluten/discussions/8491
As [4.0 support](https://github.com/apache/incubator-gluten/issues/8852) is oncomming, we will then remove 3.2 support. We had an informal vote completed at https://lists.apache.org/thread/fgnv7v5sf9ol14lkssd9v47h6cvorcxc.
Originally posted by **zhztheplayer** January 10, 2025
I've heard more and more people are upgrading their Spark clusters from 3.2 to higher versions. So perhaps it's necessary for us to discuss about dropping Spark 3.2 support from our main branch.
I have been auditing on the feasibility of this change because we can have chance to solve a long-existing issue https://github.com/apache/incubator-gluten/issues/540 after the upgradation. The issue doesn't impact the stability of Gluten but could make the columnar batch management code complex to maintain, and could slightly impact performance. There might be other bonuses so feel free to supplement here.
Contributor guide
Research direction
Start with the linked discussion, Spark 4.0 support issue #8852, and longstanding issue #540 to understand the agreed scope. Review the repository's Spark-version support points and remove the 3.2 support path; done means the main branch no longer supports Spark 3.2 without regressing supported versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100