[VL] Support Spark 4.2.x
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
## Description
Spark 4.2.0 was released on July 14, 2026. Gluten should add initial Spark 4.2.x support, following the Spark 4.1.x support process in #11339.
Spark 4.2 expands Spark SQL/data types, Data Source V2, CDC, Arrow/PySpark behavior, and streaming/state-store areas. These should be validated during the bring-up for possible shim, compatibility, test, or follow-up offload work.
## Proposed work
- Fix compatibility issues introduced by Spark 4.2
- Add Spark 4.2.x build profile
- Add Spark 4.2.x shim layer
- Add Spark 4.2.x unit test module
- Add/enable Spark 4.2.x test suites in CI
- Track Spark 4.2.x failed unit tests
## Validation areas
- new geospatial types/functions
- DSv2 CDC, transactions, schema evolution, and metrics
- Arrow/PySpark default behavior changes
- streaming and state-store changes
- new SQL functions, vector/sketch functions, and path resolution
## Goal
Initial support should focus on build, shim, and unit-test compatibility first. Native offload support for new Spark 4.2 features can be handled in follow-up issues.
Contributor guide
Research direction
Review the Spark 4.1.x support process in issue #11339 first, then identify the existing build profile, shim layer, unit-test module, and CI suites it describes. Validate Spark 4.2.x compatibility across the listed SQL, DSv2, Arrow/PySpark, streaming, and state-store areas. Done means initial build, shim, unit-test, and CI support, with failed tests tracked and new-feature offload deferred.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- build-system, ci-cd, data-engineering, stream-processing, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100