apache / apache/gluten

[VL] Support Spark 4.2.x

Open
#12,569 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Description

Spark 4.2.0 was released on July 14, 2026. Gluten should add initial Spark 4.2.x support, following the Spark 4.1.x support process in #11339.

Spark 4.2 expands Spark SQL/data types, Data Source V2, CDC, Arrow/PySpark behavior, and streaming/state-store areas. These should be validated during the bring-up for possible shim, compatibility, test, or follow-up offload work.

## Proposed work

- Fix compatibility issues introduced by Spark 4.2
- Add Spark 4.2.x build profile
- Add Spark 4.2.x shim layer
- Add Spark 4.2.x unit test module
- Add/enable Spark 4.2.x test suites in CI
- Track Spark 4.2.x failed unit tests

## Validation areas

- new geospatial types/functions
- DSv2 CDC, transactions, schema evolution, and metrics
- Arrow/PySpark default behavior changes
- streaming and state-store changes
- new SQL functions, vector/sketch functions, and path resolution

## Goal

Initial support should focus on build, shim, and unit-test compatibility first. Native offload support for new Spark 4.2 features can be handled in follow-up issues.

Contributor guide

Open the contributing guide

Research direction

Review the Spark 4.1.x support process in issue #11339 first, then identify the existing build profile, shim layer, unit-test module, and CI suites it describes. Validate Spark 4.2.x compatibility across the listed SQL, DSv2, Arrow/PySpark, streaming, and state-store areas. Done means initial build, shim, unit-test, and CI support, with failed tests tracked and new-feature offload deferred.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
build-system, ci-cd, data-engineering, stream-processing, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.