apache / apache/gluten

[VL] Spark 4.1: Support checksum-based shuffle writers (SPARK-53322)

Open
#11,915 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

## Backend
VL (Velox)

## Bug description

Spark 4.1 introduced checksum-based shuffle writers (SPARK-53322/SPARK-54439, sub-tasks of SPARK-51166 Umbrella). Gluten's shuffle writer implementation does not produce the expected checksums, causing test failures.

Spark 4.1 only.

Parent issue: #11550

### Impact

| Suite | Status | spark40 | spark41 |
|-------|--------|:-------:|:-------:|
| GlutenMapStatusEndToEndSuite | Entire suite TODO | 🟢 | 🔴 |
| GlutenKeyGroupedPartitioningSuite | `.excludeByPrefix("SPARK-53322")` | 🟢 | 🔴 |
| GlutenKeyGroupedPartitioningSuite | `.excludeByPrefix("SPARK-54439")` | 🟢 | 🔴 |

### References

- Apache Spark JIRA: [SPARK-53322](https://issues.apache.org/jira/browse/SPARK-53322)
- Apache Spark JIRA: [SPARK-54439](https://issues.apache.org/jira/browse/SPARK-54439)
- Apache Spark JIRA: [SPARK-51166](https://issues.apache.org/jira/browse/SPARK-51166) (Umbrella)

Contributor guide

Open the contributing guide

Research direction

Start with GlutenMapStatusEndToEndSuite and GlutenKeyGroupedPartitioningSuite, focusing on the SPARK-53322 and SPARK-54439 exclusions and the Spark 4.1 failures. Compare the shuffle writer behavior with Spark 4.1's checksum expectations; done means the affected suites pass without those exclusions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.