[VL] Spark 4.1: Support checksum-based shuffle writers (SPARK-53322)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
## Backend
VL (Velox)
## Bug description
Spark 4.1 introduced checksum-based shuffle writers (SPARK-53322/SPARK-54439, sub-tasks of SPARK-51166 Umbrella). Gluten's shuffle writer implementation does not produce the expected checksums, causing test failures.
Spark 4.1 only.
Parent issue: #11550
### Impact
| Suite | Status | spark40 | spark41 |
|-------|--------|:-------:|:-------:|
| GlutenMapStatusEndToEndSuite | Entire suite TODO | 🟢 | 🔴 |
| GlutenKeyGroupedPartitioningSuite | `.excludeByPrefix("SPARK-53322")` | 🟢 | 🔴 |
| GlutenKeyGroupedPartitioningSuite | `.excludeByPrefix("SPARK-54439")` | 🟢 | 🔴 |
### References
- Apache Spark JIRA: [SPARK-53322](https://issues.apache.org/jira/browse/SPARK-53322)
- Apache Spark JIRA: [SPARK-54439](https://issues.apache.org/jira/browse/SPARK-54439)
- Apache Spark JIRA: [SPARK-51166](https://issues.apache.org/jira/browse/SPARK-51166) (Umbrella)
Contributor guide
Research direction
Start with GlutenMapStatusEndToEndSuite and GlutenKeyGroupedPartitioningSuite, focusing on the SPARK-53322 and SPARK-54439 exclusions and the Spark 4.1 failures. Compare the shuffle writer behavior with Spark 4.1's checksum expectations; done means the affected suites pass without those exclusions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100