[VL] Spark 4.1: Support memory shuffle spill by size threshold (SPARK-49386)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
## Backend
VL (Velox)
**Gluten version**: main branch
## Description
Spark 4.1 introduced memory-based shuffle spill thresholds (SPARK-49386, JIRA type: Improvement). The new `spillSizeThreshold` parameter enables spilling by data size rather than only by row count. Gluten's shuffle implementation does not support this threshold.
Spark 4.1 only.
**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)
### Impact
| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenDataFrameWindowFunctionsSuite | SPARK-49386 spill | 🟢 | 🔴 |
| GlutenJoinSuite | SPARK-49386 SortMergeJoin spill | 🟢 | 🔴 |
Note: `GlutenSQLWindowFunctionSuite` has a pre-existing spill issue ("low buffer spill threshold") unrelated to SPARK-49386 — out of scope for this issue.
### References
- Apache Spark JIRA: [SPARK-49386](https://issues.apache.org/jira/browse/SPARK-49386)
Contributor guide
Research direction
Start by reading Apache Spark SPARK-49386 and the parent tracking issue #11910, then trace Gluten's Velox shuffle implementation for the Spark 4.1 path. Use GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite to reproduce the excluded spill cases; done means both Spark 4.1 cases pass without changing the unrelated window-function spill issue.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100