apache / apache/gluten

[VL] Spark 4.1: Support memory shuffle spill by size threshold (SPARK-49386)

Open
#11,922 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Backend
VL (Velox)

**Gluten version**: main branch

## Description

Spark 4.1 introduced memory-based shuffle spill thresholds (SPARK-49386, JIRA type: Improvement). The new `spillSizeThreshold` parameter enables spilling by data size rather than only by row count. Gluten's shuffle implementation does not support this threshold.

Spark 4.1 only.

**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)

### Impact

| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenDataFrameWindowFunctionsSuite | SPARK-49386 spill | 🟢 | 🔴 |
| GlutenJoinSuite | SPARK-49386 SortMergeJoin spill | 🟢 | 🔴 |

Note: `GlutenSQLWindowFunctionSuite` has a pre-existing spill issue ("low buffer spill threshold") unrelated to SPARK-49386 — out of scope for this issue.

### References

- Apache Spark JIRA: [SPARK-49386](https://issues.apache.org/jira/browse/SPARK-49386)

Contributor guide

Open the contributing guide

Research direction

Start by reading Apache Spark SPARK-49386 and the parent tracking issue #11910, then trace Gluten's Velox shuffle implementation for the Spark 4.1 path. Use GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite to reproduce the excluded spill cases; done means both Spark 4.1 cases pass without changing the unrelated window-function spill issue.

Written by the indexing model from the issue text.

Assessment

Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.