apache / apache/gluten

[VL] UnboundedWindowFunctionFrame OOM when partition is large

Open
#10,500 8 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

UnboundedWindowFunctionFrame is entire Partition Frame, velox will load the entire partition's data into memory, which may cause OOM.
For example:
```
select *, sum(c1) over () as sum from t
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied `select *, sum(c1) over () as sum from t` query with a large partition, then trace `UnboundedWindowFunctionFrame` in the Velox backend. Done means the unbounded window no longer loads the entire partition into memory and still returns correct results; add a regression test covering the large-partition case.

Written by the indexing model from the issue text.

Assessment

Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.