apache / apache/gluten

[VL] Spill related issues tracker

Open
#3,030 23 comments 6 reactions 0 assignees View on GitHub
enhancement tracker
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Mirror issue in facebookincubator/velox https://github.com/facebookincubator/velox/issues/6414

This is to list the large memory occupations that are not spillable so far, which means, that are not able to be spilled to disk, in Velox backend's query execution.

Technically the listed items should be all finally fixed ("fix" means to make then spillable), to ensure the memory stability of Gluten. Otherwise there would be chance that OOM error raises during execution that would fail the user query.

The list of non-spillable large occupations (attach PR following each item once fixing):
- [x] Buffered inputs from Velox's window operator
- [x] Streaming window
- [x] Streaming window build
- [ ] Streaming window functions without build, not planned in Velox yet
- [x] Spillable sort window
- [x] Buffered inputs from Velox's hash-aggregate operator, when aggregate is distinct aggregate
- [x] Buffered inputs from Velox's hash-aggregate operator, when aggregate is partial aggregate (needs confirmation)
- [x] Buffered input in Velox's hash-aggregate/hash-join(build)/sort operator, after all input is added
- [x] Hash-aggregate
- [x] Hash-join(build) Velox community is working on this now.
- [x] Sort
- [x] Pre-allocate split buffers from Gluten's Velox shuffle writer
- [x] A task can take use executor's memory if no other task running in the executor, TPCDS Q67. Vanilla spark does this
- [ ] External sort in fallbacked partition write ( can’t be triggered by gluten)

Contributor guide

Open the contributing guide

Research direction

Start with the mirrored Velox issue and the checklist in this issue, especially the unchecked external-sort item in fallbacked partition write. Trace the listed Gluten Velox shuffle, window, aggregate, join, and sort areas before deciding what remains actionable. Done means each triggerable large memory occupation is spillable, or is explicitly documented as not triggerable by Gluten.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
backend, data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.