[VL] Spill support of table scan
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
split and rowgroup prefetch plays the main factor of performance boost in Gluten today, but the cost is high memory consumption. We should enahnce table scan to either pause the prefetch once memory is full or support spill in table scan.
### Gluten version
None
Contributor guide
Research direction
The issue names table scan, split and rowgroup prefetch, and high memory consumption, but no file or test. Start by tracing the table-scan implementation and existing memory-limit or spill tests, then clarify whether the intended outcome is pausing prefetch or spilling. Done should include a reproducible memory-bounded test and verified table-scan behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100