apache / apache/gluten

[VL] Spill support of table scan

Open
#11,880 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

split and rowgroup prefetch plays the main factor of performance boost in Gluten today, but the cost is high memory consumption. We should enahnce table scan to either pause the prefetch once memory is full or support spill in table scan.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

The issue names table scan, split and rowgroup prefetch, and high memory consumption, but no file or test. Start by tracing the table-scan implementation and existing memory-limit or spill tests, then clarify whether the intended outcome is pausing prefetch or spilling. Done should include a reproducible memory-bounded test and verified table-scan behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.