apache / apache/gluten

[VL] pick split with most data prefetched

Open
#11,821 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently in Velox, we pick the first split where its stete is KPrepared, not the one with most data fetched. Once it's picked, we will need to wait until its data return later, even the other splits already have ready data.

Ideally we should wait there until the first split has its first rowgroup data returned, but the logic is too complex. Instead we can pick the split which has most data returned in first rowgroup.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named; start by locating the Velox split-selection logic used when splits reach KPrepared and inspect nearby tests. Done means the selected split is the prepared one with the most data returned in its first rowgroup, rather than simply the first prepared split, with coverage for competing ready splits.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.