[VL] pick split with most data prefetched
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Currently in Velox, we pick the first split where its stete is KPrepared, not the one with most data fetched. Once it's picked, we will need to wait until its data return later, even the other splits already have ready data.
Ideally we should wait there until the first split has its first rowgroup data returned, but the logic is too complex. Instead we can pick the split which has most data returned in first rowgroup.
### Gluten version
None
Contributor guide
Research direction
No file, test, or entry point is named; start by locating the Velox split-selection logic used when splits reach KPrepared and inspect nearby tests. Done means the selected split is the prepared one with the most data returned in its first rowgroup, rather than simply the first prepared split, with coverage for competing ready splits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100