Avoiding spilling in TopK queries by reinserting the to-spill data to memory buffer
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We recently added optimizations for `ORDER BY expr LIMIT` by pushing limits to individual operations (saving memory, CPU time + limiting output rows) and executing sorts in parallel.
The disk spill operation in `SortExec` currently still assumes the to-spill disk doesn't fit in memory.
However after sorting we only have to keep the batch(es) with top `fetch` rows and store those, which probably avoids spilling to disk.
**Describe the solution you'd like**
We can identify that the to-spill data fits in memory after being merged / sorted and avoid spilling to disk.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by reading the SortExec disk-spill path and the existing ORDER BY expr LIMIT limit-pushdown and parallel-sort work described in the issue. Determine how merged top-fetch batches can be measured against the memory buffer; done means fitting batches are retained in memory without disk spilling, with relevant sort and spill behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100