apache / apache/datafusion

Avoiding spilling in TopK queries by reinserting the to-spill data to memory buffer

Open
#3,579 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We recently added optimizations for `ORDER BY expr LIMIT` by pushing limits to individual operations (saving memory, CPU time + limiting output rows) and executing sorts in parallel.

The disk spill operation in `SortExec` currently still assumes the to-spill disk doesn't fit in memory.
However after sorting we only have to keep the batch(es) with top `fetch` rows and store those, which probably avoids spilling to disk.

**Describe the solution you'd like**
We can identify that the to-spill data fits in memory after being merged / sorted and avoid spilling to disk.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by reading the SortExec disk-spill path and the existing ORDER BY expr LIMIT limit-pushdown and parallel-sort work described in the issue. Determine how merged top-fetch batches can be measured against the memory buffer; done means fitting batches are retained in memory without disk spilling, with relevant sort and spill behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.