Find a way to reduce the cost of making new pruning predicate during limit pruning
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
my only concern is that this code is now called for every row file and it makes an entirely new pruning predicate (even when there is no limit and thus the results aren't used). Could we only run this code when there is a limit?
_Originally posted by @alamb in https://github.com/apache/datafusion/pull/18868#discussion_r2666021956_
Contributor guide
Research direction
Start by locating the limit-pruning code discussed in PR 18868 and trace where a new pruning predicate is created for each row file. The change is done when predicate creation is skipped when no limit is present, while preserving the existing limited-pruning behavior; the issue names no specific tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100