apache / apache/datafusion

Find a way to reduce the cost of making new pruning predicate during limit pruning

Open
#19,847 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

my only concern is that this code is now called for every row file and it makes an entirely new pruning predicate (even when there is no limit and thus the results aren't used). Could we only run this code when there is a limit?

_Originally posted by @alamb in https://github.com/apache/datafusion/pull/18868#discussion_r2666021956_

Contributor guide

Open the contributing guide

Research direction

Start by locating the limit-pruning code discussed in PR 18868 and trace where a new pruning predicate is created for each row file. The change is done when predicate creation is skipped when no limit is present, while preserving the existing limited-pruning behavior; the issue names no specific tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.