Record per-predicate statistics for effectiveness of parquet predicate pushdown
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
LGTM, I think if there any more than one predicates in one query, should we record the each predicate's input records count to calculate the efficiency 🤔 So could guide the user do the rearrangement.
_Originally posted by @Ted-Jiang in https://github.com/apache/arrow-datafusion/pull/3989#pullrequestreview-1159337459_
Contributor guide
Research direction
The issue names no files or tests. Start by tracing DataFusion's parquet predicate pushdown path and any existing query statistics, then determine how per-predicate input counts should be recorded and used to assess predicate ordering. Done means the design and implementation report each predicate's input count and provide meaningful effectiveness data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100