apache / apache/datafusion

Record per-predicate statistics for effectiveness of parquet predicate pushdown

Open
#3,998 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

LGTM, I think if there any more than one predicates in one query, should we record the each predicate's input records count to calculate the efficiency 🤔 So could guide the user do the rearrangement.

_Originally posted by @Ted-Jiang in https://github.com/apache/arrow-datafusion/pull/3989#pullrequestreview-1159337459_

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by tracing DataFusion's parquet predicate pushdown path and any existing query statistics, then determine how per-predicate input counts should be recorded and used to assess predicate ordering. Done means the design and implementation report each predicate's input count and provide meaningful effectiveness data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.