apache / apache/datafusion

Implement statistics estimation for partition columns

Open
#1,186 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The ability to read partitioned tables is added the the `ListingTable` in #1141. As partitions are constant columns, we could estimate their statistics pretty accurately (currently they are left as `Default`.

**Describe the solution you'd like**
Replace default with a column statistics that has:
- min value
- max value
- distinct count
- null number as 0

Contributor guide

Open the contributing guide

Research direction

Start with the ListingTable partition-handling entry point added in #1141 and trace where partition columns currently receive Default statistics. Define how partition values provide min, max, distinct count, and a null count of zero, then verify the resulting statistics with the repository's relevant existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.