Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Deciding whether to dictionary-encode a column requires knowing its cardinality (NDV), Datafusion does support https://docs.rs/datafusion/latest/datafusion/common/struct.ColumnStatistics.html but we need to validate this.
### Describe the solution you'd like
Improve NDV statistics collection focused on Parquet, and get these stats into DataFusion in a form the optimizer can actually use (e.g. exposed on `ColumnStatistics`).
### Describe alternatives you've considered
_No response_
### Additional context
Direct sub-issue of #22891
piece of Epic : #24111
Contributor guide
Research direction
Start by validating DataFusion's existing ColumnStatistics path and tracing how Parquet RLE and dictionary-array statistics are collected. Read the context in issues #22891 and #24111. Done means improved NDV statistics are exposed through ColumnStatistics in a form the optimizer can use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100