DataFusion not using NDV stat
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
This was brought up before here https://github.com/apache/datafusion/issues/15265
From the top of my head my I think we could use these in:
- Join swapping strategy
- Optimal number of partitions in Hash Join partitioning where we use Hash Partitioning.
- Filter pushdown decision: if we have an very low NDV it might not be worth pushing down // computing predicates?
This would likely require further investigation into how other query engines use NDV for optimization decisions 🤔
### To Reproduce
_No response_
### Expected behavior
ndv being used
### Additional context
_No response_
Contributor guide
Research direction
Read the linked issue #15265 first, then investigate how NDV statistics could affect join swapping, hash-join partition counts, and filter-pushdown decisions. Done means NDV is used for at least one justified optimization decision, with the behavior validated by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100