Statistics: Migrate to `Distribution` from `Precision`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion doesn't have a great statistics infrastructure, which will be long project to fix. Luckily, we have begun the process to implement an extensible statistics infrastructure, and we have already merged a new low-level framework to evaluate expressions in a probabilistic context (i.e. the `evaluate_statistics` and `propagate_statistics` API introduced in #14699). The next step in this project is to switch from the old `Precision` object to the new `Distribution` object across the DataFusion codebase. This way, we will be able to use the existing mechanism around the `ColumnStatistics` and `Statistics` (maybe we should rename this to `TableStatistics`?) objects to handle statistics.
### Describe the solution you'd like
Deprecating/removing `Precision`, updating its usages with `Distribution`.
### Describe alternatives you've considered
N/A.
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the evaluate_statistics and propagate_statistics APIs introduced in #14699, then locate usages of Precision across the DataFusion codebase. Replace or deprecate those usages in favor of Distribution while preserving the existing ColumnStatistics and Statistics mechanisms for handling statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100