apache / apache/datafusion

Statistics: Migrate to `Distribution` from `Precision`

Open
#14,896 9 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion doesn't have a great statistics infrastructure, which will be long project to fix. Luckily, we have begun the process to implement an extensible statistics infrastructure, and we have already merged a new low-level framework to evaluate expressions in a probabilistic context (i.e. the `evaluate_statistics` and `propagate_statistics` API introduced in #14699). The next step in this project is to switch from the old `Precision` object to the new `Distribution` object across the DataFusion codebase. This way, we will be able to use the existing mechanism around the `ColumnStatistics` and `Statistics` (maybe we should rename this to `TableStatistics`?) objects to handle statistics.

### Describe the solution you'd like

Deprecating/removing `Precision`, updating its usages with `Distribution`.

### Describe alternatives you've considered

N/A.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the evaluate_statistics and propagate_statistics APIs introduced in #14699, then locate usages of Precision across the DataFusion codebase. Replace or deprecate those usages in favor of Distribution while preserving the existing ColumnStatistics and Statistics mechanisms for handling statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.