apache / apache/datafusion

Epic: Statistics improvements

Open
#8,227 17 comments 0 reactions 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We would like to use "statistics" in our project for transformations that rely on the statisics being "correct" (e.g. that the there are no values outside the `min` and `max` range).

DataFusion has several optimizations like this too that rely on statistics being correct such as skipping file scans with limits such as in https://github.com/apache/arrow-datafusion/blob/e54894c39202815b14d9e7eae58f64d3a269c165/datafusion/core/src/datasource/statistics.rs#L34-L33. There are also suggestions of additional such optimizations like https://github.com/apache/arrow-datafusion/issues/6672

However the current Statistics code seems to make it hard to manage the 'are the statistics exact and can they be guaranteed for transformations' (@crepererum noted this quite some time ago on https://github.com/apache/arrow-datafusion/issues/5613). This has recently lead to several bugs such as
* https://github.com/apache/arrow-datafusion/pull/8049
* https://github.com/apache/arrow-datafusion/pull/8094

We would like to make it clearer what is known and what is an estimate is know (e.g. the min/max of row counts may be known, but the actual value may be an estimate after a filter). This is described in more detail on https://github.com/apache/arrow-datafusion/issues/8078

As we began exploring this concept we ran into several issues with Statistics and I think it is getting big enough to warrant its own tracking epic

Related items
- [x] https://github.com/apache/arrow-datafusion/issues/8110
- [x] https://github.com/apache/arrow-datafusion/issues/8111
- [x] https://github.com/apache/arrow-datafusion/issues/8295
- [ ] https://github.com/apache/arrow-datafusion/issues/8229
- [ ] https://github.com/apache/arrow-datafusion/issues/8228
- [ ] https://github.com/apache/arrow-datafusion/issues/8078
- [x] https://github.com/apache/datafusion/issues/14936
- [x] https://github.com/apache/arrow-datafusion/issues/8133
- [ ] https://github.com/apache/arrow-datafusion/issues/8099
- [x] https://github.com/apache/arrow-datafusion/issues/7490
- [ ] https://github.com/apache/arrow-datafusion/issues/5614
- [ ] https://github.com/apache/arrow-datafusion/issues/5613
- [x] https://github.com/apache/arrow-datafusion/issues/8309
- [ ] https://github.com/apache/arrow-datafusion/issues/2976
- [ ] https://github.com/apache/arrow-datafusion/issues/8334
- [x] https://github.com/apache/arrow-datafusion/issues/8335
- [x] https://github.com/apache/datafusion/issues/15809
- [x] https://github.com/apache/arrow-datafusion/issues/7925
- [ ] https://github.com/apache/arrow-datafusion/issues/8342
- [x] https://github.com/apache/datafusion/issues/15809 : manipulating the statistics (e.g. merging files to get partitions statistics) is slow

Pruning Improvements (maybe should be its own epic)
- [ ] https://github.com/apache/arrow-datafusion/issues/7869
- [x] https://github.com/apache/arrow-datafusion/issues/8376

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

This is somewhat related

Contributor guide

Open the contributing guide

Research direction

Begin with datafusion/core/src/datasource/statistics.rs and issue 8078, then review the linked statistics and pruning items to identify a bounded task. This epic does not define a single change, named tests, or a completion condition; its done state must come from one of the related issues.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.