Epic: Statistics improvements
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We would like to use "statistics" in our project for transformations that rely on the statisics being "correct" (e.g. that the there are no values outside the `min` and `max` range).
DataFusion has several optimizations like this too that rely on statistics being correct such as skipping file scans with limits such as in https://github.com/apache/arrow-datafusion/blob/e54894c39202815b14d9e7eae58f64d3a269c165/datafusion/core/src/datasource/statistics.rs#L34-L33. There are also suggestions of additional such optimizations like https://github.com/apache/arrow-datafusion/issues/6672
However the current Statistics code seems to make it hard to manage the 'are the statistics exact and can they be guaranteed for transformations' (@crepererum noted this quite some time ago on https://github.com/apache/arrow-datafusion/issues/5613). This has recently lead to several bugs such as
* https://github.com/apache/arrow-datafusion/pull/8049
* https://github.com/apache/arrow-datafusion/pull/8094
We would like to make it clearer what is known and what is an estimate is know (e.g. the min/max of row counts may be known, but the actual value may be an estimate after a filter). This is described in more detail on https://github.com/apache/arrow-datafusion/issues/8078
As we began exploring this concept we ran into several issues with Statistics and I think it is getting big enough to warrant its own tracking epic
Related items
- [x] https://github.com/apache/arrow-datafusion/issues/8110
- [x] https://github.com/apache/arrow-datafusion/issues/8111
- [x] https://github.com/apache/arrow-datafusion/issues/8295
- [ ] https://github.com/apache/arrow-datafusion/issues/8229
- [ ] https://github.com/apache/arrow-datafusion/issues/8228
- [ ] https://github.com/apache/arrow-datafusion/issues/8078
- [x] https://github.com/apache/datafusion/issues/14936
- [x] https://github.com/apache/arrow-datafusion/issues/8133
- [ ] https://github.com/apache/arrow-datafusion/issues/8099
- [x] https://github.com/apache/arrow-datafusion/issues/7490
- [ ] https://github.com/apache/arrow-datafusion/issues/5614
- [ ] https://github.com/apache/arrow-datafusion/issues/5613
- [x] https://github.com/apache/arrow-datafusion/issues/8309
- [ ] https://github.com/apache/arrow-datafusion/issues/2976
- [ ] https://github.com/apache/arrow-datafusion/issues/8334
- [x] https://github.com/apache/arrow-datafusion/issues/8335
- [x] https://github.com/apache/datafusion/issues/15809
- [x] https://github.com/apache/arrow-datafusion/issues/7925
- [ ] https://github.com/apache/arrow-datafusion/issues/8342
- [x] https://github.com/apache/datafusion/issues/15809 : manipulating the statistics (e.g. merging files to get partitions statistics) is slow
Pruning Improvements (maybe should be its own epic)
- [ ] https://github.com/apache/arrow-datafusion/issues/7869
- [x] https://github.com/apache/arrow-datafusion/issues/8376
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
This is somewhat related
Contributor guide
Research direction
Begin with datafusion/core/src/datasource/statistics.rs and issue 8078, then review the linked statistics and pruning items to identify a bounded task. This epic does not define a single change, named tests, or a completion condition; its done state must come from one of the related issues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100