Consolidate statistics aggregation
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
There are at least three places in DataFusion where multiple `Statistics` objects are aggregated together, and they do so inconsistently:
1. `get_statistics_with_limit`: https://github.com/apache/arrow-datafusion/blob/e54894c39202815b14d9e7eae58f64d3a269c165/datafusion/core/src/datasource/statistics.rs#L34-L33
2 . Parquet::infer_stats: https://github.com/apache/arrow-datafusion/blob/a892300a5a56c97b5b4ddc9aa4a421aaf412d0fe/datafusion/core/src/datasource/file_format/parquet.rs#L503-L581
3. Union::statistics: https://github.com/apache/arrow-datafusion/blob/c2e768052c43e4bab6705ee76befc19de383c2cb/datafusion/physical-plan/src/union.rs#L612-L611
(and we actually have another version of this in IOx)
### Describe the solution you'd like
I would like to consolidate the three implementations into a `StatisticsAggregator` that knows how to aggregate multiple `Statistics` objects that is both documented and well tested.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Compare the aggregation logic in datafusion/core/src/datasource/statistics.rs, datafusion/core/src/datasource/file_format/parquet.rs, and datafusion/physical-plan/src/union.rs. Start from get_statistics_with_limit, Parquet::infer_stats, and Union::statistics; done means the shared StatisticsAggregator is documented, the three implementations use it, and its behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100