Adapt column statistics API
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While looking at adding support for more statistics on the Delta Lake `TableProvider` implementation I bumped into some limitation in our statistics API.
Currently columnstatistics is a `Option>`.
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/datasource/datasource.rs#L37
So, it should return the statistics by (correct) index regardless of the order in the files.
**Describe the solution you'd like**
Either:
* Return a `HashMap` rather than a `Option>`
* Pass a `Schema` parameter to `TableProvider::statisitics` so the positions of the fields can be calculated.
FWIW, Delta Lake / delta-rs takes the first approach and seems straightforward to implement and use.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start with datafusion/src/datasource/datasource.rs at the linked TableProvider statistics API, then inspect the callers and uses of ColumnStatistics. Compare the two proposed API shapes and trace how column order is currently handled. Done means the selected representation is implemented consistently and statistics map to the correct columns regardless of file order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100