apache / apache/datafusion

Adapt column statistics API

Open
#717 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While looking at adding support for more statistics on the Delta Lake `TableProvider` implementation I bumped into some limitation in our statistics API.

Currently columnstatistics is a `Option>`.

https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/datasource/datasource.rs#L37

So, it should return the statistics by (correct) index regardless of the order in the files.

**Describe the solution you'd like**
Either:
* Return a `HashMap` rather than a `Option>`
* Pass a `Schema` parameter to `TableProvider::statisitics` so the positions of the fields can be calculated.

FWIW, Delta Lake / delta-rs takes the first approach and seems straightforward to implement and use.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/src/datasource/datasource.rs at the linked TableProvider statistics API, then inspect the callers and uses of ColumnStatistics. Compare the two proposed API shapes and trace how column order is currently handled. Done means the selected representation is implemented consistently and statistics map to the correct columns regardless of file order.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.