Implement statistics support for Substrait
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
A report from Twitter https://twitter.com/mim_djo/status/1740542585410814393
Says:
> a new release of #datafusion 34, still reading #Deltatable via arrow is suboptimal compared to reading Parquet Directly :( something to do with passing stats to get correct join orders.

I think the issue is that https://github.com/apache/arrow-datafusion/issues/7949 and https://github.com/apache/arrow-datafusion/issues/7950 rely on statistics to pick non bad join orders for TPCH queries.
These statistics are not available from the delta provider it seems.
@andygrove says
> RelCommon (common to all operators in Substrait) can contain a hint that has stats
```
message Stats {
double row_count = 1;
double record_size = 2;
substrait.extensions.AdvancedExtension advanced_extension = 10;
}
```
### Describe the solution you'd like
I would like the Datafusion substrait consumer/producer to handle translating
### Describe alternatives you've considered
_No response_
### Additional context
This was brought up by @Dandandan on the ASF slack: https://the-asf.slack.com/archives/C04RJ0C85UZ/p1703885214702039
Contributor guide
Research direction
Start by tracing the DataFusion Substrait consumer and producer, then read the Substrait RelCommon and Stats definitions described in the issue. Compare how statistics from the Delta provider are represented and determine what translation is needed so join-order selection can use them; done means both sides handle the statistics consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100