apache / apache/datafusion

Implement statistics support for Substrait

Open
#8,698 7 comments 1 reaction 0 assignees View on GitHub
enhancement substrait
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

A report from Twitter https://twitter.com/mim_djo/status/1740542585410814393

Says:

> a new release of #datafusion 34, still reading #Deltatable via arrow is suboptimal compared to reading Parquet Directly :( something to do with passing stats to get correct join orders.

![image](https://github.com/apache/arrow-datafusion/assets/490673/c75637da-6408-461a-be27-513a13443c3f)

I think the issue is that https://github.com/apache/arrow-datafusion/issues/7949 and https://github.com/apache/arrow-datafusion/issues/7950 rely on statistics to pick non bad join orders for TPCH queries.

These statistics are not available from the delta provider it seems.

@andygrove says

> RelCommon (common to all operators in Substrait) can contain a hint that has stats

```
message Stats {
double row_count = 1;
double record_size = 2;
substrait.extensions.AdvancedExtension advanced_extension = 10;
}
```

### Describe the solution you'd like

I would like the Datafusion substrait consumer/producer to handle translating

### Describe alternatives you've considered

_No response_

### Additional context

This was brought up by @Dandandan on the ASF slack: https://the-asf.slack.com/archives/C04RJ0C85UZ/p1703885214702039

Contributor guide

Open the contributing guide

Research direction

Start by tracing the DataFusion Substrait consumer and producer, then read the Substrait RelCommon and Stats definitions described in the issue. Compare how statistics from the Delta provider are represented and determine what translation is needed so join-order selection can use them; done means both sides handle the statistics consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.