apache / apache/datafusion

Use `min_value` and `max_value` on statistics to avoid `ExecutionPlan.execute`

Open
#10,400 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Maybe related to #5535, but I couldn't find anything identical, so created a fresh issue.

If this is a known bug and you think the fix might be moderate in scope, I'm happy to have a go at fixing it?

### To Reproduce

I have a custom `TableProvider` and `ExecutionPlan`, where calling `execute` is somewhat expensive and I want to avoid calling it if no data will match.

The execution plan can return helpful statistics from `.statistics()`, including for example, for one column:

```rs
...
ColumnStatistics {
null_count: Precision::Exact(0),
max_value: Precision::Exact(ScalarValue::Int64(Some(4))),
min_value: Precision::Exact(ScalarValue::Int64(Some(4))),
distinct_count: Precision::Exact(1),
},
```

E.g. "in this column all values are equal to 4". This is successfully used by Datafusion if I query `value is null`, the `execute()` function is never alled.

But if I query `value > 5` or `value < 0`, the statistic is ignored and `execute()` is still called.

### Expected behavior

`min_value` and `max_value` of `ColumnStatistics` should be used for pruning and the query plan should not require the "slow" execute method to be called.

### Additional context

I can give a fairly minimal example if required, but I thought best to report the issue and check if it was well known before going to that effort?

I've tried this on both `main` (as of today) and `37.1.0`

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the custom TableProvider and ExecutionPlan behavior described for predicates such as `value > 5` and `value < 0`, using the reported ColumnStatistics values. Trace the statistics-based pruning path and compare it with the `value is null` case; done means qualifying min/max statistics prevent the unnecessary `execute()` call.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.