apache / apache/datafusion

Apply guarantee rewriter to sql workflow

Open
#10,456 17 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

While deprecating`Expr::GetIndexedField`, I found there are many test cases that are not covered in sqllogictest, for example, `test_inequalities_non_null_bounded`. Since we hope to replace the `field` API with `get_field`. We could either move the test to `datafusion/core/tests` or sqllogictest. I prefer the latter, then, I found that guarantee rewrite is not applied to SQL workflow.
```
statement ok
create table t (c int) as values (1), (3), (5);

query TT
explain select struct(c) from t where c between 3 and 1;
----
logical_plan
01)Projection: struct(t.c)
02)--Filter: t.c >= Int32(3) AND t.c <= Int32(1)
03)----TableScan: t projection=[c]
physical_plan
01)ProjectionExec: expr=[struct(c@0) as struct(t.c)]
02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
03)----CoalesceBatchesExec: target_batch_size=8192
04)------FilterExec: c@0 >= 3 AND c@0 <= 1
05)--------MemoryExec: partitions=1, partition_sizes=[1]

statement ok
drop table t;
```

I expect that `FilterExec` should be removed or converted to something like `False`, since the condition here is always false.

### Describe the solution you'd like

Apply `guarantee_rewriter` to sql workflow.
If the simplification logic can be included in `Simplifier` is a plus.

### Describe alternatives you've considered

_No response_

### Additional context

PR that introduce guarantee rewrite https://github.com/apache/datafusion/pull/7467

_No response_

Contributor guide

Open the contributing guide

Research direction

Read the guarantee_rewriter introduced by PR #7467, then trace the SQL workflow and its sqllogictest coverage; compare with datafusion/core/tests. Done means the shown BETWEEN contradiction is simplified so FilterExec is removed or represented as false, with the explain output covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.