apache / apache/datafusion

PushDownFilter optimizer pushes down filters through distinct on

Open
#12,942 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

a filter should not be pushed down through a distinct on because it will change the results.
The PushDownFilter optimizer rule does not behave correctly.

### To Reproduce

Add this test to push_down_filter.rs:

```
#[test]
fn distinct_on() -> Result<()> {
let table_scan = test_table_scan()?;
let plan = LogicalPlanBuilder::from(table_scan)
.distinct_on(vec![col("a")], vec![col("a")], None)?
.filter(col("a").eq(lit(1i64)))?
.build()?;
// filter appears below Union
let expected = "\
Filter: a = Int64(1)\
\n DistinctOn: on_expr=[[test.a]], select_expr=[[a]], sort_expr=[[]]\
\n TableScan: test";
assert_optimized_plan_eq(plan, expected)
}
```

you will get the logical plan:
```
DistinctOn: on_expr=[[test.a]], select_expr=[[a]], sort_expr=[[]]
TableScan: test, full_filters=[a = Int64(1)]
```

### Expected behavior

Filters should not be pushed down through distinct on.
the logical plan should be as expected in the above test.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in push_down_filter.rs and add the distinct_on test shown in the issue. Run the relevant optimizer tests to reproduce the current plan and compare it with the expected plan. Done means the filter remains above DistinctOn and the test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.