apache / apache/datafusion

Scalar subquery filters not pushed down to TableScan

Open
#21,324 2 comments 0 reactions 1 assignee Claimed by @neilconway View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Example query (TPC-H Q22):

```sql
SELECT ... FROM customer
WHERE c_acctbal > (SELECT avg(c_acctbal) FROM customer WHERE ...)
```

It would be good to push the predicate down to the scan on `customer`.

The problem with doing this now is that `filter_partitions` calls `create_physical_expr` with a fresh `ExecutionProps`, so we don't have a way to pass the `ScalarSubqueryResults` to any `ScalarSubqueryExpr` nodes in the filter expression. This should be fixable in theory.

We'd also need to update `extract_dml_filters` to apply subquery filters.

This is distinct from using subquery filters for partition pruning, which would be more difficult (we'd need to evaluate the subquery at planning-time).

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.