apache / apache/datafusion

Consider add already pushed filters to those that passed to `supports_filters_pushdown`

Open
#16,075 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Consider the next use-case: there is table t(a, b) and some index built for this table. The index can support exactly (`TableProviderFilterPushDown::Exact`) only a single predicate. For example, it can support `a == 1` or `b > 1` but not both at the same time (it is the index itself nature).

For now, `push_down_filter` do not account that some filters were pushed down earlier. For example, if we implement the use-case above as a table provider (just a mock to show the problem), which does:

```Rust
fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> Result> {
// Can support only single filter.
Ok(if filters.is_empty() {
vec![]
} else {
let mut push = vec![TableProviderFilterPushDown::Exact; 1];
for i in 0..filters.len() - 1 {
push.push(TableProviderFilterPushDown::Unsupported);
}
push
})
}
```

And there is a query `SELECT * FROM t WHERE a > 1 AND b == 1` -- it will return [`Exact`, `Unsupported`] for the first `supports_filters_pushdown` call, and then, if the optimizer will try to push down filters again (if there are several optimizer passes), it will pass `b == 1` and got `Exact`. But it is not true for the bunch of filters.

https://github.com/apache/datafusion/blob/3e30f77f08aa9184029da80c7f7e2ec00999fa44/datafusion/optimizer/src/push_down_filter.rs#L1086-L1097

As a result, the next logical plan we got (no `Filter` node, but, in fact, there is an unsupported filter):
```
"TableScan: t projection=[a, b], full_filters=[t.a > Int64(1)], unsupported_filters=[t.b = Int64(1)]"
```

I thought about workarounds, for example:

- Remember which filters were pushed in the table provider itself. It is not working for the scenario when there are multiple parallel queries in this session.
- Return `Inexact` always for such filters. It does not satisfy, because some filters are really could be supported.

I suggest to pass already pushed filters into the `supports_filters_pushdown` and rebuild the list from scratch each time. Please, highlight the pitfalls, mis-usage, or tell that it is ok and I will work on the patch.

Contributor guide

Open the contributing guide

Research direction

Start at datafusion/optimizer/src/push_down_filter.rs around the linked lines and trace repeated optimizer passes calling supports_filters_pushdown. Reproduce the table-provider scenario with one Exact and one Unsupported predicate, then verify that an unsupported filter cannot disappear from the logical plan after another pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.