Consider add already pushed filters to those that passed to `supports_filters_pushdown`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Consider the next use-case: there is table t(a, b) and some index built for this table. The index can support exactly (`TableProviderFilterPushDown::Exact`) only a single predicate. For example, it can support `a == 1` or `b > 1` but not both at the same time (it is the index itself nature).
For now, `push_down_filter` do not account that some filters were pushed down earlier. For example, if we implement the use-case above as a table provider (just a mock to show the problem), which does:
```Rust
fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> Result> {
// Can support only single filter.
Ok(if filters.is_empty() {
vec![]
} else {
let mut push = vec![TableProviderFilterPushDown::Exact; 1];
for i in 0..filters.len() - 1 {
push.push(TableProviderFilterPushDown::Unsupported);
}
push
})
}
```
And there is a query `SELECT * FROM t WHERE a > 1 AND b == 1` -- it will return [`Exact`, `Unsupported`] for the first `supports_filters_pushdown` call, and then, if the optimizer will try to push down filters again (if there are several optimizer passes), it will pass `b == 1` and got `Exact`. But it is not true for the bunch of filters.
https://github.com/apache/datafusion/blob/3e30f77f08aa9184029da80c7f7e2ec00999fa44/datafusion/optimizer/src/push_down_filter.rs#L1086-L1097
As a result, the next logical plan we got (no `Filter` node, but, in fact, there is an unsupported filter):
```
"TableScan: t projection=[a, b], full_filters=[t.a > Int64(1)], unsupported_filters=[t.b = Int64(1)]"
```
I thought about workarounds, for example:
- Remember which filters were pushed in the table provider itself. It is not working for the scenario when there are multiple parallel queries in this session.
- Return `Inexact` always for such filters. It does not satisfy, because some filters are really could be supported.
I suggest to pass already pushed filters into the `supports_filters_pushdown` and rebuild the list from scratch each time. Please, highlight the pitfalls, mis-usage, or tell that it is ok and I will work on the patch.
Contributor guide
Research direction
Start at datafusion/optimizer/src/push_down_filter.rs around the linked lines and trace repeated optimizer passes calling supports_filters_pushdown. Reproduce the table-provider scenario with one Exact and one Unsupported predicate, then verify that an unsupported filter cannot disappear from the logical plan after another pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100