apache / apache/datafusion

Improve filter push-down

Open
#19,929 6 comments 0 reactions 1 assignee Claimed by @askalt View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

This issue covers two related filter push-down improvements.

## Pass previously pushed filters to supports_filters_pushdown

Currently, the optimization does not pass filters that were pushed in a previous run (`TableScan::filters`) to `TableProvider::supports_filters_pushdown(...)`.

If the optimizer runs multiple times, it may try to push filters into the table provider multiple times. In our DataFusion-based project, `supports_filters_pushdown(...)` has context-dependent behavior: the provider supports any single filter like `column = value`, but not multiple such filters at the same time.

Consider the following optimizer pipeline pattern:

1. Try to push `a = 1, b = 1`.
`supports_filters_pushdown` returns `[Exact, Inexact]`
OK: the optimizer records that `a = 1` is pushed and creates a filter node for `b = 1`.

...
Another optimization iteration.

2. Try to push b = 1.
`supports_filters_pushdown` returns `[Exact]`. Of course, the table provider can’t remember
all previously pushed filters, so it has no choice but to answer `Exact`.
Now, the optimizer thinks the conjunction `a = 1 AND b = 1` is supported exactly, but it is not.

To prevent this problem, I suggest passing filters that were already pushed into the scan earlier to `supports_filters_pushdown(...)`.

## Do not assume that filter support decision is stable

Consider the next scenario:

1. `supports_filters_pushdown` returns `Exact` on some filter, e.g. "a = 1", where column "a" is not
required by the query projection.

2. "a" is removed from the table provider projection by "optimize projection" rule.

3. `supports_filters_pushdown` changes a decision and returns `Inexact` on this filter the next time.
For example, input filters were changed and it prefers to use a new one.

4. "a" is not returned to the table provider projection which leads to filter that references a column which is
not a part of the schema.

Suggest to extend logic with the following actions:

1. Collect columns that are not used in the current table provider projection, but required for filter
expressions. Call it `additional_projection`.

2. If `additional_projection` is empty -- leave all as is.

3. Otherwise extend a table provider projection and wrap a plan with an additional projection node
to preserve schema used prior to this rule.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.