apache / apache/datafusion

Support dynamic filters and work-table plans state reset

Open
#19,905 1 comment 0 reactions 1 assignee Claimed by @askalt View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

There is a `reset_plan_states` function which is intended to re-use an existing `ExecutionPlan`:

https://github.com/apache/datafusion/blob/d90d0746d64bf6e91a81b3ec6954369bd0851bb2/datafusion/physical-plan/src/execution_plan.rs#L1401-L1423

Currently, it does not support dynamic filters and work table of the recursive query, because they are injected into nodes once on the planning stage and then never updated. Actually, to re-use a plan, dynamic filters must be reset.

## Possible implementation

- Add `ExecutionPlan::dynamic_filters(...)` method which is intended to return current version of dynamic filters.
- Introduce `PushedDynamicFilters` struct to pass into `ExecutionPlan::with_new_state` method.
- Modify `reset_plan_states` to use the following algorithm:
* Create an empty `PushedDynamicFilters`
* Create an option `work_table` (`None` initially)
* Transform plan up to bottom
* When process a node: reset its state, use `with_new_state(pushed_filters)` and `with_new_state(work_table)` if `work_table` is Some. Then collect new dynamic filters from it and possibly a work table if it is a `RecursiveQueryExec` (successfull downcast).
- For each node that polls dynamic filter support `with_new_state` where will be an attempt to down-cast state into `PushedDynamicFilters`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.