Support dynamic filters and work-table plans state reset
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
There is a `reset_plan_states` function which is intended to re-use an existing `ExecutionPlan`:
https://github.com/apache/datafusion/blob/d90d0746d64bf6e91a81b3ec6954369bd0851bb2/datafusion/physical-plan/src/execution_plan.rs#L1401-L1423
Currently, it does not support dynamic filters and work table of the recursive query, because they are injected into nodes once on the planning stage and then never updated. Actually, to re-use a plan, dynamic filters must be reset.
## Possible implementation
- Add `ExecutionPlan::dynamic_filters(...)` method which is intended to return current version of dynamic filters.
- Introduce `PushedDynamicFilters` struct to pass into `ExecutionPlan::with_new_state` method.
- Modify `reset_plan_states` to use the following algorithm:
* Create an empty `PushedDynamicFilters`
* Create an option `work_table` (`None` initially)
* Transform plan up to bottom
* When process a node: reset its state, use `with_new_state(pushed_filters)` and `with_new_state(work_table)` if `work_table` is Some. Then collect new dynamic filters from it and possibly a work table if it is a `RecursiveQueryExec` (successfull downcast).
- For each node that polls dynamic filter support `with_new_state` where will be an attempt to down-cast state into `PushedDynamicFilters`.
Contributor guide
Assessment
This issue has not been assessed yet.