refactor: make DynamicFilterPhysicalExpr.fetch a shared reference to SortExec.fetch
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Background
In #21580, `DynamicFilterPhysicalExpr` was given a `fetch: Option` field that stores the TopK K value. This value is copied from `SortExec.fetch` at filter creation time in `SortExec::create_filter()`.
## Problem
If a future optimizer calls `SortExec::with_fetch()` independently (without recreating the filter), the `DynamicFilterPhysicalExpr.fetch` would go stale. Currently this doesn't happen because `create_filter` and `fetch` are set in the same method, but the coupling is fragile.
Raised by @xudong963 in https://github.com/apache/datafusion/pull/21580#discussion_r3123894244
## Proposed fix
Replace `fetch: Option` with a shared reference (e.g., `Arc`) that reads directly from `SortExec.fetch`. This way any update to fetch is automatically visible to the parquet reader's stats init and cumulative pruning logic.
## Related
- #21580 — PR that introduced `fetch` on `DynamicFilterPhysicalExpr`
Contributor guide
Assessment
This issue has not been assessed yet.