apache / apache/datafusion

refactor: make DynamicFilterPhysicalExpr.fetch a shared reference to SortExec.fetch

Open
#21,780 0 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Background

In #21580, `DynamicFilterPhysicalExpr` was given a `fetch: Option` field that stores the TopK K value. This value is copied from `SortExec.fetch` at filter creation time in `SortExec::create_filter()`.

## Problem

If a future optimizer calls `SortExec::with_fetch()` independently (without recreating the filter), the `DynamicFilterPhysicalExpr.fetch` would go stale. Currently this doesn't happen because `create_filter` and `fetch` are set in the same method, but the coupling is fragile.

Raised by @xudong963 in https://github.com/apache/datafusion/pull/21580#discussion_r3123894244

## Proposed fix

Replace `fetch: Option` with a shared reference (e.g., `Arc`) that reads directly from `SortExec.fetch`. This way any update to fetch is automatically visible to the parquet reader's stats init and cumulative pruning logic.

## Related

- #21580 — PR that introduced `fetch` on `DynamicFilterPhysicalExpr`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.