apache / apache/datafusion-comet
feat: IO-level dynamic filter pushdown into the native Delta scan (delta-kernel-rs predicate reuse)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
> _Disclosure: this issue was drafted with the help of an AI assistant ._
Follow-up to #4807 (runtime dynamic filter pushdown for native hash joins). Blocked on the native Delta scan contrib landing (tracking PR #4366).
The generic phase of #4807 applies the join's `DynamicFilterPhysicalExpr` as a row-level filter above the probe-side scan, which the Delta contrib's `DeltaKernelScanExec` inherits with no format-specific work. This issue covers the IO-level upgrade so the filter also prunes inside the delta-kernel-rs read.
## Why Delta is well positioned
`DeltaKernelScanExec` (contrib, PR #4366) is a kernel-read operator like `IcebergScanExec` — not a `ParquetSource` — so it does not get DataFusion's optimizer-driven scan pushdown. However, the contrib already ships a Spark -> delta-kernel predicate converter (`contrib/delta/native/src/predicate.rs`) used for static data skipping. The IO-level dynamic filter upgrade can reuse it rather than writing a new conversion layer.
## Proposed shape
1. Snapshot the populated `DynamicFilterPhysicalExpr` at first poll of the probe-side scan stream (join build completes before the probe stream is polled; may require deferring kernel-read construction into the stream, mirroring the Iceberg follow-up).
2. Convert the convertible conjuncts (min/max bounds always; `InList` when present) through the existing `predicate.rs` converter and attach to the kernel scan's physical predicate. The `HashTableLookupExpr` strategy is not convertible; keep bounds only (row-level wrapper from #4807 still applies the full filter).
3. Verify how much row-group pruning delta-kernel-rs's default-engine Parquet read actually performs with an executor-side predicate — this bounds the payoff and should be measured before/while implementing.
4. File-level pruning is out of scope: the file list is resolved driver-side against Delta log stats.
## Decision input
`dynamic_filter_rows_pruned` / selectivity metrics from #4807 on Delta star joins.
Contributor guide
Assessment
This issue has not been assessed yet.