apache / apache/datafusion-comet

feat: IO-level dynamic filter pushdown into the native Delta scan (delta-kernel-rs predicate reuse)

Open
#4,809 0 comments 0 reactions 1 assignee Claimed by @schenksj View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

> _Disclosure: this issue was drafted with the help of an AI assistant ._

Follow-up to #4807 (runtime dynamic filter pushdown for native hash joins). Blocked on the native Delta scan contrib landing (tracking PR #4366).

The generic phase of #4807 applies the join's `DynamicFilterPhysicalExpr` as a row-level filter above the probe-side scan, which the Delta contrib's `DeltaKernelScanExec` inherits with no format-specific work. This issue covers the IO-level upgrade so the filter also prunes inside the delta-kernel-rs read.

## Why Delta is well positioned

`DeltaKernelScanExec` (contrib, PR #4366) is a kernel-read operator like `IcebergScanExec` — not a `ParquetSource` — so it does not get DataFusion's optimizer-driven scan pushdown. However, the contrib already ships a Spark -> delta-kernel predicate converter (`contrib/delta/native/src/predicate.rs`) used for static data skipping. The IO-level dynamic filter upgrade can reuse it rather than writing a new conversion layer.

## Proposed shape

1. Snapshot the populated `DynamicFilterPhysicalExpr` at first poll of the probe-side scan stream (join build completes before the probe stream is polled; may require deferring kernel-read construction into the stream, mirroring the Iceberg follow-up).
2. Convert the convertible conjuncts (min/max bounds always; `InList` when present) through the existing `predicate.rs` converter and attach to the kernel scan's physical predicate. The `HashTableLookupExpr` strategy is not convertible; keep bounds only (row-level wrapper from #4807 still applies the full filter).
3. Verify how much row-group pruning delta-kernel-rs's default-engine Parquet read actually performs with an executor-side predicate — this bounds the payoff and should be measured before/while implementing.
4. File-level pruning is out of scope: the file list is resolved driver-side against Delta log stats.

## Decision input

`dynamic_filter_rows_pruned` / selectivity metrics from #4807 on Delta star joins.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.