apache / apache/iceberg-rust

IcebergTableProvider::supports_filters_pushdown marks every filter as Inexact, causing a redundant FilterExec above IcebergTableScan

Open
#2,363 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

`IcebergTableProvider::supports_filters_pushdown` in [`crates/integrations/datafusion/src/table/mod.rs:163-169`](https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/table/mod.rs#L163-L169) unconditionally returns `TableProviderFilterPushDown::Inexact` for every filter:

```rust
fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> DFResult> {
Ok(vec![TableProviderFilterPushDown::Inexact; filters.len()])
}
```

`Inexact` tells DataFusion: *"the scan may apply this filter, but don't trust it, re-evaluate it above the scan."* DataFusion therefore leaves a `FilterExec` on top of `IcebergTableScan` that re-evaluates the same predicate on every row the scan emits.

However, the scan **does** already apply the predicate exactly. In [`crates/iceberg/src/arrow/reader.rs:244-256`](https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/arrow/reader.rs#L244-L256), the bound predicate is pushed into the Parquet reader as an Arrow `RowFilter` (`ArrowPredicateFn`), so every batch yielded by the scan already satisfies the filter. The `FilterExec` above is pure overhead.

#### Reproducer

`EXPLAIN ANALYZE` output with the query `SELECT * FROM t WHERE foo = 1` (a filter that is losslessly convertible to an iceberg `Predicate`):

```
CoalesceBatchesExec: ..., metrics=[output_rows=1, elapsed_compute=19µs]
FilterExec: foo@0 = 1, metrics=[output_rows=1, elapsed_compute=90µs] <-- redundant
RepartitionExec: ..., metrics=[fetch_time=4.4ms, ...]
IcebergTableScan predicate:[foo = 1] metrics=[]
```

`FilterExec` evaluates `foo@0 = 1` a second time on every row the scan emits.

### Describe the solution you'd like

In `supports_filters_pushdown`, return `TableProviderFilterPushDown::Exact` for filters that `convert_filter_to_predicate` translates **losslessly** into an iceberg `Predicate`, and `Inexact` (or `Unsupported`) otherwise.

The non-trivial part is detecting lossy conversions.

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Start in crates/integrations/datafusion/src/table/mod.rs at IcebergTableProvider::supports_filters_pushdown, then trace convert_filter_to_predicate and the predicate handling in crates/iceberg/src/arrow/reader.rs. Determine which conversions are lossless and verify that exact filters no longer require redundant FilterExec evaluation while other filters retain safe behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.