apache / apache/datafusion-comet

Preserve primitive Iceberg pruning beside unsupported complex null conjuncts

Open
#5,883 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem?

Iceberg residual serialization currently requires both sides of an `AND` to convert. Collection null checks cannot bind in the pinned iceberg-rust version, so a query such as `WHERE l IS NOT NULL AND id > 5` drops the entire residual and loses native row-group pruning on `id`.

The post-scan filter still enforces the complete predicate, so this is a performance limitation, not incorrect results. PR #5732 allows collection-null scans to remain native. Its review also requests skipping unbindable complex-column null predicates before serialization; the conservative whole-conjunction behavior should remain until safe partial pushdown is implemented.

### Proposed improvement

Preserve representable conjuncts when safely weakening a residual in positive polarity. Keep the exact post-scan filter. Do not simply change every `AND` to keep whichever side converted: beneath `NOT`, that would strengthen the predicate and can incorrectly prune rows. Likewise, dropping an unsupported `OR` branch is unsafe.

### Validation

- A list/map null predicate combined with a primitive predicate retains primitive row-group pruning and matches Spark results.
- Nested `NOT`, `OR`, and mixed conjunctions preserve all qualifying rows.
- Missing or unsupported predicates continue to disable pushdown whenever safe weakening cannot be established.
- Cover null values and files containing both matching and nonmatching primitive values; verify actual pruning metrics, not only serialized predicate shape.

Requested in the review of #5732: https://github.com/apache/datafusion-comet/pull/5732#pullrequestreview-5187277929

Contributor guide

Open the contributing guide

Research direction

Start with the residual serialization behavior described in PR #5732 and its linked review, then trace how conjuncts involving unsupported collection null checks are handled. Validate against Spark results with nested NOT, OR, and mixed conjunctions, including null values, and verify actual row-group pruning metrics while preserving the complete post-scan filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.