apache / apache/datafusion-comet

Iceberg scan falls back to Spark on IS NULL/IS NOT NULL over list/map columns (stale complex-type check)

Open
#5,731 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

## Describe the bug

`CometScanRule` declines every Iceberg scan whose pushed filters contain an `IS NULL`/`IS NOT NULL` predicate on a complex-typed column, with the reason:

```
IS NULL / IS NOT NULL predicates on complex type columns (struct/array/map) are not yet supported by iceberg-rust
```

The check (added in #2528) predates the current iceberg-rust pin (bumped in #5651). iceberg-rust's Arrow predicate visitor now evaluates `is_null`/`is_not_null` on **list** and **map** columns through arrow's native `is_null`/`is_not_null` (`crates/iceberg/src/arrow/reader/predicate_visitor.rs`); only **struct** columns are rejected (`project_column` errors on `DataType::Struct`).

## Impact

The check is a false positive for list columns, which is a common real pattern: Spark's optimizer pushes `isnotnull(arr)` below `Generate` for `explode(arr)`:

```
BatchScan local_tpch.orders_nested [COMET: IS NULL / IS NOT NULL predicates on complex type columns ...]
```

Each declined scan forces the whole plan back to Spark and cascades into further fallbacks (`Comet aggregate that merges intermediate buffers requires a Comet child aggregate ...`). On a 24-query derived TPC-H benchmark (SF1, 3 runs, cold caches, Spark 4.1.3, Iceberg hadoop catalog, table with `array` line-item column):

| metric | check active | list false-positive removed |
|---|---:|---:|
| Comet/Iceberg total (24q) | 20.63 s | 16.93 s (−17.9%) |
| `q1_nested_explode` | 2.20 s | 0.77 s (−65%) |
| `e14_nested` | 2.03 s | 0.86 s (−58%) |
| Comet/Iceberg vs Comet/Parquet total | +26% | +3.7% |

Parity vs vanilla Spark verified on all 24 queries in both configurations.

A fix that narrows the check to struct-only columns will be submitted as a PR.

Contributor guide

Open the contributing guide

Research direction

Start in CometScanRule, then read crates/iceberg/src/arrow/reader/predicate_visitor.rs to compare the supported null predicates for list, map, and struct columns. Run the affected Iceberg scan coverage or reproduction described in the issue; done means list/map scans no longer fall back to Spark while struct-column predicates remain rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.